Composing What Each Teacher Learned: Multi-Teacher On-Policy Distillation through Teacher-Relative Shifts
作者: Hejian Sang, Zhengze Zhou, Shayan Mohajer Hamidi, Xiaomin Li, Rohit Jain, Alborz Geramifard
分类: cs.LG, cs.AI
发布日期: 2026-10-07
💡 一句话要点
提出Δ-MOPD以优化多教师在线蒸馏过程
🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture)
关键词: 多教师蒸馏 在线学习 强化学习 知识转移 模型组合
📋 核心要点
- 现有的多教师蒸馏方法在转移教师策略时，可能受到基础策略的影响，导致性能下降。
- 本文提出的Δ-MOPD方法，通过重新锚定教师的对数偏移，旨在优化教师信号的组合效果。
- 实验结果显示，Δ-MOPD在多个基准测试中显著提高了性能，尤其是在分阶段路由设置下。
📝 摘要（中文）
多教师在线蒸馏（MOPD）在两种设置中使用：在共同领域组合中，多个教师对来自一个提示领域的学生回合进行评分，并形成单一目标；在路由领域蒸馏中，不同领域的提示分配给相应的专家。现有方法通常转移每个教师的最终策略，混合了后训练变化与教师基础的偏好。本文提出了Δ-MOPD，通过在学生的冻结初始化处重新锚定每个教师的教师减去基础的对数偏移，比较了两种设置下的最终监督。实验结果表明，偏移目标在教师信号组合时特别有效，Δ-MOPD在多个基准测试中超越了最终组合，且在分阶段路由下表现更佳。
🔬 方法详解
问题定义：本文旨在解决现有多教师在线蒸馏方法中，教师策略转移受到基础策略影响的问题，导致性能未能达到预期。
核心思路：提出Δ-MOPD，通过在学生的冻结初始化处重新锚定教师的对数偏移，优化教师信号的组合，减少基础策略的干扰。
技术框架：整体架构包括两个主要阶段：首先是教师信号的收集与组合，其次是基于组合信号进行学生策略的更新，确保在每个阶段都能有效利用教师的知识。
关键创新：Δ-MOPD的核心创新在于引入了教师减去基础的对数偏移作为目标，显著减少了基础策略的影响，与传统的最终策略转移方法形成鲜明对比。
关键设计：在实验中，教师选择保持固定，损失函数设计为KL散度，确保目标与学生策略之间的有效对齐，同时在不同的路由策略下进行评估。
🖼️ 关键图片
📊 实验亮点
实验结果表明，使用Δ-MOPD方法时，三位教师组合的性能超越了传统的最终组合，提升幅度达到4.11分，而在分阶段路由下，平均性能也显著提高，减少了观察到的顺序差距，从10.50分降至6.42分。
🎯 应用场景
该研究的潜在应用领域包括自然语言处理、强化学习和多模态学习等领域，能够有效提升模型在复杂任务中的学习效率与性能。未来，Δ-MOPD方法可扩展到更多的教师-学生模型组合中，推动智能系统的进一步发展。
📄 摘要（原文）
Multi-teacher on-policy distillation (MOPD) is used in two settings. In common-domain composition, several teachers score each student rollout from one prompt domain and their signals form a single target; in routed-domain distillation, prompts from different domains are assigned to the corresponding specialist. Both settings usually transfer each teacher's endpoint policy, which mixes what post-training changed with preferences inherited from the teacher's base. We introduce $Δ$-MOPD, which transfers each teacher's teacher-minus-base logit shift re-anchored at the student's frozen initialization, and compare it with endpoint supervision in both settings while holding teacher selection fixed. We first expose the mechanism that impedes endpoint transfer: inherited base pull can exceed the post-training shift. Removing it reduces the teacher-term norm ratio and target--student KL. Across our experiments, the results suggest that shift targets are particularly useful when teacher signals are combined at a state. With three composed teachers, $Δ$-MOPD exceeds endpoint composition by $4.11$ Math and $1.95$ five-benchmark points; with two, it matches endpoint accuracy. Under phased routing, it achieves higher mean performance in both phase orders and reduces the observed order gap from $10.50$ to $6.42$ points. Under interleaved routing, where each update involves one teacher, the two targets perform comparably. The phased results provide supporting evidence that the benefit may extend to signals accumulated across training phases. Target construction is thus an independent design axis in MOPD, complementary to teacher selection.