A Fisher-Geometric View of Multi-Teacher On-Policy Distillation

📄 arXiv: 2610.26525v1 📥 PDF

作者: Liu Kaiyuan, Zhuang Ziyuan, Ma Haoxuan, Gu Yannian, Song Mingfei, Weng Rongxiang, Ye Jieping

分类: cs.LG, cs.CL

发布日期: 2026-10-19

💡 一句话要点

提出RGBM方法以优化多教师在线蒸馏过程

🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture) 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 模型合并 在线蒸馏 多教师学习 Fisher合并 残差学习 计算效率 深度学习

📋 核心要点

  1. 现有的多教师在线蒸馏方法在计算开销和跨任务梯度冲突方面存在显著挑战。
  2. 本文提出的RGBM方法通过将合并过程分为两个阶段，优化了模型整合的效率和效果。
  3. 实验结果显示，RGBM在多个基准测试中超越了传统的多教师蒸馏方法，且没有增加额外的训练步骤。

📝 摘要（中文）

整合专业基础模型为统一的通用模型是后训练的核心目标。现有的两种常用方法分别是模型合并和多教师在线蒸馏。前者速度较快但被视为启发式，后者虽然能匹配分布但计算开销大且存在跨任务梯度冲突。本文展示了这两者在不同阶次下的统一目标，并提出了残差主导的分支与合并（RGBM）方法，优化了合并过程。实验表明，RGBM在两个开源整合套件上表现优于完全训练的多教师蒸馏，且在不增加联合训练步骤的情况下进一步提升了性能。

🔬 方法详解

问题定义：本文旨在解决多教师在线蒸馏过程中存在的计算开销大和跨任务梯度冲突的问题。现有的模型合并方法虽然快速，但缺乏理论支持，且多教师蒸馏方法计算复杂，难以高效实现。

核心思路：论文提出RGBM方法，通过将模型合并过程分为两个阶段，利用Fisher合并目标来优化多教师蒸馏的效果。这样设计的目的是在保持合并效率的同时，降低计算复杂度。

技术框架：RGBM的整体架构包括两个阶段：第一阶段限制合并到任务向量子空间，第二阶段根据残差谱来决定哪些领域可以添加更新。

关键创新：最重要的创新在于将Fisher合并与多教师在线蒸馏目标联系起来，提出了残差主导的合并策略，显著提高了模型整合的效果。

关键设计：在第一阶段，合并过程仅在任务向量子空间内进行，利用隐式奖励的混合系数来优化未标记提示的残差。第二阶段则根据能量归一化的领域残差谱来控制更新的写回，确保低残差领域被动化。

🖼️ 关键图片

fig_0
fig_1

📊 实验亮点

实验结果表明，RGBM的第一阶段合并在多个开源整合套件上与完全训练的多教师蒸馏方法相当或更优，而RGBM进一步提升了性能，且没有增加任何额外的联合训练步骤，显示出其高效性。

🎯 应用场景

该研究的潜在应用领域包括自然语言处理、计算机视觉等多个需要模型整合的任务。通过优化模型合并过程，RGBM方法能够提升基础模型的性能，降低计算资源的消耗，具有广泛的实际价值和未来影响。

📄 摘要（原文）

Consolidating specialized foundation models into a unified generalist is a central goal of post-training. Two approaches are in common use: model merging, which is fast but treated as a heuristic, and multi-teacher on-policy distillation, which matches distributions but demands heavy rollout compute and suffers from cross-task gradient conflict. We show that the two are one objective seen at different orders. Within a shared basin of homologous specialists, the multi-teacher on-policy distillation objective is, to second order, exactly the Fisher-merging objective. Its minimizer is therefore the Fisher merge, a classic model-merging method given by a closed-form combination of the specialists, so merging is optimal, to second order, as a closed-form stage of distillation. Building on this identification, we propose Residual-Governed Branch-and-Merge(RGBM), a two-phase consolidation procedure. Concretely, a full-parameter Fisher merge is intractable on foundation models, so Phase~1 restricts the merge to the task-vector subspace. Extending tangent-space analyses of merging, we show that the same coefficients that mix task vectors also mix implicit rewards, which lets a residual program on unlabelled prompts solve for them. The program minimizes the worst per-domain residual and reports its energy-normalized values as the domain residual spectrum. Phase~2 lets this spectrum govern what may be added: RGBM passivates domains with low normalized residuals, trains isolated branches only for domains above the spectral gap, and writes back the admitted updates with calibrated shrinkage. On two open-source consolidation suites, the Phase~1 merge alone matches or outperforms fully trained multi-teacher distillation, and RGBM improves on it further with zero extra joint-training steps, whereas every un-gated write-back, including continued joint distillation from the merge, falls below it.