Rethinking Relative Policy Differences Transfer of On-Policy Distillation
作者: Rong Wu, Daocheng Fu, Yu Yang, Xuemeng Yang, Siqi Li, Shu Zou, Yao Zhu, Botian Shi, Bo Zhang, Cong Zhang, Pinlong Cai, Jianbiao Mei
分类: cs.LG
发布日期: 2026-10-09
💡 一句话要点
提出相对策略差异转移以优化在线蒸馏训练
🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture)
关键词: 在线蒸馏训练 策略优化 知识转移 强化学习 模型适应性
📋 核心要点
- 现有的在线蒸馏训练方法通常依赖于教师模型显著优于目标模型的假设，限制了其应用范围。
- 本文提出通过代理专家与代理参考的对数概率比来定义隐式奖励，支持更灵活的策略优化和模型转移。
- 实验结果显示，任务特定训练引起的差异具有较强的转移性，而模型规模引起的差异则转移效果较差。
📝 摘要（中文）
在线蒸馏训练（OPD）通过从更强的教师模型获取标记级监督来训练目标策略，但通常依赖于教师模型显著优于目标模型的前提。本文提出了一种统一视角，通过代理专家与代理参考之间的对数概率比来定义隐式奖励，从而实现KL正则化的策略优化。这种方法不仅恢复了传统的OPD作为特例，还支持不同规模、训练阶段或生成的模型之间的任意转移。实验结果表明，任务特定训练引起的差异具有一致的转移性，而单纯由模型规模引起的差异则转移效果较弱。此外，专家和参考模型在转移中扮演互补角色，成功的转移需要更新能够将目标模型朝向专家模型移动并超越采样回合。
🔬 方法详解
问题定义：本文旨在解决在线蒸馏训练中教师模型与目标模型能力差异过大所带来的限制，现有方法在不同模型间的转移效果不佳。
核心思路：通过引入代理专家与代理参考之间的对数概率比，定义隐式奖励，从而实现KL正则化的策略优化，允许不同规模和训练阶段的模型之间的灵活转移。
技术框架：整体架构包括三个主要模块：代理专家模型、代理参考模型和目标策略模型。通过对代理专家与参考模型的对比，生成隐式奖励并更新目标策略。
关键创新：最重要的创新在于提出了相对策略差异的转移机制，打破了传统方法对教师模型能力的依赖，允许在不同模型间进行有效的知识转移。
关键设计：设计中使用了KL正则化损失函数，确保目标策略在更新过程中能够有效吸收来自专家的知识，同时过滤掉不必要的共享行为。
🖼️ 关键图片
📊 实验亮点
实验结果表明，任务特定训练引起的策略差异在转移中表现出一致性，而单纯的模型规模差异则转移效果较弱。通过适当的参考模型，即使是原本不成功的转移也能得到改善，展示了该方法在策略优化中的有效性。
🎯 应用场景
该研究的潜在应用领域包括强化学习、机器人控制和自动驾驶等领域，能够提升模型在不同任务和环境下的适应性和性能。未来，随着模型规模和复杂度的增加，该方法有望在更广泛的应用场景中发挥重要作用。
📄 摘要（原文）
On-policy distillation (OPD) trains a target policy on its own rollouts using token-level supervision from a stronger teacher, but typically relies on the teacher being substantially more capable than the target. Recent work relaxes this requirement by transferring the relative policy difference between an expert and a reference policy rather than the expert's absolute distribution. Yet it remains unclear which differences provide useful supervision and which targets can benefit from them. We develop a unified view in which the log-probability ratio between a proxy expert and a proxy reference defines an implicit reward for KL-regularized policy optimization. This formulation induces an on-policy update for the target, recovers conventional OPD as a special case, and enables any-to-any transfer across models that differ in scale, training stage or generation. Our systematic experiments yield three findings. First, a stronger expert is not necessarily a better source of supervision: differences induced by task-specific training, including reinforcement learning and supervised fine-tuning, transfer consistently, whereas those arising from model scale alone transfer weakly or not at all. Second, the expert and reference serve complementary roles: the expert provides the capability to be transferred, while the reference filters out shared behavior and isolates the relevant residual. Third, even a useful difference does not transfer to every target: success requires the induced update to move the target toward the expert and generalize beyond sampled rollouts. Raw policy distance does not predict transferability, whereas an appropriate reference can rescue otherwise unsuccessful transfer. These results characterize when relative policy differences provide effective supervision for OPD.