Projected Distillation: On Learning to Solve Hard Problems from Close, Correct Targets
作者: Shripad V. Deshmukh, Christian Bock, Kangrui Ruan, Abhijeet Awasthi, Vijay Lingam, Rein Houthooft, Manish Prajapat
分类: cs.LG, cs.AI
发布日期: 2026-10-20
💡 一句话要点
提出投影蒸馏以解决困难问题的学习挑战
🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture)
关键词: 投影蒸馏 强化学习 错误修正 对比更新 模型训练 自然语言处理 代码生成
📋 核心要点
- 现有的基于策略的强化学习在困难问题上缺乏有效的奖励信号，导致学习停滞。
- 论文提出的投影蒸馏方法通过最小化编辑失败尝试，利用对比更新实现近似基于策略的学习。
- 实验结果显示，该方法在数学、代码、科学和工具使用等领域表现优异，提升了专业化与保留的平衡。
📝 摘要（中文）
在后训练阶段，基于策略的学习是核心，但在困难问题上，所有尝试失败时缺乏奖励信号。论文提出的投影蒸馏方法通过最小化编辑学生的失败尝试，将其转化为正确目标，利用对比更新来优先考虑修正。这种方法允许在所有尝试失败时实现近似基于策略的学习。理论证明学习遗憾与编辑大小成正比，实验证明该方法在多个领域中表现优异，显著提升了专业化与保留的平衡。
🔬 方法详解
问题定义：本论文旨在解决在困难问题上，基于策略的强化学习因缺乏有效奖励信号而导致的学习停滞。现有的基于策略蒸馏方法在失败的情况下，后续的监督信息往往受到早期错误的影响，导致学习效果不佳。
核心思路：论文提出的投影蒸馏方法通过最小化编辑学生的失败尝试，将其转化为正确的目标，从而实现近似基于策略的学习。该方法通过对比更新，优先考虑修正而非失败尝试，使得学生在学习过程中能够不断调整和优化。
技术框架：该方法的整体架构包括两个主要模块：首先是失败尝试的编辑模块，其次是对比更新模块。编辑模块负责将学生的错误尝试转化为正确目标，而对比更新模块则通过对比损失来优化学习过程。
关键创新：最重要的技术创新在于将基于策略的学习视为一个连续体，通过投影蒸馏方法实现了在所有尝试失败时的有效学习。这与传统的行为克隆和DAgger方法有本质区别，后者在面对错误时往往无法有效调整。
关键设计：在技术细节上，论文采用了对比损失函数来强化修正的效果，并通过在线修正机制不断更新目标。此外，实验中使用了Qwen3-8B和OLMo-2-7B两个学生模型，涵盖了多个领域的应用。具体参数设置和网络结构的细节在实验部分进行了详细描述。
🖼️ 关键图片
📊 实验亮点
实验结果显示，投影蒸馏方法在九个测试单元中，针对GRPO和OPD方法的表现最佳，八个单元中表现优于对比基线，七个单元中优于SPOT和SODA，专业化与保留的平衡显著提升，保留率从0.916降至0.744时，最小修正被替换为学生无关的黄金解决方案。
🎯 应用场景
该研究的潜在应用领域包括自然语言处理、代码生成、科学计算等多个需要高效学习的场景。通过投影蒸馏方法，模型能够在面对复杂任务时更好地保留已有能力，同时提升学习效率，具有重要的实际价值和未来影响。
📄 摘要（原文）
On-policy learning is central to post-training: training on a model's own generations limits distribution shift and helps preserve existing capabilities. On hard problems, however, on-policy RL provides no reward-based learning signal when all attempts fail. On-policy distillation provides next-token supervision from a teacher, but on hard problems, if a rollout goes wrong early, subsequent supervision is conditioned on prefixes that already contain the mistake. Correct behavior can instead come from a gold or teacher solution, but such off-policy targets can differ substantially from the model's generations and, as we show, may erode existing capabilities. We therefore treat on-policyness as a continuum and propose projected distillation, which minimally edits the student's failed attempts into correct targets. The method uses contrastive updates to favor corrections over failed attempts and refreshes these corrections as the student evolves, enabling near-on-policy learning even when all student attempts fail. Theoretically, we prove that the student's learning regret grows with the edit size: it scales linearly with the horizon under minimal correction, as in DAgger, and quadratically under full-answer replacement, as in behavior cloning. Empirically, across two students (Qwen3-8B and OLMo-2-7B) and five domains spanning math, code, science, and tool use, projected distillation gives the highest area under the specialization-retention frontier (best on 8 of 9 cells against GRPO and OPD, and 7 of 9 against SPOT and SODA). Our ablations highlight the importance of online corrections, the contrastive loss, and on-policyness: moving targets farther off-policy steadily worsens the specialization-retention trade-off, with retention falling from 0.916 to 0.744 when minimal corrections are replaced by student-agnostic gold solutions.