H$^2$SD: Hybrid Hindsight Self-Distillation

📄 arXiv: 2607.18955v1 📥 PDF

作者: Qiye Cai, Yichuan Ma, Linyang Li, Peiji Li, Yongkang Chen, Qipeng Guo, Yicheng Zou, Tao Gui, Xiaocheng Feng, Bing Qin

分类: cs.LG, cs.CL

发布日期: 2026-07-21

💡 一句话要点

提出H$^2$SD框架以解决稀疏监督问题

🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture) 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 强化学习 自蒸馏 推理能力 教师模型 稀疏监督 自然语言处理 数学推理

📋 核心要点

  1. 现有RLVR方法在奖励分配上存在稀疏性，导致推理能力受限，尤其在令牌级别的信用分配上表现不佳。
  2. H$^2$SD框架通过根据轨迹的成功与否灵活调整教师模型的使用方式，从而实现更有效的自蒸馏。
  3. 实验结果显示，H$^2$SD在多个推理基准上均优于RLVR、OPSD和RLSD等基线方法，优化过程稳定且生成效率高。

📝 摘要（中文）

强化学习与可验证奖励（RLVR）显著提升了大型语言模型在数学推理和代码生成等任务上的推理能力。然而，现有RLVR方法通常将标量奖励分配给整个轨迹，导致稀疏监督和有限的令牌级信用分配。本文提出的H$^2$SD框架通过根据轨迹的正确性不同地使用教师模型，解决了这一问题。成功轨迹中，教师模型根据确认正确的学生响应进行调整；失败轨迹中，则通过参考提示来指导学生。实验表明，H$^2$SD在多个推理基准上优于现有方法，优化稳定且生成效率高。

🔬 方法详解

问题定义：本文旨在解决现有RLVR方法在奖励分配上的稀疏性问题，导致推理能力受限及令牌级信用分配不足。

核心思路：H$^2$SD框架通过根据轨迹的成功与否，灵活地使用教师模型来提供更密集的监督，从而避免信息泄漏和优化不稳定的问题。

技术框架：H$^2$SD的整体架构包括两个主要模块：成功轨迹的教师信号调节和失败轨迹的参考提示引导。成功轨迹中，教师模型根据确认的学生响应进行调整；失败轨迹中，使用参考提示来指导学生。

关键创新：H$^2$SD的创新点在于根据轨迹的正确性灵活调整教师模型的使用方式，避免了直接匹配教师分布带来的信息泄漏问题。

关键设计：在成功轨迹中，教师模型的概率用于调节更新幅度，而在失败轨迹中，通过最小化学生与教师之间的反向KL散度来进行优化。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

在多个推理基准测试中，H$^2$SD框架的表现优于RLVR、OPSD和RLSD等基线方法，优化过程稳定，生成效率高。具体实验结果显示，H$^2$SD在推理任务上的性能提升幅度达到了X%（具体数据未知），展示了其在复杂推理任务中的有效性。

🎯 应用场景

H$^2$SD框架在自然语言处理、数学推理和代码生成等领域具有广泛的应用潜力。其通过提高推理能力和生成效率，能够为智能助手、自动编程和教育技术等应用提供更强大的支持，推动相关技术的进步。

📄 摘要（原文）

Reinforcement learning with verifiable rewards (RLVR) has substantially improved the reasoning capabilities of large language models on tasks such as mathematical reasoning and code generation. However, most RLVR methods assign a scalar outcome reward to an entire trajectory, resulting in sparse supervision and limited token-level credit assignment. On-policy distillation (OPD) provides denser supervision by distilling token-level distributions from a stronger teacher model, but requires an additional teacher and typically assumes a shared vocabulary. On-policy self-distillation (OPSD) removes this dependency by conditioning the same model on privileged information to construct a teacher policy. However, directly matching the teacher distribution may cause information leakage and unstable optimization. RLSD avoids direct matching by using the teacher signal only to modulate update magnitudes, but it cannot provide an explicit correction direction when the sampled reasoning fails. To address this tradeoff, we introduce $\mathrm{H}^{2}\mathrm{SD}$, a hybrid hindsight self distillation framework that uses the teacher differently according to trajectory correctness. For successful trajectories, the teacher receives the student response confirmed as correct together with a rephrasing instruction, and its probabilities on the original response tokens are used to modulate update magnitudes without changing the direction determined by the reward. For failed trajectories, we condition the teacher on a reference hint containing key reasoning steps and a verified answer, and minimize the reverse KL divergence from the student to the teacher. Experiments on multiple challenging reasoning benchmarks show that H$^2$SD consistently outperforms representative RLVR, OPSD, and RLSD baselines while maintaining stable optimization and favorable generation efficiency.