Flash-OPD: Fast On-Policy Distillation
作者: Wei Chen, Junle Chen, Yitong Yang, Zhaoyang Xu, Jiaxin Lin, Yuxuan Liang, Xiaofang Zhou, Kai Wang, Rui Chen
分类: cs.LG
发布日期: 2026-10-05
备注: 21 pages, 9 figures
💡 一句话要点
提出Flash-OPD以解决长轨迹生成的高训练成本问题
🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture)
关键词: 政策蒸馏 轨迹生成 强化学习 教师-学生模型 高效训练 适应性验证
📋 核心要点
- 现有的政策蒸馏方法在生成和评估长轨迹时面临高昂的训练成本，且监督兼容性差异使得轨迹长度难以统一。
- 本文提出Flash-OPD，通过轨迹级的适应性边界验证，避免了对可靠性边界的预测，从而提高了效率。
- 在多种数据集和教师-学生设置下，Flash-OPD实现了2.2倍至7.5倍的速度提升，同时保持或改善了模型的准确性。
📝 摘要（中文）
在政策蒸馏（OPD）中，教师对学生生成的轨迹提供密集的监督，但生成和评估长轨迹会导致显著的训练成本。现有加速方法通过开放式或闭环的轨迹调度来降低成本，但由于轨迹间的监督兼容性差异，单一的轨迹长度难以匹配其可靠性。本文提出Flash-OPD，通过适应性轨迹级边界验证，避免了对轨迹特定可靠性边界的预测，从而实现了更高效的监督验证。实验表明，Flash-OPD在保持或提高准确度的同时，相比标准OPD实现了2.2倍至7.5倍的加速。
🔬 方法详解
问题定义：本文旨在解决在政策蒸馏中生成和评估长轨迹所带来的高训练成本问题。现有方法通过固定的轨迹长度进行监督，导致有效监督的截断或计算资源的浪费。
核心思路：Flash-OPD的核心思想是通过适应性轨迹级边界验证，避免在生成前预测轨迹特定的可靠性边界。通过观察到的前缀，准确判断是否达到了边界，从而实现高效的监督验证。
技术框架：Flash-OPD的整体架构包括轨迹生成、教师验证和停止决策三个主要模块。生成过程中，缓存生成与教师验证交替进行，每条轨迹根据其观察到的兼容性事件独立停止。
关键创新：Flash-OPD的创新在于将轨迹长度控制转变为适应性边界验证，避免了对可靠性边界的预测，显著提高了效率并减少了估计错误导致的提前终止。
关键设计：在设计中，使用最近事件率来调度下一个验证点，而实际的停止决策依赖于精确的累计计数。这种分离设计有效减少了验证开销，确保了生成过程的高效性。
🖼️ 关键图片
📊 实验亮点
实验结果显示，Flash-OPD在多种数据集和教师-学生设置下，相比标准OPD实现了2.2倍至7.5倍的速度提升，同时在准确性上保持或有所改善。这一显著的性能提升证明了其在高效监督学习中的有效性。
🎯 应用场景
Flash-OPD在强化学习和机器人控制等领域具有广泛的应用潜力。通过提高轨迹生成的效率，该方法可以加速训练过程，降低计算资源的消耗，进而推动智能体在复杂环境中的学习能力和适应性。未来，该技术可能在自动驾驶、智能制造等领域发挥重要作用。
📄 摘要（原文）
On-policy distillation (OPD) provides dense teacher supervision on student-generated trajectories, but generating and evaluating long rollouts incurs substantial training cost. Existing acceleration methods reduce this cost through open-loop rollout schedules or closed-loop horizon adaptation. However, supervision compatibility can vary substantially across trajectories, making a single rollout horizon difficult to match their heterogeneous reliable lengths: an overly short horizon may truncate useful supervision, while an overly long one wastes computation beyond reliable regions. Our key insight is that the trajectory-specific reliability boundary need not be predicted before generation. By viewing reliability as the first-passage of accumulated low teacher--student compatibility events, the boundary is inherently unknown before sampling, yet whether it has been reached can be determined exactly from the observed prefix. Building on this insight, we propose Flash-OPD, which shifts from rollout-horizon control to adaptive trajectory-level boundary verification. Flash-OPD interleaves cached generation with teacher verification and independently stops each trajectory according to its observed compatibility events. To reduce verification overhead, the recent event rate is used only to schedule the next verification point, while the actual stopping decision always relies on the exact cumulative count. This separation prevents estimation errors from causing premature termination while enabling efficient verification during generation. Extensive experiments across diverse datasets and teacher--student settings show that Flash-OPD achieves $2.2\times$--$7.5\times$ speedups over standard OPD while maintaining or improving accuracy.