Sharpen Without Search: On-Policy Distillation of Sequence-Level Power Distribution

📄 arXiv: 2610.06804v1 📥 PDF

作者: Erfan Baghaei Potraghloo, Seyedarmin Azizi, Arya Fayyazi, Saeid Shokoufa, Mehdi Kamal, Souvik Kundu, Massoud Pedram

分类: cs.LG, cs.AI

发布日期: 2026-10-05

🔗 代码/项目: GITHUB

💡 一句话要点

提出在政策蒸馏中优化序列级功率分布的方法

🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture)

关键词: 在线政策蒸馏 功率分布 语言模型 推理能力 数学问题

📋 核心要点

  1. 现有语言模型在生成答案时，常常会受到错误答案的影响，导致正确答案的采样概率降低。
  2. 论文提出的在线政策蒸馏（OPPD）方法，通过训练模型生成候选答案，并利用冻结教师模型的功率分布进行加权，从而提升生成答案的质量。
  3. 实验结果显示，OPPD在MATH500和GSM8K数据集上分别提高了23.0和27.3的准确率，且在HumanEval上也有显著提升。

📝 摘要（中文）

语言模型在生成答案时，尽管某个正确答案的概率高于任何单个错误答案，但通常仍会采样到错误答案。这是因为错误答案的总概率更高。本文提出了一种新的方法，通过在训练过程中使用在线政策蒸馏（OPPD），使模型能够在一次生成中产生更高概率的答案。该方法通过一个冻结的教师模型的功率分布来加权生成的候选答案，从而提高了模型的推理能力。实验结果表明，OPPD在多个基准数据集上显著提高了模型的准确性，尤其是在数学问题上，单次生成的准确率提升达23.0点和27.3点。

🔬 方法详解

问题定义：本文旨在解决语言模型在生成答案时，因错误答案的总概率较高而导致正确答案采样概率降低的问题。现有方法需要多个候选答案进行评分，效率较低。

核心思路：论文提出的在线政策蒸馏（OPPD）方法，通过训练模型在一次生成中直接产生高概率答案，避免了传统方法的多次采样过程。

技术框架：OPPD的整体架构包括一个训练模型和一个冻结的教师模型。训练模型生成候选答案，教师模型的功率分布对这些答案进行加权，最终通过最大似然更新来优化模型。

关键创新：OPPD的核心创新在于通过教师模型的功率分布来加权生成的候选答案，从而在一次生成中提升答案的准确性。这一方法与传统的多候选评分方法本质上不同，显著提高了效率。

关键设计：在训练过程中，使用一个损失系数来调整模型吸收的功率分布的指数，范围在1.19到2.02之间，相较于普通的政策蒸馏（1.14）有显著提升。

🖼️ 关键图片

img_0
img_1
img_2

📊 实验亮点

实验结果表明，OPPD在MATH500和GSM8K数据集上分别提高了23.0和27.3的准确率，且在HumanEval上提升了5.3点。与传统的64候选答案的功率采样相比，OPPD在单次生成中也表现出更优的性能，恢复了94%的增益。

🎯 应用场景

该研究的潜在应用领域包括教育、自动问答系统和智能助手等。通过提升语言模型在数学和逻辑推理方面的能力，可以为用户提供更准确的答案，进而提高学习和工作效率。未来，该方法有望推广到更广泛的自然语言处理任务中。

📄 摘要（原文）

A language model can give a correct answer more probability than any single incorrect answer and still usually sample an incorrect one, because the incorrect answers together hold more probability. The power distribution raises each complete answer's probability to a power above one and renormalizes, shifting probability toward answers the model finds most likely (sharpening). Sampling from it improves reasoning without changing parameters, but needs many scored candidates per query. We show that a model can instead be trained to produce such answers in one generation. On-policy power distillation (OPPD) runs a sequential Monte Carlo sampler in which the model being trained generates candidates and a frozen teacher's power distribution weights them; the same probabilities weight each answer in a maximum-likelihood update. Training raises single-generation accuracy by up to 23.0 points on MATH500 and 27.3 on GSM8K over the untrained model at the same temperature, and one generation scores 2.4 and 3.5 points above published power sampling with 64 candidates, recovering 94 percent of the gain that 16 candidates give the untrained model. For context, against GRPO trained with verified rewards from the same checkpoint and budget, OPPD scores 3.8, 4.0 and 5.4 points higher on MATH500, GSM8K and AIME using no reference answers; the two are complementary, and OPPD applied after GRPO adds up to 9.3 points. Trained only on mathematics, OPPD raises HumanEval accuracy by up to 5.3 points. One loss coefficient moves the sharpening exponent the model absorbs between 1.19 and 2.02, against 1.14 for ordinary on-policy distillation, and it rises mostly on the model's own answers. Gains hold across model families and sizes, including a model already trained with verified rewards, where lowering the temperature gives nothing and OPPD adds 4.4 points on MATH500. Code: https://github.com/ArminAzizi98/OPPD.