Better Starts, Better Ends: Bootstrapped Iterative Self-Reasoning Distillation for Compressed Reasoning

📄 arXiv: 2607.15736 📥 PDF

作者: Leichao Dong, Dongxu Zhang, Yiding Sun, Qirui Wang, Yuhan Wang, Lin Chen, Jihua Zhu

分类: cs.CL

发布日期: 2026-07-20

💡 一句话要点

提出BIRD方法以解决推理模型冗余计算问题

🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture) 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 推理模型 自蒸馏 长链推理 反向KL蒸馏 简洁解 高效推理 机器学习 模型优化

📋 核心要点

  1. 现有自蒸馏方法在推理过程中存在初始化瓶颈，导致训练效果受限于冗余和噪声数据。
  2. BIRD方法通过两阶段的自我推理蒸馏，先生成简洁的解，再进行反向KL蒸馏，提升了推理的有效性。
  3. 在Qwen3系列模型上，BIRD在MATH-500和AIME基准测试中显著提高了准确性和效率，表现优于传统方法。

📝 摘要（中文）

大型推理模型通常通过长链推理（CoT）解决问题，但大量计算用于冗余推导、自我验证和无效的绕行，这些都未能提升最终答案的质量。现有的自蒸馏方法通过将学生模型与自身的简洁副本匹配来减少这种成本，但存在初始化瓶颈。本文提出BIRD（Bootstrapped Iterative Self-Reasoning Distillation），一种两阶段自我推理蒸馏方法，旨在改善推理分布。BIRD首先在简洁指令下从基础模型中采样简洁解，并仅保留正确答案的轨迹，然后进行轻量级的提示切换SFT步骤。接着，BIRD在更清晰的信息前缀上应用反向KL蒸馏，从而实现更高的准确性和效率。

🔬 方法详解

问题定义：本文旨在解决大型推理模型在推理过程中存在的冗余计算和初始化瓶颈问题。现有方法在自蒸馏时，训练数据往往包含噪声和冗余信息，影响最终推理效果。

核心思路：BIRD方法通过两阶段的自我推理蒸馏，首先在简洁指令下生成高质量的推理轨迹，然后在这些轨迹上进行反向KL蒸馏，从而改善推理分布，提升模型性能。

技术框架：BIRD的整体架构包括两个主要阶段：第一阶段是从基础模型中采样简洁解并进行轻量级的提示切换SFT；第二阶段是在更清晰的信息前缀上应用反向KL蒸馏。

关键创新：BIRD的创新点在于通过简洁指令生成的轨迹来改善自蒸馏过程，避免了传统方法中因噪声和冗余导致的性能下降。

关键设计：在BIRD中，简洁指令的设计和轻量级的提示切换是关键，确保生成的轨迹既简洁又准确，同时在反向KL蒸馏中使用的损失函数也经过优化，以适应新的推理分布。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

在Qwen3-8B模型上，BIRD方法将MATH-500的准确率从86.2%提升至92.0%，同时将平均响应长度从3,099个标记减少至1,115个标记。这些结果表明，前缀支持在高效推理蒸馏中起着关键作用。

🎯 应用场景

该研究的潜在应用领域包括教育、科学计算和复杂决策支持等场景。通过提高推理模型的效率和准确性，BIRD方法能够在实际应用中显著减少计算资源的消耗，并提升用户体验。未来，该方法可能在更广泛的推理任务中得到应用，推动智能系统的发展。

📄 摘要（原文）

Large reasoning models often solve problems through long chain-of-thought (CoT) traces, yet much of this computation is spent on redundant derivations, repeated self-verification, and detours that do not improve the final answer. Existing on-policy self-distillation methods reduce this cost by matching a student model to a concise copy of itself on prefixes sampled from the student's own rollouts. We show that this objective has an initialization bottleneck. Since supervision is applied only to visited prefixes, training from a verbose base model places the KL loss on contexts that are often noisy, redundant, or already off track. In such regions, a concise teacher can provide only local corrections, while the student continues to explore trajectories that an efficient reasoner should avoid. In this paper, we propose BIRD(Bootstrapped Iterative Self-Reasoning Distillation), a two-stage self-reasoning distillation method that improves the rollout distribution before on-policy training. BIRD first samples concise solutions from the base model under a brevity instruction, keeps only answer-correct traces, and performs a lightweight prompt-switch SFT step. The traces are generated with the brevity instruction but learned under the original task prompt, turning instruction-induced conciseness into a default reasoning behavior. Starting from this warm model, BIRD then applies on-policy reverse-KL distillation with a concise self-teacher, now on cleaner and more informative prefixes. Across Qwen3 series models, BIRD achieves a stronger accuracy-efficiency trade-off than prompting and cold-start on-policy distillation on MATH-500 and AIME benchmarks. On Qwen3-8B, it improves MATH-500 accuracy from 86.2% to 92.0% while reducing the average response length from 3,099 to 1,115 tokens. These results highlight prefix support as a central factor in efficient reasoning distillation.