Learning More by Training Less: Uncertainty-Guided Progressive Contraction for Efficient Reasoning Post-Training
作者: Suorong Yang, Jiaxin Wang, Jiaming Su, Chentao Yue, Furao Shen, Peng Ye, Soujanya Poria
分类: cs.AI
发布日期: 2026-10-09
💡 一句话要点
提出不确定性引导的渐进收缩框架以提高推理效率
🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)
关键词: 推理效率 不确定性引导 渐进收缩 大型语言模型 在线学习
📋 核心要点
- 现有方法在推理后训练中面临成本与性能提升不成比例的挑战。
- 本文提出的不确定性引导的渐进收缩框架，通过集中监督和优化，提升推理效率。
- 实验结果显示，该方法在多种推理基准上使用的训练预算大幅减少，同时保持或提升了推理性能。
📝 摘要（中文）
大型语言模型在复杂推理方面取得了显著进展。然而，随着语料库规模的扩大，推理后训练的成本显著增加，而性能提升却不成比例。为此，本文提出了一种不确定性引导的渐进收缩框架，旨在在不牺牲能力的前提下压缩推理监督。该框架通过构建蒙特卡洛推理树，集中优化最有价值的推理轨迹，从而显著减少学生的训练预算。实验表明，该方法在保持或提升推理性能的同时，仅使用了全数据SFT优化预算的0.22%，训练暴露量减少了450倍。
🔬 方法详解
问题定义：本文旨在解决大型语言模型推理后训练中成本高昂且性能提升有限的问题。现有方法在扩大语料库时，训练成本显著增加，但性能提升却不明显。
核心思路：提出的不确定性引导的渐进收缩框架，通过集中监督和优化最有价值的推理轨迹，来压缩推理监督，从而降低训练成本。
技术框架：该框架分为两个主要阶段：第一阶段通过构建蒙特卡洛推理树，收缩原始语料库；第二阶段在学生学习状态演变的基础上进行在线路由，集中优化。
关键创新：最重要的创新在于通过不确定性分析来指导监督和优化的收缩过程，这与传统方法的全数据训练方式有本质区别。
关键设计：在技术细节上，采用了不确定性分析来构建推理监督池，并在在线学习过程中动态调整优化策略，以适应学生的学习进展。具体的损失函数和网络结构设计尚未详细披露。
🖼️ 关键图片
📊 实验亮点
实验结果显示，该方法在多种推理基准上仅使用了全数据SFT优化预算的0.22%，训练暴露量减少了450倍，同时保持或提升了推理性能。这表明该框架在不同学生模型和教师模型之间具有良好的泛化能力。
🎯 应用场景
该研究的潜在应用领域包括自然语言处理、智能问答系统和教育技术等。通过降低训练成本并提升推理能力，该方法能够在资源有限的情况下，帮助更多的研究机构和企业实现高效的模型训练，推动AI技术的普及与应用。
📄 摘要（原文）
Large language models have achieved remarkable advances in complex reasoning. However, scaling reasoning post-training with ever-larger corpora can substantially increase training cost without proportional performance gains. In this study, we therefore ask: how far can reasoning supervision be compressed without sacrificing capability? To this end, we propose an uncertainty-guided progressive contraction framework that concentrates supervision and optimization on the reasoning trajectories most valuable for learning. Specifically, rather than treating a teacher-generated chain-of-thought (CoT) as a single realized path, we construct uncertainty-profiled reasoning supervision by building Monte Carlo reasoning trees around intermediate reasoning states to expose process-level uncertainty. This uncertainty first contracts the original corpus into a compact teacher-informed pool. A second, learner-aware contraction then performs online routing during SFT, concentrating optimization according to the student's evolving learning state. Together, the two stages progressively contract both supervision and optimization, substantially reducing the student training budget. Experiments across diverse reasoning benchmarks and LLM families show that our method uses as little as 0.22% of the optimization budget of full-data SFT, over 450x less training exposure, while maintaining or improving reasoning performance. Notably, our framework generalizes well across heterogeneous student families, black-box teachers, and weak-to-strong transfer. This indicates that it is not only a way to accelerate training but can also enhance reasoning capability.