Sharpening Tax in Post-Training

📄 arXiv: 2610.01509v1 📥 PDF

作者: Changdae Oh, Qi Zeng, Qi Qi, Andrey Zhmoginov, Deren Lei, Yun He, Hoang Phan, Hangoo Kang, Azalia Mirhoseini, Sharon Li

分类: cs.AI, cs.LG

发布日期: 2026-10-01

💡 一句话要点

提出Sharpening Tax以量化后训练对解决方案覆盖率的影响

🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture) 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 后训练 大型语言模型 强化学习 解决方案覆盖率 动态采样 代理任务 量化指标

📋 核心要点

  1. 现有的后训练方法在提升单次任务准确性时，往往会导致解决方案覆盖率的下降，尤其在复杂的代理任务中更为明显。
  2. 论文提出Sharpening Tax这一新指标，以量化后训练对模型解决方案覆盖率的影响，并引入后验温度分组采样（PTGS）来优化采样过程。
  3. 实验结果表明，PTGS在多个代理环境中表现优于固定温度基线，解决更多任务的同时提升了单次准确性，验证了Sharpening Tax的有效性。

📝 摘要（中文）

关于大型语言模型（LLMs）后训练的一个新兴假设是，它仅仅是对基础模型现有行为的锐化，提升单次准确性但牺牲了解决方案覆盖率。尽管这一权衡在数学和编码任务中已被观察到，但在需要多轮工具使用和交互的代理任务中并不一定适用。研究发现，预训练的LLMs在轻量推理框架下能够作为有效的代理，尽管准确率较低，但在解决方案覆盖率上往往优于后训练模型。论文提出了Sharpening Tax这一诊断指标，量化后训练带来的测试时间可扩展性损失，并通过实验验证了其在多种设置中的普遍性和有效性。

🔬 方法详解

问题定义：论文旨在解决后训练对大型语言模型在代理任务中解决方案覆盖率的负面影响，现有方法在提升准确性时常导致覆盖率下降。

核心思路：提出Sharpening Tax作为量化指标，评估后训练对模型可扩展性的影响，同时引入PTGS，通过动态调整采样温度来优化任务解决能力。

技术框架：整体框架包括基础模型的预训练、后训练阶段的性能评估，以及PTGS的实施。主要模块包括模型评估、Sharpening Tax计算和PTGS采样策略。

关键创新：Sharpening Tax是首个量化后训练影响的指标，揭示了后训练导致的任务解决极化现象，PTGS则通过动态调整采样温度来提高任务解决率。

关键设计：PTGS的设计包括对每个提示的难度估计，动态调整采样温度以适应不同任务的复杂性，确保在多次采样中提升解决方案覆盖率。实验中使用了14对基础/后训练模型和3个代理基准进行验证。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

实验结果显示，PTGS在两个代理环境中相比固定温度基线减少了Sharpening Tax，解决了更多任务，同时单次准确性也有所提升。具体而言，PTGS在多次采样中表现出更高的任务解决率，验证了其有效性。

🎯 应用场景

该研究的潜在应用领域包括智能代理、自动化编程和复杂决策系统等。通过优化后训练过程和采样策略，可以在实际应用中提升模型的解决方案覆盖率和准确性，推动智能系统在多轮交互中的表现。

📄 摘要（原文）

An emerging hypothesis about reinforcement learning (RL) post-training of large language models (LLMs) is that it merely sharpens existing behaviors of a base model, improving single-shot accuracy at the cost of solution coverage. Although this trade-off has been observed in math and coding tasks, it need not extend to agentic tasks, where multi-turn tool use and interaction may require capabilities newly acquired during post-training. Our surprising finding is that pre-trained LLMs, equipped with a light inference harness, can serve as capable agents. Despite far lower accuracy (pass@1), they often surpass their post-trained counterparts in solution coverage (pass@K) given a sufficient test-time budget. We further analyze the underlying mechanism and show that post-training pushes tasks toward two extremes, always solved or never solved, and thereby improves sampling efficiency and consistency at the cost of solution coverage. To measure this cost, we propose Sharpening Tax, a diagnostic metric that quantifies the loss in test-time scalability after post-training. Across 14 base/post-trained model pairs from four families and three agentic benchmarks (42 cases in total), the tax is prevalent in most settings, can be estimated from a few rollouts, and correlates well with other metrics. Finally, we present posterior-tempered group sampling (PTGS), a simple plug-and-play Bayesian sampler that adapts the sampling temperature per prompt to its estimated difficulty. Applied during RL training in two agentic environments, PTGS pays a smaller tax than the fixed-temperature baseline, solving more tasks under repeated sampling while also improving single-shot accuracy.