Probe with Participation Trophies: Random-Reward RL as a Probe of LLM Capability

📄 arXiv: 2610.01066v1 📥 PDF

作者: Yu Mao, Lei Yu, Zining Zhu, Yusheng Zheng, Haohang Li, Freda Shi, Yutong Yin, Zhaoran Wang, Jingcheng Niu

分类: cs.CL

发布日期: 2026-10-01

💡 一句话要点

提出随机奖励强化学习以探测大语言模型能力

🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture) 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 随机奖励 强化学习 大语言模型 模型探测 训练可达性 数据污染 性能提升

📋 核心要点

  1. 现有方法未能清晰揭示探测性能与模型能力之间的关系，导致对模型能力的理解存在争议。
  2. 论文提出随机奖励强化学习作为探测工具，旨在评估模型在特定约束下的可达性，而不仅仅依赖于当前性能。
  3. 实验结果显示，随机奖励的RL在不同训练阶段对模型性能的提升效果显著，尤其在中期训练阶段表现突出。

📝 摘要（中文）

本文将虚假奖励悖论与模型的可达性联系起来，提出随机奖励强化学习（RL）作为探测工具，解决了关于探测性能实际揭示模型能力的长期争论。研究表明，随机奖励的RL可以探测模型在特定约束下从当前状态所能达到的进一步训练效果。通过对OLMo检查点的分析，发现不同训练阶段的响应模式，随机奖励在中期训练中能够显著提升模型性能，提供了对训练潜力的新视角。

🔬 方法详解

问题定义：本文旨在解决探测性能与模型能力之间的关系不明确的问题，现有方法在揭示模型能力时存在数据污染和机制解释的争议。

核心思路：论文提出随机奖励强化学习（RL）作为探测工具，探讨模型在无正确反馈情况下的可达性，强调训练潜力的评估。

技术框架：研究通过对OLMo检查点的分析，分为预训练早期、预训练后期和中期训练三个阶段，观察不同奖励机制对模型性能的影响。

关键创新：最重要的创新在于将随机奖励的RL视为探测模型能力的新方法，突破了传统依赖正确反馈的限制，提供了新的视角。

关键设计：在实验中，采用了不同的奖励机制和训练阶段设置，分析了模型在相同准确率下的不同表现，揭示了训练响应的多样性。

🖼️ 关键图片

fig_0
img_1
img_2

📊 实验亮点

实验结果表明，在相同的准确率下，OLMo检查点在随机奖励的RL下的最佳表现分别达到了8.5%和55%的提升，显示出随机奖励在中期训练阶段的显著效果，突破了传统训练方法的局限。

🎯 应用场景

该研究为大语言模型的训练和评估提供了新的方法论，特别是在无标签或不完全标签的情况下，能够有效探测模型的潜在能力。未来可应用于自然语言处理、对话系统等领域，推动模型的优化与应用。

📄 摘要（原文）

We connect the spurious-reward paradox to a model's reachability and propose random-reward reinforcement learning (RL) as a useful tool for the probing enterprise, addressing a decade-long debate over what probing performance actually reveals about a model. There are two prevailing explanations for the surprising finding that even random rewards can improve the performance of large language models (LLMs): one attributes the gains to particular mechanisms within RL training; the other to data contamination. Our results motivate a different view: spurious-reward RL can probe a model's reachability, or what further training can attain from its current state under specified constraints, beyond what is reflected in its current performance. Two OLMo checkpoints with the same accuracy on synthetic arithmetic (3.5%), for example, reach 8.5% and 55% in their best runs under the same correctness-rewarded RL. Examining OLMo checkpoints across pre-training and mid-training reveals three distinct regimes of training response: early on, RL produces little improvement even when correct answers are rewarded; later in pre-training, rewarding correct answers becomes effective while random rewards remain weak; and, upon entering mid-training, even random rewards can produce large gains. A similar ordering appears in a number-masked supervised fine-tuning (SFT) analysis of these checkpoints, suggesting that the pattern is not specific to a particular RL mechanism. Moreover, RL with random rewards offers a distinctive perspective on what training can make an LLM do, since its reward signal supplies no information about which answers are correct. By asking what training can attain without correctness feedback, it addresses the label-leakage side of a central problem in decodability-based probing: whether a successful probe reveals the model's capabilities or learns the task itself.