Semifactual Credit-Augmented Policy Optimization

📄 arXiv: 2609.40360v1 📥 PDF

作者: Junshu Pan, Zhizhang Fu, Shulin Huang, Yiran Ding, Zifan Cheng, Wenqi Shao, Qiaosheng Zhang, Yue Zhang

分类: cs.LG, cs.AI, cs.CL

发布日期: 2026-09-30

🔗 代码/项目: GITHUB

💡 一句话要点

提出半事实信用增强策略优化以改善推理能力

🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture) 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 强化学习 语言模型 推理能力 信用分配 半事实干预 模型优化 数学基准 分布外评估

📋 核心要点

  1. 现有的Group Relative Policy Optimization（GRPO）方法在处理任务相关性时存在不足，可能导致模型对无关特征的过度依赖。
  2. 本文提出的半事实信用增强策略优化（SCAPO）通过引入半事实稳定性，优化了标记级的信用分配，从而改善推理能力。
  3. 实验结果表明，SCAPO在多个数学基准和分布外基准上均优于GRPO，显示出显著的性能提升。

📝 摘要（中文）

强化学习与可验证奖励（RLVR）提升了大型语言模型（LLMs）的推理能力，但其预测仍对与任务无关的提示特征敏感。本文通过半事实提示干预研究这种敏感性，发现抑制高漂移的候选标记可在解码过程中提高推理准确性，而无需更新模型权重。基于此，提出了半事实信用增强策略优化（SCAPO），该方法在标记级信用分配中引入半事实稳定性，显著提高了模型在数学基准和分布外基准上的表现。SCAPO在Qwen3-4B-Base和Qwen3-1.7B-Base上分别提高了5.63和4.17个百分点，展示了其在推理和泛化方面的有效性。

🔬 方法详解

问题定义：本文旨在解决现有GRPO方法在标记级信用分配中对无关特征的敏感性问题，导致推理能力不足。

核心思路：SCAPO通过引入半事实稳定性，调整标记的信用分配，抑制高漂移标记的优势，从而提高推理准确性。

技术框架：SCAPO的整体架构包括半事实干预、标记概率漂移测量和稳定性评分的归一化，分为训练阶段和评估阶段。

关键创新：SCAPO的核心创新在于结合半事实稳定性进行标记级信用分配，区别于GRPO对所有标记赋予相同的优势，避免了潜在的虚假依赖。

关键设计：在训练过程中，SCAPO对不稳定标记减少优势分配，同时不单独给予稳定性额外信用，确保模型在早期训练阶段的有效性。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

实验结果显示，SCAPO在Qwen3-4B-Base和Qwen3-1.7B-Base模型上分别提高了5.63和4.17个百分点的准确率，且在大多数数学基准和所有分布外基准中表现最佳，显著优于其他比较方法。

🎯 应用场景

该研究的潜在应用领域包括自然语言处理、智能问答系统和教育技术等。通过改善模型的推理能力，SCAPO可以在更复杂的任务中提供更准确的结果，具有重要的实际价值和未来影响。

📄 摘要（原文）

Reinforcement learning with verifiable rewards (RLVR) has improved the reasoning capabilities of large language models (LLMs), yet their predictions remain sensitive to task-irrelevant prompt features. We investigate this sensitivity through semifactual prompt interventions that preserve the underlying problem and its answer. Our analysis reveals substantial variation in token-level sensitivity and shows that suppressing high-drift token candidates during decoding improves reasoning accuracy without updating model weights. These findings highlight a limitation of Group Relative Policy Optimization (GRPO), which assigns the same outcome-derived advantage to every response token and may reinforce potential spurious dependence alongside useful reasoning. Motivated by this observation, we introduce Semifactual Credit-Augmented Policy Optimization (SCAPO), a causally inspired variant of GRPO that incorporates semifactual stability into token-level credit assignment. SCAPO measures token probability drift for fixed responses under semifactual interventions and uses normalized stability scores to reduce advantages for relatively unstable tokens during early training, while granting no additional credit for stability alone. On Qwen3-4B-Base and Qwen3-1.7B-Base, SCAPO improves AIME 2024-2026 accuracy over GRPO by 5.63 and 4.17 percentage points, respectively. At both model scales, SCAPO achieves the best results on most evaluated mathematics benchmarks and all evaluated out-of-distribution benchmarks among the compared methods. These results suggest that semifactual stability provides an effective training signal for improving reasoning and generalization through finer-grained credit assignment in RLVR. The code is available at https://github.com/DtYXs/SCAPO.