ProCredit: From Outcome Rewards to Progress Credit in Agentic Reinforcement Learning
作者: Ming Ma, Yi Zhu, Yiran Zhong, Feida Zhu, Chonghan Liu, Pengkun Jiao, Qichao Wang, Yanhao Jia, Tianming Yang, Steven Hoi
分类: cs.LG, cs.CL
发布日期: 2026-09-23
💡 一句话要点
提出ProCredit以解决长时间代理任务的奖励信号不足问题
🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture)
关键词: 长时间代理任务 奖励信号 进展信用 智能体学习 Qwen3.5模型 任务完成率 环境状态评估
📋 核心要点
- 现有方法在长时间代理任务中仅依赖最终结果奖励，导致无成功轨迹的组别无法提供有效训练信号。
- ProCredit通过在每一步后重新评估进展，将中间状态的进展转化为奖励，增强了训练信号的有效性。
- 实验结果显示，ProCredit在不同规模的模型上均优于传统的结果奖励基线，提升了任务完成率。
📝 摘要（中文）
长时间代理任务要求代理通过一系列工具调用修改环境，成功由最终状态决定。现有方法在任务结束时仅分配单一结果奖励，导致无成功轨迹的组别无法提供训练信号，失败尝试无法区分接近完成的程度。本文提出ProCredit，通过在每一步后重新运行接受检查，将验证的进展转化为奖励，从而为每一步的进展赋予信用。实验表明，ProCredit在任务完成率上超越了基于结果和进展的基线，尤其在4B模型上超出最强结果基线4.1个百分点。
🔬 方法详解
问题定义：本文旨在解决长时间代理任务中，现有方法仅依赖最终结果奖励的问题。这种方法导致无成功轨迹的组别无法提供有效的训练信号，且无法区分失败尝试的接近程度。
核心思路：ProCredit的核心思路是将中间状态的进展转化为奖励，通过在每一步后重新运行接受检查，给予每一步的进展以信用。这种设计使得进展的验证与最终结果同样重要。
技术框架：ProCredit的整体架构包括三个主要模块：环境状态评估模块、进展奖励计算模块和信用分配模块。首先评估当前状态，然后计算进展奖励，最后将奖励分配到每一步和每个尝试中。
关键创新：ProCredit的关键创新在于将进展信用化，允许在每一步中对进展进行奖励，而不仅仅依赖于最终结果。这与现有方法的本质区别在于，现有方法通常只关注最终成功。
关键设计：在参数设置上，ProCredit使用了动态调整的奖励机制，损失函数设计为考虑进展的变化，网络结构则基于Qwen3.5模型进行了优化，以适应不同规模的任务。实验中还进行了消融实验，验证了进展信用的重要性。
🖼️ 关键图片
📊 实验亮点
实验结果表明，ProCredit在不同规模的模型上均优于传统的结果奖励基线，尤其在4B模型上超出最强结果基线4.1个百分点。此外，在第二个环境中也显示出相同的改进方向，验证了方法的有效性。
🎯 应用场景
ProCredit的研究成果在多个领域具有潜在应用价值，尤其是在需要长时间决策的任务中，如机器人控制、自动驾驶和复杂游戏AI等。通过提供更细粒度的奖励信号，ProCredit能够加速学习过程，提高任务完成率，推动智能体在复杂环境中的表现。
📄 摘要（原文）
Long-horizon agentic tasks require an agent to modify an environment through a sequence of tool calls, with success determined by the final state. The standard recipe assigns a single outcome reward at the end and compares trajectories sampled for the same task. As a result, a group with no successful trajectory yields no training signal, failed attempts cannot be told apart by how close they came to completion, and turns that advance the task receive the same credit as turns that only query the environment. Prior work refines the unit of comparison from the trajectory to the step, or trains a reward model to supply intermediate signal: the former still derives its signal from final success alone, and the latter estimates it with a model. We observe that the acceptance checks that decide success can also be run on intermediate states, so progress is as verifiable as the outcome. We propose ProCredit, which turns this verified progress into credit: it reruns the acceptance checks after each turn, rewards the turn by its change in progress, and uses these rewards to assign credit both across attempts at the same task and across the turns within a trajectory. Starting from Qwen3.5 base models at three scales on AppWorld, ProCredit outperforms outcome-reward baselines and progress-based baselines in task completion rate at every scale on both test sets, exceeding the strongest outcome-reward baseline by 4.1 percentage points at 4B, and results in a second environment show the same direction of improvement. Ablations show that adding the final progress to the trajectory score alone does not improve performance: the gain comes from crediting progress to the turn where it occurs.