Capable Teacher, Myopic Scorer: Reward-Free Teacher Repair for Agentic On-Policy Distillation

📄 arXiv: 2610.28586v1 📥 PDF

作者: Shu Zhao, Tan Yu

分类: cs.CL, cs.AI

发布日期: 2026-10-20

备注: 17 pages, 5 figures, 13 tables. Project page: https://tree-shu-zhao.github.io/r2s/

🔗 代码/项目: PROJECT_PAGE

💡 一句话要点

提出R2S方法以解决Agentic OPD中的评分误差问题

🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture)

关键词: Agentic OPD 蒸馏训练 评分器修复 强化学习 智能体训练 教师接管 高分歧状态

📋 核心要点

  1. 现有Agentic OPD方法在学生代理的错误导致教师评分不可靠时，表现不佳，无法有效训练。
  2. 论文提出R2S方法，通过修复评分器，利用教师接管高分歧状态，确保评分更准确。
  3. 在三个环境中，R2S方法在所有十二个教师-学生-环境组合中均超越了三种最先进的Agentic OPD方法。

📝 摘要（中文）

Agentic on-policy distillation (OPD)训练学生代理在其自身轨迹上，而教师对每个token进行评分。现有方法因学生错误导致的复合误差而受限，无法有效训练。本文提出Rollout-to-Scorer Distillation (R2S)，通过挖掘高分歧状态并进行教师接管，修复评分器，从而提升学生的表现。实验表明，R2S在多个环境中超越了三种最先进的Agentic OPD方法，平均提升12.8个成功点。

🔬 方法详解

问题定义：论文要解决的问题是Agentic OPD中教师评分的短视性，导致学生在错误轨迹上训练，无法有效完成任务。现有方法通过限制训练区域来应对复合误差，但效果有限。

核心思路：论文的核心思路是通过R2S方法修复评分器，挖掘学生与教师意见分歧的状态，并在这些状态上进行教师接管，从而提升评分的准确性。

技术框架：R2S方法的整体架构包括三个主要阶段：首先从未训练的学生中挖掘高分歧状态；其次在这些状态上进行教师接管；最后对教师进行微调，使其在每个辅助token上给出接管信息的评分。

关键创新：最重要的技术创新点在于提出了“rollout-scorer gap”的概念，强调教师评分与实际接管行为之间的差异，并通过修复评分器来解决这一问题。

关键设计：在R2S方法中，教师在修复过程中不需要任务奖励，且在微调时保持冻结状态，确保其作为唯一评分器的稳定性。

🖼️ 关键图片

img_0
img_1
img_2

📊 实验亮点

实验结果显示，R2S方法在三个不同环境中，针对两种教师和四种学生的组合，均超越了三种最先进的Agentic OPD方法，平均提升12.8个成功点，展现出显著的性能优势。

🎯 应用场景

该研究的潜在应用领域包括自主智能体的训练、自然语言处理和机器人控制等。通过提升教师评分的准确性，R2S方法能够在多种复杂任务中提高智能体的表现，具有广泛的实际价值和未来影响。

📄 摘要（原文）

Agentic on-policy distillation (OPD) trains a student agent on its own trajectories while a teacher scores every token. A natural explanation of where it goes wrong is compounding error: the student's mistakes carry its trajectories beyond the region where the teacher is reliable, so recent methods keep training inside that region, through episode-length curricula, teacher-guided turns, prefix replay, or teacher intervention where the two disagree. We show this explanation is incomplete. From the states where the student departs most from the teacher, the teacher's own takeovers still finish the task, yet the student trained on its token scores keeps running out of turns. We attribute this to the myopia of next-token scoring. The teacher rates each token without seeing where the episode ends, so its scores let through the safe, delaying moves that outcome-based RL would punish with zero reward, a penalty agentic OPD lacks. We call this difference between what the teacher does and what it scores the rollout-scorer gap. We propose Rollout-to-Scorer Distillation (R2S), which repairs the scorer before distillation by mining high-disagreement states from the untrained student, running teacher takeovers from them, keeping the states where enough takeovers pass a reward-free check such as consensus or an LLM judge, and fine-tuning the teacher to give its takeover-informed scores on every assistant token from the divergence turn onward, without the takeover in context. The repaired teacher is frozen as the sole scorer for standard OPD, requiring no task reward during repair or training. Across three environments, two teachers and four students, R2S outperforms three state-of-the-art agentic-OPD methods in all twelve teacher-student-environment cells, by 12.8 success points on average over the best of them in each cell. Project page: https://tree-shu-zhao.github.io/r2s/.