State-OPD: State-Grounded On-Policy Distillation for Multi-Turn Agents
作者: Zhentao Zhang, Yanzhao Zheng, Yanlin Liu, Yuanqiang Yu, Tianze Xu, Chao Ma, Jihuai Zhu, Lelun Wu, Baohua Dong, Hangcheng Zhu, Ruohui Huang, Gang Yu
分类: cs.LG
发布日期: 2026-10-09
💡 一句话要点
提出State-OPD以解决多回合智能体的蒸馏训练问题
🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture)
关键词: 多回合智能体 蒸馏训练 推理能力 教师监督 交互状态 动作选择 响应多样性 状态基础
📋 核心要点
- 现有的On-Policy Distillation方法在多回合智能体环境中应用时面临挑战，特别是在推理能力的转移和动作选择的多样性方面。
- 本文提出State-Grounded On-Policy Distillation框架，通过交互状态来优化教师监督，增强学生推理的有效性和多样性。
- 实验结果表明，State-OPD在ALFWorld、ScienceWorld和WebShop等环境中显著提高了成功率，尤其在Qwen3-4B模型中成功率提升至31.74%。
📝 摘要（中文）
On-Policy Distillation (OPD)作为一种强大的能力转移范式，面临在多回合智能体环境中应用的挑战。本文通过交互状态研究智能体OPD，区分目标任务与执行的动作-观察记录与学生推理的内部语义轨迹。我们识别出学生推理在多回合OPD中的两个角色：先前推理扭曲教师监督，而当前推理塑造在同一状态下采样的动作选择。我们提出State-Grounded On-Policy Distillation框架，将教师监督与交互状态结合，并在高响应熵的状态下重新分配采样预算，以生成多个完整的推理-动作响应，同时保留完整的历史学生生成。综合评估显示，State-OPD在多个环境中取得了最高的宏观平均成功率。
🔬 方法详解
问题定义：本文旨在解决在多回合智能体环境中，现有On-Policy Distillation方法在推理能力转移和动作选择多样性方面的不足。现有方法往往无法有效利用学生推理的历史信息，导致教师监督效果不佳。
核心思路：论文提出的State-Grounded On-Policy Distillation框架，通过将教师监督与交互状态相结合，优化了学生推理的过程，允许在高响应熵的状态下生成多样化的推理-动作响应。
技术框架：该框架包括两个主要模块：一是通过交互状态来识别和优化教师监督，二是重新分配采样预算以生成多个完整的推理-动作响应，同时保留学生的完整历史生成。
关键创新：最重要的创新在于将教师监督与交互状态相结合，识别学生推理的两个角色，从而有效提升了教师的监督效果和学生的推理能力。与现有方法相比，State-OPD在处理多回合交互时展现出更高的灵活性和适应性。
关键设计：在设计中，关键参数包括响应熵的计算方式和采样预算的分配策略，损失函数则考虑了教师的目标与学生的推理轨迹之间的关系，以优化最终的推理-动作输出。整体网络结构则采用了适应性调整机制，以增强模型的学习能力。
🖼️ 关键图片
📊 实验亮点
实验结果显示，State-OPD在Qwen3-4B模型上将宏观平均成功率从26.07%提升至31.74%，超越了Qwen3-32B教师在ALFWorld Seen环境中的表现。此外，State-OPD显著减少了持续失败动作的重复，提升了任务成功率。
🎯 应用场景
该研究的潜在应用领域包括智能对话系统、游戏AI和人机交互等场景。通过提升多回合智能体的推理能力和响应多样性，State-OPD能够显著改善用户体验，推动智能体在复杂环境中的应用。未来，该方法可能在更广泛的智能体学习和决策领域产生深远影响。
📄 摘要（原文）
On-Policy Distillation (OPD) has emerged as a powerful paradigm for transferring reasoning capabilities from frontier models to smaller models, while extending its success to multi-turn agentic environments remains challenging. We study agentic OPD through interaction states, distinguishing the objective task and executed action-observation record from the internal semantic trajectory of student reasoning. Through this lens, we identify two roles of student reasoning in multi-turn OPD: prior reasoning distorts teacher supervision, while current reasoning shapes the action alternatives sampled at the same state. Specifically, at matched interaction states, removing prior student reasoning improves teacher continuation success and changes the teacher's token-level targets, with larger continuation gains at later turns. Meanwhile, resampling complete reasoning-action responses yields greater action diversity at the same state than resampling actions under a fixed reasoning trace. We introduce State-Grounded On-Policy Distillation, a framework that grounds teacher supervision in interaction states and reallocates the sampling budget to generate multiple complete reasoning-action responses at states with high response entropy, while retaining full-history student generation. Comprehensive evaluations across ALFWorld, ScienceWorld and WebShop demonstrate that State-OPD achieves the highest macro-average success rate among the evaluated methods for both Qwen3-1.7B and Qwen3-4B students. Notably, for Qwen3-4B, State-OPD raises macro-average success from 26.07\% with the strongest baseline to 31.74\% and surpasses its Qwen3-32B teacher on ALFWorld Seen. Further analyses reveal that State-OPD achieves higher task success with fewer supervised turn responses and substantially reduces persistent failed-action repetition; qualitative traces illustrate decision revision based on interaction feedback.