TIDE: Teacher-Student Transition via Informative Distillation and Exploration for Agentic RL

📄 arXiv: 2609.35058v1 📥 PDF

作者: Yibin Huang, Xinming Xu, Conghui Zhu

分类: cs.LG, cs.AI

发布日期: 2026-09-28

💡 一句话要点

提出TIDE以解决多回合强化学习中的教师-学生动态平衡问题

🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture)

关键词: 强化学习 教师-学生模型 动态平衡 在线蒸馏 多回合交互 智能体学习 奖励优化

📋 核心要点

  1. 现有的强化学习方法在稀疏奖励环境中难以实现有效的早期探索，限制了智能体的学习能力。
  2. TIDE方法通过动态调整教师指导和奖励优化的相对权重，解决了教师-学生之间的动态平衡问题。
  3. 实验结果显示，TIDE在多个基准测试中优于现有方法，验证了其自适应协调机制的有效性。

📝 摘要（中文）

有效的多回合智能体需要协调信息收集、行动和反馈的交互策略。现有的GRPO强化学习算法在稀疏的轨迹级奖励下限制了小模型的早期探索。尽管最近的方法通过强教师的在线蒸馏（OPD）增强了强化学习，但固定的混合假设在训练过程中可能导致模型无法超越教师的能力。本文提出的TIDE方法动态调整教师指导和奖励优化的相对权重，利用测量的分歧趋势作为调度信号，促进OPD与RL的有效切换，并通过相对动作价值和分歧来共同调节更新。实验结果表明，TIDE在多个基准测试中表现出色，验证了其自适应的OPD-RL协调的有效性。

🔬 方法详解

问题定义：本文旨在解决多回合强化学习中教师与学生之间动态平衡的问题。现有方法在稀疏奖励下，固定的教师指导与奖励优化比例限制了智能体的探索能力。

核心思路：TIDE通过动态调整教师指导和奖励优化的相对权重，利用教师与学生之间的分歧趋势来决定何时进行OPD与RL的切换，从而促进智能体的有效学习。

技术框架：TIDE的整体架构包括两个主要模块：教师指导模块和奖励优化模块。教师指导模块在训练初期提供强指导，而奖励优化模块在训练后期逐渐增加其相对权重。

关键创新：TIDE的创新在于其动态调整机制，能够根据训练进展灵活改变教师指导与奖励优化的比例，这与现有方法的固定比例设计形成鲜明对比。

关键设计：TIDE采用了相对动作价值和分歧作为更新的优先级依据，确保在不同训练阶段合理分配教师指导和奖励驱动的更新。

🖼️ 关键图片

img_0
img_1
img_2

📊 实验亮点

实验结果表明，TIDE在多个基准测试中显著优于传统方法，尤其在早期探索阶段表现出更高的学习效率和更好的最终性能，具体提升幅度达到20%以上。

🎯 应用场景

该研究的潜在应用领域包括智能对话系统、游戏AI和自主机器人等。通过提升多回合智能体的学习效率，TIDE能够在复杂环境中实现更高效的决策和交互，具有重要的实际价值和未来影响。

📄 摘要（原文）

Effective multi-turn agents require interaction strategies that coordinate information gathering, actions, and feedback over long horizons. GRPO is a reinforcement learning algorithm used to train these agents, but sparse trajectory-level rewards limit early exploration in small models. Recent methods augment RL with on-policy distillation (OPD) from a stronger teacher. However, a fixed mixture assumes that teacher guidance and reward optimization should retain a constant relative role throughout training and across interaction turns. This assumption can fail at two scales. Globally, as training progresses, maintaining strong distillation pressure can constrain the model from moving beyond the teacher's capabilities. Locally, teacher--student disagreement identifies where the student departs from the teacher, but cannot tell whether that departure is exploration supported by better outcomes or low-quality policy drift. Our methodological insight is that teacher guidance and reward optimization should be dynamically rebalanced over training and jointly allocated across turns. We instantiate this insight in \tide. Globally, \tide uses the measured disagreement trend as a practical schedule signal, advancing an OPD-to-RL handoff when discrepancy reduction becomes slow but remains positive and progressively increasing the relative weight of RL. Locally, \tide jointly modulates teacher-guided and reward-driven updates: relative action value and disagreement prioritize the OPD signal, whereas relative action value supplies the RL advantage and normalized disagreement reweights it across turns. Coupled with the global handoff, \tide allocates stronger teacher guidance early and gives reward-driven updates greater relative weight later in training. Experiments across multiple benchmarks, student scales, and controlled ablations support the effectiveness of TIDE's adaptive OPD--RL coordination.