RISED: RubrIcs for agentic multi-environment Selection and sElf-Distillation
作者: Jingtan Wang, Sirajul Salekin, Young mok Jung, Javier Movellan, Bryan Kian Hsiang Low, Manjot Bilkhu
分类: cs.AI
发布日期: 2026-10-01
💡 一句话要点
提出RISED以解决多环境强化学习中的数据选择问题
🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture) 支柱九：具身大模型 (Embodied Foundation Models)
关键词: 多环境强化学习 大规模语言模型 自蒸馏 评分标准 数据选择 策略监督 行为分析
📋 核心要点
- 现有多环境强化学习方法在数据选择上存在局限，未能有效利用环境间的关系和奖励信号。
- RISED框架通过使用评分标准来指导数据选择和策略监督，提升了模型的学习效率和效果。
- 实验结果显示，RISED在不同模型骨干上均取得了显著的性能提升，平均通过率最高，且在各个环境中表现优异。
📝 摘要（中文）
在多环境强化学习中，训练单一大规模语言模型（LLM）代理以适应多样的交互环境逐渐受到关注。现有的课程和数据选择策略往往在环境层面进行训练分配，或优先考虑局部奖励信号，而未明确考虑不同环境间的关系。为此，本文提出RISED框架，通过使用描述性评分标准（rubrics）来指导在线数据选择和策略监督，从而提升模型在多环境中的表现。实验结果表明，RISED在各个环境中均取得了最高的平均通过率，并在所有个体环境中排名第一或第二。
🔬 方法详解
问题定义：本文旨在解决多环境强化学习中数据选择的不足，现有方法未能有效利用环境间的关系和奖励信号，导致训练效率低下。
核心思路：RISED框架通过引入描述性评分标准（rubrics），不仅作为奖励信号，还用于指导数据选择和策略监督，从而提升模型的学习效果。
技术框架：RISED的整体架构包括多个模块：首先，LLM评估每个回合的表现并使用预定义的评分标准进行标记；其次，基于评分结果选择与整体行为组成相符的数据；最后，正负评分标准分别用于引导自蒸馏和避免失败模式。
关键创新：RISED的主要创新在于将评分标准用于数据选择和策略监督，突破了传统方法仅依赖标量奖励的局限，提供了更丰富的反馈信息。
关键设计：在设计上，使用了共享的评分词汇来标记回合，正向评分标准为自蒸馏教师提供了额外的标记级监督，而负向评分标准则帮助生成更优的回合，避免重复失败。
🖼️ 关键图片
📊 实验亮点
RISED在不同模型骨干上实现了最高的平均通过率，并在每个个体环境中均排名第一或第二，显示出其在多环境强化学习中的显著优势。通过评分标准的分析，进一步揭示了模型行为变化的特征。
🎯 应用场景
RISED框架在多环境强化学习中具有广泛的应用潜力，尤其适用于需要在复杂环境中进行决策的智能代理系统，如机器人控制、游戏AI和自动驾驶等领域。其创新的评分标准方法可以为未来的多任务学习和自适应学习提供新的思路和技术支持。
📄 摘要（原文）
Training a single LLM agent jointly across diverse interactive environments has attracted increasing attention as a route to generalist agents. Existing curriculum and data-selection strategies often allocate training at the environment level or prioritize local reward-based signals, without explicitly considering relationships between current rollouts across environments for prompt-group selection. Meanwhile, as environments are learned at different rates, all-failure and all-success rollout groups can coexist within a batch, leaving those data without group-relative reward signals. Both challenges highlight limitations of relying solely on scalar rewards in multi-environment RL: they provide limited information about cross-environment relationships and no within-group reward contrast when rewards are identical. This motivates richer textual feedback, such as rubrics describing rollout behaviours, to guide learning. Beyond rubrics' usage as reward, we repurpose rubrics to guide both online data selection and policy supervision. An LLM judge tags each rollout using a predefined rubric vocabulary shared across environments. The resulting profiles guide the selection of data that aligns with the overall behavioural composition of the mixed-environment batch while limiting overlap with already-selected data. Available positive rubrics (describing desired behaviours) provide privileged context for an on-policy self-distillation teacher, supplying additional token-level supervision, while negative rubrics (describing undesired behaviours) guide subsequent rollout generation away from recurring failure modes. Together, these components form RISED. Across model backbones, RISED achieves the highest mean pass rate across environments and ranks first or second in every individual environment. Rubric-based analysis of RISED can further characterize the behavioural changes accompanying these gains.