Enhancing Rubric-based RL via Self-Distillation

📄 arXiv: 2607.18082v1 📥 PDF

作者: Mingxuan Xia, Yuhang Yang, Chao Ye, Shuai Zhu, Shenzhi Yang, Guangcheng Zhu, Yuhang Zhang, Cheng Peng, Haobo Wang, Siqing Wang

分类: cs.LG, cs.AI

发布日期: 2026-07-20

💡 一句话要点

提出CriPO以解决Rubric-based RL中的探索不足问题

🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture)

关键词: 强化学习 自蒸馏 标准优化 模型训练 开放式任务

📋 核心要点

  1. 现有的Rubric-based RL方法在探索方面存在不足，未满足的标准（UC）缺乏优化信号，导致性能受限。
  2. 本文提出CriPO，通过在政策优化中引入自蒸馏机制，解决UC和被抑制标准（SC）的问题，提升模型性能。
  3. 实验结果显示，CriPO在多个基准测试中表现优于传统Rubric-based RL，优化步骤减少约2倍，最终性能显著提升。

📝 摘要（中文）

Rubric-based RL在开放式任务中展现出良好潜力，但其局限在于探索不足，导致未被满足的标准（UC）缺乏优化信号。现有方法通过外部指导来改善探索，但在推理时缺乏一致性，导致错误累积。此外，论文提出的被抑制标准（SC）问题也未得到有效解决。为此，本文提出了Criterion-Distilled Policy Optimization（CriPO），通过在政策优化中引入自蒸馏机制，解决UC和SC问题。实验结果表明，CriPO在医学和科学基准测试中表现优异，优化步骤减少约2倍。

🔬 方法详解

问题定义：本文旨在解决Rubric-based RL中的探索不足问题，尤其是未被满足的标准（UC）和被抑制的标准（SC）所带来的性能瓶颈。现有方法在推理时缺乏一致性，导致错误累积，且未能有效利用某些满足标准的回滚信息。

核心思路：CriPO通过引入自蒸馏机制，构建自教师来优化政策，既解决了UC问题，又保留了SC的有用信息。具体而言，CriPO通过局部前向KL损失来注入缺失行为，并利用反事实自教师来调整负优势回滚中的标准相关标记。

技术框架：CriPO的整体架构包括两个主要模块：标准注入自教师和反事实自教师。前者负责处理UC，通过引导政策学习缺失的行为；后者则定位并调整SC中被抑制的标记，确保有用模式得以保留。

关键创新：CriPO的核心创新在于同时解决UC和SC问题，而不引入训练与推理的不匹配。通过自蒸馏机制，CriPO能够有效利用回滚信息，提升政策的学习效率。

关键设计：在损失函数设计上，CriPO采用局部前向KL损失来实现标准注入，并通过调整标记的优势值来优化SC。此外，模型的参数设置和网络结构经过精心设计，以确保高效的学习和优化过程。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

实验结果表明，CriPO在医学和科学基准测试中表现优于传统的Rubric-based RL方法，最终性能显著提升，优化步骤减少约2倍，展示了其在提升模型效率和效果方面的潜力。

🎯 应用场景

该研究的潜在应用领域包括医疗、科学研究等开放式任务场景，能够有效提升大规模语言模型在复杂任务中的表现。未来，CriPO的自蒸馏机制可能在其他强化学习和生成模型中得到广泛应用，推动相关领域的发展。

📄 摘要（原文）

Rubric-based RL has recently shown promise in improving LLMs on open-ended tasks. A widely recognized limitation of rubric-based RL is limited exploration: criteria that no rollout manages to satisfy (Unexplored Criteria, UC) receive no optimization signal. Recent methods address this by incorporating rubric information as external guidance during rollout, yet they introduce a train-inference mismatch: the policy is optimized on rollouts produced under external guidance while this guidance is absent at inference time, causing error accumulation through autoregressive decoding. Moreover, these exploration-focused approaches overlook a fundamentally different failure mode that we term Suppressed Criteria (SC) -- criteria that are satisfied by some rollouts yet whose learning signals are lost during optimization because scalar reward aggregation assigns them non-positive aggregate advantages. Our analysis reveals that SC are remarkably prevalent: over 57% of samples exhibit this failure mode throughout training, with an average of 1.8 SC per sample. To simultaneously address both UC and SC without introducing training-inference mismatch, we propose Criterion-Distilled Policy Optimization (CriPO), which enhances rubric-based RL via on-policy self-distillation. For UC, CriPO constructs a criterion-injection self-teacher and computes a localized forward-KL loss to inject missing behaviors into the policy. For SC, CriPO employs a counterfactual self-teacher to locate criterion-relevant tokens in negative-advantage rollouts and flips their token-level advantages to positive values, preserving useful patterns that would otherwise be suppressed. Experiments on medicine and science benchmarks demonstrate that CriPO consistently outperforms rubric-based RL, achieving stronger final performance with approximately $2\times$ fewer optimization steps.