Revisiting On-policy Adversarial Black-Box Distillation: Calibrating Groupwise Reward Geometry for Effective Advantage Construction

📄 arXiv: 2609.39757v1 📥 PDF

作者: Xiao Cui, Mo Zhu, Yulei Qin, Yuze Wu, Wengang Zhou, Houqiang Li

分类: cs.LG, cs.CV

发布日期: 2026-09-30

备注: NeurIPS 2026

🔗 代码/项目: GITHUB

💡 一句话要点

提出GRGC以解决现有黑箱蒸馏中的奖励几何问题

🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture) 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 黑箱蒸馏 奖励几何 对抗训练 模型优化 强化学习

📋 核心要点

  1. 现有的黑箱蒸馏方法在奖励计算上存在目标不匹配的问题，导致优化信号脆弱。
  2. 本文提出了群体奖励几何条件化（GRGC），通过塑造奖励组来改善优势构建。
  3. 实验结果显示，GRGC在多种设置下均优于现有方法，且开销极小。

📝 摘要（中文）

黑箱蒸馏是一种将大型语言模型能力转移到小型学生模型的有效方法。近期的对策方法如GAD通过在评论员和学生之间形成对抗循环来优化学生策略。然而，现有方法在奖励计算上存在目标不匹配的问题，导致奖励组的规模崩溃或边际脆弱。为此，本文提出了群体奖励几何条件化（GRGC），通过在评论员训练和策略优化中塑造学生侧奖励组，改善优势构建。通过高斯群体最优传输校准，增强评论员训练的条件性，确保奖励组的分布不崩溃且平滑。实验表明，GRGC在多种教师、学生模型和数据集上均表现出色，且对GAD的开销微乎其微。

🔬 方法详解

问题定义：本文旨在解决现有黑箱蒸馏方法中奖励计算的目标不匹配问题，导致奖励组规模崩溃和优化信号脆弱。

核心思路：通过群体奖励几何条件化（GRGC），在评论员训练和策略优化中塑造学生侧奖励组，以改善优势构建的效果。

技术框架：GRGC包括两个主要阶段：首先，通过高斯群体最优传输校准来正则化评论员，确保奖励组的分布不崩溃；其次，在策略优化阶段，重塑奖励组以提高优化相关的边际可分性。

关键创新：最重要的创新在于引入高斯群体最优传输校准，确保奖励组的平滑分布和非崩溃状态，这与现有方法的目标不匹配形成鲜明对比。

关键设计：在评论员训练中，采用排序的提示奖励与群体中心高斯分位数匹配的方式进行正则化；在策略优化中，重塑奖励组以保持评论员引导的排序，同时增强优化相关的边际可分性。

🖼️ 关键图片

fig_0
fig_1

📊 实验亮点

实验结果表明，GRGC在多种教师和学生模型上均显著提升了性能。在分布内和分布外评估中，GRGC相较于GAD方法的性能提升幅度达到了X%（具体数据待补充），且引入的开销几乎可以忽略不计。

🎯 应用场景

该研究的潜在应用领域包括自然语言处理中的模型蒸馏、对抗训练和强化学习等。通过提高小型模型的性能，GRGC可以在资源受限的环境中实现更高效的模型部署，具有重要的实际价值和广泛的未来影响。

📄 摘要（原文）

Black-box distillation is a practical route for transferring capabilities from API-accessible large language models that expose only text outputs into smaller student models. Recent on-policy adversarial methods such as GAD improve over SeqKD by forming an adversarial loop between a critic and a student, where the critic provides rewards for GRPO-based student policy optimization over the student's sampled responses. However, GRPO computes advantages from the within-group relative rewards of student samples for the same prompt, whereas the critic is trained primarily to distinguish teacher responses from student responses. This objective mismatch can produce reward groups with collapsed scale or fragile margins, leading to brittle grouped optimization signals. We propose Groupwise Reward Geometry Conditioning (GRGC), a two-stage framework that improves advantage construction by shaping student-side reward groups during both critic training and policy optimization. To improve critic-side conditioning, Gaussian groupwise Optimal Transport calibration regularizes the critic during training to produce reward groups with non-collapsed spread and smooth rank-wise gaps by matching sorted prompt-wise rewards to group-centered Gaussian quantiles. Building on this conditioned reward geometry, policy-side group power modulation reshapes the prompt-wise reward groups before they are converted into advantages, preserving the critic-induced ordering while increasing optimization-relevant margin separability. Extensive experiments across diverse teachers, student model families and scales, and training datasets demonstrate the effectiveness of GRGC on both in-distribution and out-of-distribution evaluations, while introducing negligible overhead over GAD. The code is available at https://github.com/2018cx/GRGC.