Multi-LLM Collaborative Alignment via Stackelberg Games

📄 arXiv: 2609.39076v1 📥 PDF

作者: Christina Hahn, Shangbin Feng, Dean Light, Swastik Roy, Hila Gonen, Yulia Tsvetkov

分类: cs.AI, cs.LG

发布日期: 2026-09-30

💡 一句话要点

提出Stackelberg Alignment以解决多语言模型协作学习问题

🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture) 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 多语言模型 协作学习 游戏理论 自适应课程 模型评估 声誉机制 动态采样 机器学习

📋 核心要点

  1. 现有方法在多语言模型协作学习中，通常均匀采样指令，未能适应指令有效性的动态变化。
  2. 论文提出Stackelberg Alignment框架，通过游戏理论设计领导者-追随者机制，实现指令选择的自适应课程学习。
  3. 实验结果显示，该方法在三种不同模型池和12个基准测试中，宏平均性能提升最高达7.4%，超越了最强基线。

📝 摘要（中文）

本文提出了一种名为Stackelberg Alignment的框架，旨在通过游戏理论中的领导者-追随者模型来优化多语言模型之间的协作学习。现有方法通常均匀采样指令，而忽视了指令的有效性随着模型的提升而变化。该框架通过EXP3赌博机作为领导者，动态分配采样预算，并结合指令难度和响应可区分性来更新采样分布。语言模型作为追随者，响应选定指令并相互评估，从而通过DPO或GRPO学习偏好信号。实验表明，该方法在多个基准测试中表现优异，显著提升了模型的协作效果。

🔬 方法详解

问题定义：本文旨在解决多语言模型在协作学习中指令选择的非适应性问题，现有方法未能考虑指令有效性随模型性能变化的动态特性。

核心思路：提出Stackelberg Alignment框架，利用游戏理论中的领导者-追随者机制，将指令选择转化为自适应课程学习，以提高模型间的协作效率。

技术框架：框架主要分为两个部分：领导者部分使用EXP3赌博机动态分配指令采样预算，追随者部分则是语言模型根据选定指令进行响应和评估。

关键创新：最重要的创新在于引入Elo风格的声誉加权同行评判和基于声誉的对手匹配机制，显著提高了模型间的互动质量和学习效果。

关键设计：在参数设置上，领导者通过结合指令难度和响应可区分性来更新采样分布，追随者则通过DPO或GRPO算法进行学习，确保模型在竞争中不断进化。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

实验结果显示，Stackelberg Alignment在三种不同模型池的宏平均性能提升最高达7.4%，超越了最强训练时基线，并在静态推理基线中提升12-25%。这些结果表明，该方法在多语言模型的协作学习中具有显著的优势。

🎯 应用场景

该研究的潜在应用领域包括自然语言处理、智能对话系统和自动化内容生成等。通过优化多语言模型的协作学习，能够提升模型的整体性能和适应性，推动相关技术的进步与应用。未来，该框架可能在多模态学习和跨领域知识迁移等方面发挥重要作用。

📄 摘要（原文）

A pool of language models can collaborate and improve collectively by learning from one another's responses. These interactions depend on the instructions used during training. Existing methods typically sample instructions uniformly, even though their usefulness may change as the models improve: an instruction on which models' responses once differed in quality may later be answered equally well, while a previously difficult instruction may begin to provide a useful learning signal. We propose Stackelberg Alignment, a game-theory-inspired leader-follower framework that turns instruction selection into an adaptive curriculum. An EXP3 bandit acts as the leader, allocating a fixed sampling budget across instructions and updating its sampling distribution using a reward that combines instruction difficulty and response discriminability. The language models act as followers: they respond to the selected instructions, evaluate one another's responses, and learn from the resulting preference signals through DPO or GRPO. The framework uses Elo-style reputation-weighted peer judgment and reputation-based opponent matching to support reliable and competitive model interactions. Experiments across three heterogeneous model pools and 12 benchmarks spanning scientific discovery, reasoning, code, instruction following, and knowledge show that Stackelberg Alignment achieves the highest macro-average across three diverse model pools, outperforming the strongest training-time baseline by up to 7.4% and the best static inference baseline by 12-25%. Analysis confirms that the adaptive leader concentrates duels on the most informative instructions, and ablations show that both reputation-weighted judgment and reputation-based matching improve the effectiveness of multi-LLM evolution.