From Proprietary to Open-Source: Bridging the Distribution Gap via Multi-Agent Protocol Distillation in Agentic Search

📄 arXiv: 2607.24280v1 📥 PDF

作者: Junlin Liu, Jiangwang Chen, Zixin Song, Shuaiyu Zhou, Chunji Lv, Hank Wu, Kailin Jiang, Jinyang Wu, Bohan Yu, Chenxi Zhou

分类: cs.AI

发布日期: 2026-07-27

💡 一句话要点

提出多智能体协议蒸馏以解决知识密集型任务的监督稀疏问题

🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture) 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 知识蒸馏 强化学习 多智能体系统 推理能力 自然语言处理

📋 核心要点

  1. 现有的知识密集型任务优化方法在监督信号稀疏性方面存在挑战，导致推理能力不足。
  2. 提出的多智能体协议蒸馏(MAPD)框架通过结构化协议提供密集的蒸馏信号，结合强化学习优化。
  3. 实验结果显示，MAPD在多个基准测试中超越了竞争对手，成功率显著提升，验证了其有效性。

📝 摘要（中文）

Agentic search使大型语言模型能够通过多步骤推理与检索相结合来解决知识密集型任务，但基于结果的强化学习(RL)优化仅提供稀疏监督。知识蒸馏可以提供更密集的指导，而先进的专有模型则是有潜力的教师。传统的logit匹配因隐藏logits和不匹配的tokenizer而受到限制，而自然语言轨迹模仿则转移了表面风格特征而非核心推理能力。为了解决异构蒸馏问题并弥合分布差距，本文提出了多智能体协议蒸馏(MAPD)，这是一个联合蒸馏和RL框架，使用结构化、风格标准化的协议作为中间表示。实验表明，MAPD在七个QA基准上表现优异，成功率分别为39.4%和44.4%。

🔬 方法详解

问题定义：本文旨在解决知识密集型任务中监督信号稀疏的问题，现有方法在从专有模型蒸馏时面临logit匹配和风格转移的挑战。

核心思路：提出多智能体协议蒸馏(MAPD)框架，通过使用结构化、风格标准化的协议作为中间表示，提供更密集的监督信号，结合强化学习进行优化。

技术框架：MAPD框架包括一个离线多智能体系统(MAS)，该系统分解查询、检索支持证据、修复失败搜索，并将探索轨迹转换为JSON协议，包含任务类型、推理计划和基础事实。

关键创新：MAPD的主要创新在于使用结构化协议作为中间表示，解决了传统蒸馏方法中的隐藏logits和不匹配tokenizer问题，显著提高了蒸馏信号的密度。

关键设计：在训练过程中，协议仅提供给学生策略的特权分支，确保其token分布提供密集的蒸馏信号，同时保持稀疏的RL目标。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

实验结果表明，MAPD在七个QA基准测试中表现优异，Qwen3-1.7B模型的平均成功率达到39.4%，而Qwen3-4B模型则达到44.4%。这些结果显著超越了现有的蒸馏和强化学习方法，验证了MAPD的有效性和鲁棒性。

🎯 应用场景

该研究的潜在应用领域包括智能问答系统、知识检索和自动化推理等。通过提高模型的推理能力和知识获取效率，MAPD框架有望在教育、医疗和客户服务等多个行业中发挥重要作用，推动智能系统的进一步发展。

📄 摘要（原文）

Agentic search enables large language models to solve knowledge-intensive tasks by interleaving multi-step reasoning with retrieval, yet optimizing this with outcome-based reinforcement learning (RL) provides only sparse supervision. Knowledge distillation can supply denser guidance, and advanced proprietary models with their strong reasoning capabilities are promising teachers. While distilling from proprietary models can densify this supervisory signal, conventional logit-matching is precluded by hidden logits and mismatched tokenizers, whereas raw natural language trajectory imitation transfers superficial stylistic artifacts rather than core reasoning competence. To address the heterogeneous distillation problem and bridge the distribution gap, we propose Multi-Agent Protocol Distillation (MAPD), a joint distillation and RL framework uses a structured, style-normalized protocol as an intermediate representation. An offline multi-agent system (MAS) decomposes each query, retrieves supporting evidence, repairs failed searches, and converts the resulting exploration trace into a JSON protocol containing the task type, reasoning plan, and extractive grounding facts. During training, the protocol is provided only to a privileged branch of the student policy, whose token distributions furnish a dense distillation signal alongside the sparse RL objective. Extensive evaluations across seven QA benchmarks demonstrate that MAPD consistently outperforms competitive distillation and RL, achieving average success rates of 39.4\% on Qwen3-1.7B and 44.4\% on Qwen3-4B. Crucially, the framework generalizes robustly across diverse proprietary teachers while effectively mitigating the student policy from style drift and verbosity degeneration.