Mixture of Probes: Learning from Privileged Modalities in Multimodal LLMs Through Probing

📄 arXiv: 2607.08839v1 📥 PDF

作者: Dominick Reilly, Qiyu Wu, Hiromi Wakaki, Srijan Das, Yuki Mistufuji

分类: cs.CV, cs.LG

发布日期: 2026-07-09

备注: Preprint (16 pages)

🔗 代码/项目: GITHUB

💡 一句话要点

提出混合探测器以解决多模态大语言模型训练中的特权模态问题

🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 多模态大语言模型 特权模态 混合探测器 跨模态学习 结构化探测机制 模型训练 信息解耦 性能提升

📋 核心要点

  1. 现有的多模态大语言模型假设训练和推理时可用的模态一致，导致在特权模态设置下的性能不足。
  2. 本文提出混合探测器（MoP），通过解耦模态特定和通用信号，提升模型在不同模态间的可迁移性。
  3. MoP在八个任务和四种模态上进行评估，相较于基线模型实现了高达65%的性能提升，证明了辅助模态的有效性。

📝 摘要（中文）

多模态大语言模型（MLLMs）通常假设训练期间可用的所有模态在推理时也可用。然而，许多现实场景违反了这一假设，要求模型在特权模态设置下运行，即辅助模态仅在训练期间可用。现有的MLLMs未能有效利用这些模态，因为它们将模态视为可互换的输入，而非互补监督的来源。本文提出了混合探测器（MoP），通过结构化探测机制提取和组织共享模态编码器的中间表示信息，从而实现模态特定和模态通用信号的解耦。我们还引入了MoP跨模态训练（MoP-X），通过探测解耦损失防止探测器崩溃，促进跨模态学习。实验表明，MoP在多个任务和模态上表现优异，相较于强基线模型提升达65%。

🔬 方法详解

问题定义：本文旨在解决多模态大语言模型在特权模态设置下的训练与推理不一致的问题。现有方法未能有效利用训练期间可用的辅助模态信息，导致模型性能受限。

核心思路：提出混合探测器（MoP）框架，通过结构化探测机制解耦模态特定和模态通用信号，使模型能够在不同模态间学习可迁移的表示。

技术框架：MoP的整体架构包括一个共享模态编码器和多个探测器，探测器负责提取和组织中间表示信息。通过探测解耦损失，MoP-X训练策略进一步支持模态信息的有效利用。

关键创新：MoP的主要创新在于其结构化探测机制，区别于现有方法仅依赖最终层对齐，能够更好地利用中间表示信息。

关键设计：在训练过程中，采用探测解耦损失来防止探测器崩溃，并促进跨模态学习。模型的参数设置和网络结构经过精心设计，以确保不同模态间的有效信息传递。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

在实验中，MoP在八个任务和四种模态上进行了评估，结果显示其相较于强基线模型实现了高达65%的相对提升。这一结果表明，即使在推理时辅助模态不可用，训练期间有效利用这些模态仍能显著提高模型性能。

🎯 应用场景

该研究具有广泛的应用潜力，尤其在需要处理多模态信息的任务中，如图像描述生成、视频理解和人机交互等领域。通过有效利用辅助模态，模型能够在推理时实现更高的准确性和鲁棒性，推动智能系统的进步。未来，MoP框架可能会在更多实际应用中得到推广，提升多模态学习的效率和效果。

📄 摘要（原文）

Multimodal Large Language Models (MLLMs) are typically designed under the assumption that all modalities available during training will also be accessible at inference. However, many real-world settings violate this assumption, requiring models to operate under a privileged modality setting, where auxiliary modalities are available only during training. While these modalities contain valuable information, existing MLLMs largely fail to leverage them effectively, as they treat modalities as interchangeable inputs rather than sources of complementary supervision. We propose Mixture of Probes (MoP), a novel framework that disentangles modality-specific and modality-general signals within the MLLM, allowing the model to preserve modality-dependent structure while learning transferable representations across modalities. At its core, MoP achieves this through a structured probing mechanism that extracts and organizes information from intermediate representations of a shared modality encoder, rather than relying only on final-layer alignment as done in existing MLLMs. To support this disentanglement, we further introduce MoP Cross-modal Training (MoP-X), a training strategy for MoP centered around a probe disentanglement loss that prevents probe collapse and encourages cross-modal learning. We evaluate MoP across two domains spanning eight tasks and four modalities under a comprehensive evaluation protocol tailored to the privileged modality setting, where each modality is independently treated as the sole input at inference time. MoP consistently outperforms strong MLLM baselines, achieving up to 65% relative improvement, demonstrating that auxiliary modalities, even when unavailable at inference, can provide substantial gains when effectively leveraged during training. Code, model checkpoints, and evaluation protocols will be made available at https://github.com/Sony/MoP.