Rethinking Projector Training in Multimodal LLMs

📄 arXiv: 2610.18598v1 📥 PDF

作者: Fabian Gröger, Naël Ouerghemi, Shuo Wen, Maria Brbić

分类: cs.CV, cs.AI, cs.CL

发布日期: 2026-10-13

备注: NeurIPS 2026 camera-ready

💡 一句话要点

提出PORTAL以简化多模态LLM的投影器训练

🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 多模态学习 大语言模型 投影器训练 无监督学习 特征对齐

📋 核心要点

  1. 现有的多模态LLM训练方法中，投影器训练阶段的作用不明确，且通常需要大量配对数据。
  2. 论文提出PORTAL，通过计算特征分布的最优传输映射，消除对配对数据和梯度优化的需求。
  3. 实验表明，PORTAL在多个设置中表现优于传统训练管道，且在大多数情况下达到了98%的性能。
  4. method_zh

📝 摘要（中文）

多模态大语言模型（LLMs）通常通过在大规模图像-文本数据集上进行投影器训练来对齐视觉编码器和语言模型。然而，投影器训练阶段的作用尚不明确。研究发现，仅使用10%至20%的数据进行投影器训练即可恢复大部分下游性能。此外，不同数据集上训练的投影器可以线性插值而性能基本保持不变。基于这些发现，提出了PORTAL，一种无需配对图像-文本数据或基于梯度优化的训练自由投影器初始化方法。PORTAL通过计算视觉和语言特征分布的高斯近似之间的最优传输映射，完全跳过投影器训练阶段。PORTAL在没有配对数据的情况下，能够与标准管道相匹配，并在多个设置中表现优异。

🖼️ 关键图片

fig_0
img_1
img_2

📊 实验亮点

PORTAL在没有配对数据的情况下，能够与标准管道在558k图像-文本对上的表现相匹配，并在7种设置中有3种超越了传统方法，达到了平均98%的得分。

🎯 应用场景

该研究的成果可广泛应用于多模态学习领域，特别是在需要处理图像和文本数据的任务中，如图像描述生成、视觉问答等。通过简化训练过程，PORTAL有潜力降低数据需求和计算成本，推动多模态模型的实际应用。

📄 摘要（原文）

Multimodal LLMs are typically trained by first aligning a vision encoder with a language model via projector training on large image-caption datasets, followed by joint training of the full multimodal model. Despite its widespread use, the role of the projector training stage remains poorly understood. We find that (i) using only $10$ to $20\%$ of the data for training the projector already recovers most downstream performance gains, and (ii) projectors trained on different datasets can be linearly interpolated while largely preserving performance. These results suggest that projector training mainly performs a coarse alignment, which many different projectors achieve equally well. Motivated by these findings, we propose PORTAL, a training-free projector initialization that requires neither paired image-caption data nor gradient-based optimization, but only per-modality summary statistics. PORTAL computes an optimal-transport-based map between Gaussian approximations of the vision and language feature distributions on a shared principal subspace, so that the projector training stage can be skipped entirely. Without any paired data, PORTAL matches the standard pipeline trained on 558k image-caption pairs across six LLM backbones, two vision encoders, and 16 benchmarks. It outperforms that pipeline in $3$ of $7$ (vision encoder, LLM) settings and reaches an average $98\%$ of its score across the other four.