FuseAdapt: Parameter-Efficient Multimodal Fusion for Semantic Segmentation with Missing Modalities

📄 arXiv: 2610.21779v1 📥 PDF

作者: Xin Zhang, Robby T. Tan

分类: cs.CV

发布日期: 2026-10-15

备注: 25 pages, 6 figures, 19 tables. Accepted to NeurIPS 2026

🔗 代码/项目: PROJECT_PAGE

💡 一句话要点

提出FuseAdapt以解决多模态语义分割中的缺失模态问题

🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 多模态分割 语义分割 视觉基础模型 适应空间融合 预测模态建模 深度学习 计算机视觉

📋 核心要点

  1. 现有多模态语义分割方法在模态缺失时性能显著下降，缺乏有效的融合策略。
  2. FuseAdapt通过在冻结的视觉基础模型适应空间内进行轻量级模态适应和融合，提升了多模态分割的鲁棒性。
  3. 在MCubeS、DELIVER和MUSES三个基准上，FuseAdapt在完整模态下提升了3.01 mIoU，在缺失模态下提升了12.56 mIoU，表现优于现有方法。

📝 摘要（中文）

多模态语义分割依赖于互补传感器，但在测试时某些模态的缺失会导致性能显著下降。现有方法多依赖于专用的融合模块或多分支跨模态编码器，缺失模态时融合效果不佳。本文提出FuseAdapt，一个参数高效的框架，通过在冻结的视觉基础模型（VFM）适应空间内进行融合，探索VFM先验是否可以作为多模态分割的有效共享骨干。FuseAdapt为每个模态分配轻量级适应分支，通过保持低秩适应私有化到每个模态，并在冻结骨干内路由仅FFN低秩残差激活来实现适应空间融合。为处理缺失模态，我们进一步引入预测模态建模（PMM），在训练期间掩蔽目标模态并从可见模态预测其嵌入。FuseAdapt在三个多模态分割基准上均表现出色，显著提升了完整模态和缺失模态的分割效果。

🔬 方法详解

问题定义：本文旨在解决多模态语义分割中由于模态缺失导致的性能下降问题。现有方法在模态缺失时依赖于复杂的融合模块，导致不可靠的融合效果。

核心思路：FuseAdapt的核心思路是利用冻结的视觉基础模型（VFM）进行模态适应和融合，避免了对独立融合网络的依赖，从而提高了参数效率和鲁棒性。

技术框架：FuseAdapt的整体架构包括轻量级适应分支和适应空间融合模块。每个模态都有独立的适应分支，低秩适应保持私有，跨模态的低秩残差激活在冻结骨干网络中进行路由。

关键创新：FuseAdapt的主要创新在于引入了适应空间融合和预测模态建模（PMM），使得在模态缺失情况下仍能有效进行分割，且不增加推理时的开销。

关键设计：在设计中，FuseAdapt采用了轻量级的适应分支结构，损失函数通过PMM进行监督，确保模态间的有效交互和信息共享。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

FuseAdapt在三个多模态分割基准上表现出色，完整模态下提升了3.01 mIoU，缺失模态下提升了12.56 mIoU，显著优于现有最先进的方法，展示了其在多模态语义分割中的有效性和优势。

🎯 应用场景

FuseAdapt在自动驾驶、医学影像分析和机器人视觉等领域具有广泛的应用潜力。通过提高多模态分割在缺失模态情况下的鲁棒性，该方法能够增强系统在复杂环境中的适应能力，提升决策质量和安全性。

📄 摘要（原文）

Multi-modal semantic segmentation benefits from complementary sensors, but this benefit is fragile when some modalities are degraded or missing at test time. Many existing methods rely on dedicated fusion modules or multi-branch cross-modal encoders, and the fusion often becomes unreliable when modalities are absent, leading to substantial performance drops. We propose \textbf{FuseAdapt}, a parameter-efficient framework that performs fusion within the adaptation space of a frozen vision foundation model (VFM), exploring whether VFM priors can serve as an effective shared backbone for multi-modal segmentation. Instead of attaching a standalone fusion network, FuseAdapt assigns each modality a lightweight adaptation branch and performs \emph{adaptation-space fusion} by keeping Q/V low-rank adaptation private to each modality and routing only FFN low-rank residual activations across modalities inside the frozen backbone. The pretrained VFM remains fixed while modality-specific corrections and explicit cross-modal interaction are learned through the adaptation paths. To handle missing modalities, we further introduce \textbf{Predictive Modality Modeling (PMM)}, a direct representation-level objective for cross-modal redundancy. During training, PMM masks a target modality and predicts its embedding from the visible ones, supervised by a teacher target computed under full-modality input. PMM is used only during training and adds no inference-time overhead. Across three multi-modal segmentation benchmarks (MCubeS, DELIVER, MUSES), FuseAdapt consistently improves both modality-complete and modality-incomplete segmentation with a small trainable budget, yielding up to +3.01 mIoU under full modalities and +12.56 mIoU under missing modalities over prior state-of-the-art. Project page: https://devinxzhang.github.io/FuseAdapt-Page/.