Seeing What Should Be Heard: Diagnosing and Repairing Cross-Modal Shortcuts in Omni-Modal LLMs
作者: Yueran Ma, Ronghao Lin
分类: cs.CV, cs.CL, cs.LG
发布日期: 2026-09-29
备注: 25 pages, 11 figures, 16 tables
💡 一句话要点
提出DMC-Repair以解决多模态大语言模型中的跨模态捷径问题
🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture) 支柱九：具身大模型 (Embodied Foundation Models)
关键词: 多模态大语言模型 跨模态捷径 因子化模态诊断 DMC-Repair 音频问题回答 模型训练 人工智能
📋 核心要点
- 现有的多模态大语言模型在回答问题时常常依赖于冗余的模态信息，导致模型未能准确遵循问题所指的模态。
- 论文提出DMC-Repair，通过对跨模态样本进行训练，确保模型根据问题指定的模态进行回答，从而抑制跨模态捷径的影响。
- 实验结果显示，DMC-Repair将图像引起的答案影响减少了59.9%，并在不同模型和未见数据集上保持了良好的性能。
📝 摘要（中文）
多模态大语言模型（LLMs）应根据问题所指的模态进行回答。然而，现有训练方法很少验证模型是否真正遵循这一模态，因为来自同一样本的多模态输入往往提供冗余证据。本文揭示了多模态LLMs中普遍存在的跨模态捷径：在回答与音频相关的问题时，模型不仅依赖音频，还依赖图像，甚至有时更依赖图像。为系统性地诊断这一行为，本文提出了因子化模态诊断方法，通过在样本之间独立交换音频和图像来隔离每种模态的因果贡献。实验表明，DMC-Repair有效减少了图像对答案的影响，提升了音频问题回答的性能。
🔬 方法详解
问题定义：本文旨在解决多模态大语言模型在回答音频相关问题时，过度依赖图像信息的问题。现有方法未能有效验证模型是否遵循问题所指的模态，导致模型利用冗余信息进行回答。
核心思路：论文的核心思路是通过因子化模态诊断方法，独立交换样本中的音频和图像，以隔离每种模态的因果贡献，从而识别和修复模型的跨模态捷径。
技术框架：整体架构包括因子化模态诊断和DMC-Repair两个主要模块。因子化模态诊断用于识别模型对不同模态的依赖程度，而DMC-Repair则通过对跨模态样本进行训练，确保模型根据问题指定的模态进行回答。
关键创新：最重要的技术创新点在于提出了DMC-Repair方法，该方法通过对样本进行跨模态交换训练，抑制了模型对冗余视觉信息的依赖，显著提升了音频问题的回答性能。
关键设计：在DMC-Repair中，模型的训练过程采用了特定的监督机制，确保模型在回答时依据问题指定的模态。此外，损失函数的设计也考虑了不同模态间的相互影响，以优化模型的学习效果。
🖼️ 关键图片
📊 实验亮点
实验结果表明，DMC-Repair方法有效减少了图像对答案的影响，具体表现为图像引起的答案影响减少了59.9%。该方法在不同模型和未见数据集上均表现出良好的泛化能力，验证了其有效性和实用性。
🎯 应用场景
该研究的潜在应用领域包括多模态信息检索、智能助手和自动化问答系统等。通过有效抑制跨模态捷径，模型能够更准确地理解和回答用户的问题，提升人机交互的质量和效率。未来，该方法可能推动多模态学习领域的进一步发展，促进更智能的AI系统的构建。
📄 摘要（原文）
Omni-modal large language models (LLMs) are expected to answer a question using the modality it explicitly refers to. However, existing training paradigms rarely verify whether models actually follow this modality, because multimodal inputs from the same sample often provide redundant evidence for the same answer. In this work, we uncover a pervasive cross-modal shortcut in omni-modal LLMs: when asked an audio-related question, models rely on the image as much as on the audio, and sometimes even more. To systematically diagnose this behavior, we introduce the Factorized Modality Diagnostic, which independently swaps audio and images between samples to isolate each modality's causal contribution. Across two model families in different settings, we find that this shortcut persists throughout supervised fine-tuning and reinforcement learning post-training, while judge-based RL may further amplify such reliance on irrelevant visual information. Based on this finding, we propose DMC-Repair, which trains models on the same kind of cross-modal swapped samples while assigning supervision according to the modality specified by the question. This prevents models from exploiting the spurious correspondence between modalities within the same clip. Experiments demonstrate that DMC-Repair reduces the image-induced share of the answer effect by 59.9%, effectively suppressing the cross-modal shortcut without compromising audio-question answering performance. The reduction in shortcut reliance generalizes across two model families and zero-shot to an unseen dataset and an unseen benchmark, and persists through subsequent post-training. Code is available at https://anonymous.4open.science/r/DMC-Repair.