OP-CAD: On-Policy Clean-Audio Distillation for Robust Audio-Visual Reasoning

📄 arXiv: 2609.39150v1 📥 PDF

作者: Xingming Shui, Dapeng Chen, Bowei Liu, Jingqi Tian, Minfu Li, Kun Yi, Jiapeng Hong, Yansong Tang

分类: cs.CV, cs.SD

发布日期: 2026-09-30

💡 一句话要点

提出OP-CAD以解决环境噪声下的音频-视觉理解问题

🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture) 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 音频-视觉理解 鲁棒性 自蒸馏 环境噪声 多模态学习

📋 核心要点

  1. 现有方法在环境噪声和竞争性语音下的音频-视觉理解能力不足，难以有效抵抗声学干扰。
  2. OP-CAD通过课程学习的方式，逐步引入噪声，利用干净音频的教师反馈来指导学生生成响应。
  3. 实验结果显示，OP-CAD在各种噪声条件下均表现优异，尤其在强干扰情况下能够更好地保留正确答案。

📝 摘要（中文）

在现实环境中，部署的全模态大型语言模型面临外部噪声的干扰，影响其对多模态输入的感知和理解。本文研究了其在音频-视觉理解中的鲁棒性，重点关注在环境噪声和竞争性语音下的问答能力。我们提出了OP-CAD（On-Policy Clean-Audio Distillation），一种基于课程的特权自蒸馏框架，旨在增强音频-视觉理解的鲁棒性。通过逐步增加环境噪声和竞争性语音的强度，OP-CAD在每个阶段提供选择性标记级监督。实验结果表明，OP-CAD在所有评估的噪声条件下均优于对比方法，并且在强干扰下能够更好地保留干净的正确答案，未观察到整体清晰准确率的惩罚。

🔬 方法详解

问题定义：本文旨在解决在环境噪声和竞争性语音下，音频-视觉理解模型的鲁棒性不足的问题。现有方法在处理声学干扰时，未能有效优先考虑受影响的音频信息。

核心思路：OP-CAD的核心思想是通过课程学习逐步引入噪声，并利用干净音频的教师反馈来指导学生生成响应，从而增强模型对音频干扰的抵抗能力。

技术框架：OP-CAD的整体架构包括教师和学生模型，教师使用干净音频和验证答案进行监督，而学生则从受损的音频-视觉输入生成响应。训练过程从轻微噪声开始，逐步增加到严重噪声和竞争性语音。

关键创新：OP-CAD的主要创新在于其选择性标记级监督机制，通过比较教师在不同上下文下的预测，强调声学敏感位置的监督，从而提高音频-视觉推理的鲁棒性。

关键设计：在设计中，OP-CAD采用了有界加权规则，以强调受信号影响的标记位置，同时在整个响应生成过程中保持监督。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

实验结果表明，OP-CAD在所有评估的噪声条件下均优于对比方法，尤其在强干扰情况下，能够更好地保留干净的正确答案，未观察到整体清晰准确率的惩罚。这表明OP-CAD在音频-视觉推理中的有效性和鲁棒性。

🎯 应用场景

该研究的潜在应用领域包括智能助理、自动字幕生成和多模态信息检索等。通过提高音频-视觉理解的鲁棒性，OP-CAD能够在嘈杂环境中提供更可靠的服务，具有重要的实际价值和广泛的应用前景。

📄 摘要（原文）

Omni-modal large language models deployed in real-world environments encounter external noise that can interfere with their perception and understanding of multimodal inputs. We study their robustness in audio-visual understanding, focusing on question answering under environmental noise and competing speech. The challenge is to resist acoustic interference while preserving useful audio evidence. On-policy distillation provides dense teacher feedback on student-generated responses, but uniform token weighting does not explicitly prioritize positions affected by acoustic interference. We introduce OP-CAD (On-Policy Clean-Audio Distillation), a curriculum-based privileged self-distillation framework for robust audio-visual understanding. Training progresses from mild to severe environmental noise and competing speech, with selective token-level supervision at each stage. The student generates responses from corrupted audio-visual input, while a frozen teacher uses clean audio and the verified answer to supervise the same response prefixes. To allocate this supervision, OP-CAD compares teacher predictions under clean, corrupted, and visual-only contexts without revealing the answer. These matched comparisons measure sensitivity to audio removal and corruption; a bounded weighting rule emphasizes positions identified by either signal while retaining supervision throughout the response. OP-CAD outperforms the compared methods across all evaluated noise conditions. Paired analyses further show improved preservation of clean-correct answers under strong interference, with no observed aggregate clean-accuracy penalty. These results demonstrate the value of directing clean-teacher supervision toward acoustically sensitive predictions for robust audio-visual reasoning.