ANCHOR: Audio-Visually Grounded Chain-of-Thought Reasoning Benchmark
作者: Joel Julin, Souraja Kundu, Liza Dahiya, George Z. Wei, Liuyue Xie, Jingru Yi, Xu Zhang, Hao Yang, Laszlo A. Jeni
分类: cs.CV
发布日期: 2026-10-16
备注: Accepted at NeurIPS 2026, Evaluations & Datasets Track. Joel Julin and Souraja Kundu are co-first authors
💡 一句话要点
提出ANCHOR基准以解决多模态推理中的证据对齐问题
🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)
关键词: 多模态推理 音视频问答 证据对齐 推理透明性 基准评估
📋 核心要点
- 现有多模态大语言模型在音视频问答中表现良好，但缺乏对推理过程与感知证据的有效评估。
- 本文提出ANCHOR基准，旨在通过结构化推理痕迹和多模态证据来评估推理的对齐性。
- 实验结果表明，模型在答案正确性和推理支持证据之间存在显著差距，强调了对推理过程的深入分析的重要性。
📝 摘要（中文）
多模态大语言模型在音视频问答中表现良好，但正确答案并不意味着其推理过程与相关感知证据相一致。现有基准主要评估最终答案的准确性，缺乏对模型生成的推理是否正确识别支持预测的视觉区域、时间间隔和音频事件的深入洞察。本文提出了ANCHOR，一个用于测量多模态推理中基础推理对齐的基准和评估框架，包含2,319个视频和8,272个问答对，配有结构化推理痕迹、时间证据、空间边界框和显著音频线索。实验结果显示，现有模型在答案性能相似的情况下，其解释是否得到注释证据的支持存在显著差异。
🔬 方法详解
问题定义：本文旨在解决现有多模态推理基准在评估模型推理过程与感知证据对齐方面的不足。现有方法主要关注最终答案的准确性，忽视了推理过程的透明性和合理性。
核心思路：ANCHOR基准通过引入结构化推理痕迹、时间证据和空间边界框，提供了一种全面评估模型推理对齐性的框架。该设计旨在确保模型的推理过程能够与实际的感知证据相匹配。
技术框架：ANCHOR基准包含2,319个视频和8,272个问答对，配有详细的推理痕迹和证据标注。评估过程包括答案正确性、幻觉、逻辑偏差和简洁性等多个维度的测量。
关键创新：ANCHOR的主要创新在于其系统性地评估多模态推理的基础推理对齐，填补了现有基准在推理透明性方面的空白。与传统方法相比，ANCHOR强调了推理过程的证据支持。
关键设计：在实验中，移除视频、边界框和音频对模型准确性的影响进行了评估，结果显示视频的移除导致准确性下降38.4-41.8个百分点，而边界框和音频的移除则导致较小的、模型依赖性的准确性下降。实验设计中采用了人类精炼的测试集，以确保评估的有效性和可靠性。
🖼️ 关键图片
📊 实验亮点
实验结果显示，现有多模态大语言模型在答案性能相似的情况下，其推理支持证据的对齐性存在显著差异。具体而言，视频移除导致准确性下降38.4-41.8个百分点，而边界框和音频的移除则导致较小的下降，表明视频在推理中的重要性。
🎯 应用场景
该研究的潜在应用领域包括教育、智能助理和自动化内容生成等。通过提供更透明的推理过程，ANCHOR能够帮助开发更可靠的多模态系统，提升用户体验和信任度。未来，该基准可能推动多模态推理技术的进一步发展与应用。
📄 摘要（原文）
Multimodal large language models (MLLMs) perform well on audio-visual question answering, but correct answers do not necessarily imply that their stated reasoning is grounded in relevant perceptual evidence. Existing benchmarks mainly evaluate final-answer accuracy, offering limited insight into whether model-generated rationales correctly identify visual regions, temporal intervals, and audio events supporting predictions. We introduce ANCHOR, a benchmark and evaluation framework for measuring grounded-rationale alignment in multimodal reasoning, comprising 2,319 videos and 8,272 question-answer pairs with structured reasoning traces, temporal evidence, spatial bounding boxes, and salient audio cues, including a human-refined test set of 224 videos and 975 question-answer pairs. We separately measure answer correctness, hallucination, logical divergence, and conciseness, with direct spatial and temporal grounding measurements on a focused subset. Across the full benchmark, twelve contemporary MLLMs show a substantial alignment gap: models with comparable answer performance can differ markedly in whether their explanations are supported by annotated evidence. On a matched subset of 100 videos and 446 question-answer pairs, rankings remain highly stable across human-refined, Gemini-generated, and GPT-generated references (mean Pearson r=0.99), providing evidence against model-reference circularity within this subset. Grounding-cue ablations of three MLLMs on the full human-refined test set reveal a consistent modality hierarchy: removing video reduces accuracy by 38.4-41.8 points, while removing bounding boxes (3.2-7.7 points) and audio (4.2-5.6 points) causes smaller, complementary, model-dependent drops. These findings show answer-only evaluation obscures important grounding failures and position ANCHOR as a testbed for evaluating accurate and evidence-aligned audio-visual reasoning.