Omni Demand Understanding: A Benchmark for Contextual User-Intent Inference in Multimodal Interaction

📄 arXiv: 2609.21392v1 📥 PDF

作者: Qi Chen, Yunfei Chu, Haolin He, Yifan Yang, Zihan Liu, Yuxuan Wang, Ziyang Ma, Ruiyang Xu, Meng Gao, Yinsong Yan, Ling Wang, Hui Wang, Wen Huang, Yiheng Chen, Guanrou Yang, Qiuqiang Kong, Jin Xu, Xie Chen

分类: cs.CL, cs.CV, cs.MM, cs.SD

发布日期: 2026-09-18

💡 一句话要点

提出Omni Demand Understanding以解决多模态交互中的用户需求推断问题

🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 多模态交互 用户需求推断 上下文理解 智能助手 模型评估

📋 核心要点

  1. 现有的多模态交互基准主要集中在响应质量上，未能有效评估模型推断用户需求的能力。
  2. 本文提出Omni Demand Understanding (ODU)问题，旨在通过多模态和对话上下文推断用户的真实需求。
  3. 实验结果显示，最强模型仅能恢复44.7%的关键信息，且大多数模型在非需求场景下的误触发率超过50%。

📝 摘要（中文）

自然的音视频交互正成为AI助手的重要接口，使用户能够通过语音和视觉进行交流。然而，现有的交互能力基准主要关注响应质量，未能深入探讨模型能否从复杂的多模态交互中正确推断用户的潜在需求。用户的真实需求往往在语音中表述不清，需要从多模态线索和对话历史中推断。为此，本文提出Omni Demand Understanding (ODU)作为一个独特的多模态上下文推断问题，评估模型在单轮和多轮交互中识别用户需求和推断意图的能力。通过构建ODU-Bench并评估14个本地多模态语言模型，发现当前模型在推断用户需求方面存在显著能力差距。

🔬 方法详解

问题定义：本文旨在解决多模态交互中用户需求推断的能力不足问题。现有方法往往忽视了用户需求的复杂性，导致推断不准确。

核心思路：ODU通过分析用户的多模态交互流，识别用户需求的存在与否，并从中推断意图。该方法强调上下文信息的重要性，尤其是在复杂和模糊的表达情况下。

技术框架：ODU-Bench的构建包括挑战驱动的分类法、基于分类法的智能视频生成以及人类记录的交互，最后通过媒体基础的注释和人类验证来确保数据质量。

关键创新：ODU作为一个新的多模态上下文推断问题，填补了现有基准的空白，特别是在用户需求的识别和推断方面。与传统方法相比，ODU更注重多模态信息的综合利用。

关键设计：在模型评估中，采用了多轮和单轮交互的评估维度，关注模型在不同场景下的表现，设置了多种损失函数以优化推断准确性。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

实验结果显示，即使是最强的模型Gemini 3.1 Pro，仅能恢复44.7%的关键信息，且14个模型中有11个在非需求场景下的误触发率超过50%。这些结果揭示了当前多模态语言模型在推断用户需求方面的系统性能力差距。

🎯 应用场景

该研究的潜在应用领域包括智能助手、客户服务和人机交互等。通过提升模型对用户需求的理解能力，可以显著改善用户体验，使AI助手能够更准确地响应用户的真实意图，进而推动智能交互技术的发展。

📄 摘要（原文）

Natural audio-visual interaction is emerging as an important interface for AI assistants, allowing users to communicate through speech and vision rather than carefully composed text prompts. However, existing benchmarks of interactive capabilities still focus primarily on response quality, leaving a more fundamental question underexplored: can a model correctly infer the user's underlying demand from complex multimodal interaction? Real-world user demands are often underspecified in speech and must be inferred from multimodal cues and dialogue history. This inference is further complicated by ambiguous or disfluent expression and noisy acoustic environments. Conversely, request-like speech may not constitute a demand to the assistant, leading to false triggers. We establish Omni Demand Understanding (ODU) as a distinct multimodal contextual inference problem: given an interaction stream, a model must detect whether a user demand is present and infer intent from multimodal and conversational context. ODU evaluates this capability along five dimensions, covering both single-turn and multi-turn interactions. We construct ODU-Bench using a challenge-driven taxonomy, taxonomy-guided agentic video generation, and human-recorded interactions, followed by media-grounded annotation and human verification. We evaluate 14 native MLLMs. Even the strongest, Gemini 3.1 Pro, recovers only 44.7% of key information that must be inferred from visual, acoustic, or conversational context. Moreover, 11 of the 14 models exhibit false-trigger rates above 50% on non-demand scenarios. These results reveal a systematic capability gap in current MLLMs' ability to infer contextual user demands. We hope ODU can establish the evaluation of a previously underexplored yet essential capability in multimodal interaction: correctly understanding user demands before generating an appropriate response.