DIVE: Benchmarking Full-Duplex Interaction with Visual Engagement in Continuous Streams

📄 arXiv: 2610.28405v1 📥 PDF

作者: Yuhang Chen, Guangyue Liu, Shilong Liu

分类: cs.AI

发布日期: 2026-10-20

备注: Preprint. 19 pages, 4 figures

💡 一句话要点

提出DIVE基准以解决实时多模态交互评估问题

🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 多模态交互 实时响应 基准评估 视觉跟踪 人机交互 任务设计 模型评估

📋 核心要点

  1. 现有方法主要通过时间戳问答评估多模态感知，无法有效支持实时交互的需求。
  2. DIVE基准通过六个任务评估模型在连续多模态流中的全双工交互能力，涵盖主动干预、持续参与和交互协调等场景。
  3. 实验表明，领先的实时模型在响应延迟上表现优异，但响应质量和窗口响应率仍有待提升，尤其是在适应新信息方面。

📝 摘要（中文）

近年来，多模态大语言模型（MLLMs）的进展显著提升了语音交互和视频理解能力。现有的在线和流媒体基准主要通过时间戳问答评估因果多模态感知，而现实世界的交互需要模型能够持续识别任务相关的多模态线索，并决定何时、如何响应。为填补这一空白，本文提出DIVE基准，用于评估在连续和真实多模态流中进行全双工交互的能力。DIVE组织了六个任务，涵盖三种互补的交互场景，旨在推动更自然和高效的实时多模态交互系统的发展。

🔬 方法详解

问题定义：论文要解决的问题是如何在连续的多模态流中评估模型的全双工交互能力。现有方法无法有效处理实时交互中的动态信息和多模态线索的变化。

核心思路：DIVE基准通过设计六个任务，评估模型在不同交互场景下的表现，旨在促进模型在实时多模态交互中的能力提升。

技术框架：DIVE基准包含三个主要交互场景：主动干预、持续参与和交互协调。每个场景下又细分为多个任务，涵盖了从视觉跟踪到多方交互的各个方面。

关键创新：DIVE的创新在于其全面的任务设计和细致的人类注释，能够更真实地模拟实际应用中的多模态交互场景，区别于传统的静态问答评估。

关键设计：DIVE包含525个音频/视频样本，配有精确的时间元数据和人类标注，评估协议分离了响应质量、参与决策和响应时机等指标，确保评估的全面性和准确性。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

实验结果显示，领先的实时模型在响应延迟上达到了亚秒级的平均水平，但其平均窗口响应率仅为53.77%。不同模型和任务间的响应质量差异显著，尤其是在适应新信息方面表现出挑战。

🎯 应用场景

DIVE基准的潜在应用领域包括智能助手、视频会议系统和教育技术等，能够帮助开发更自然的多模态交互系统。通过提升模型在实时交互中的表现，DIVE有望推动人机交互的进一步发展，提升用户体验。

📄 摘要（原文）

Recent advances in multimodal large language models (MLLMs) have substantially improved spoken interaction and video understanding. Existing online and streaming benchmarks primarily assess causal multimodal perception through timestamped question answering, whereas real-world interaction requires models to continuously identify task-relevant multimodal cues and decide whether, when, and how to respond. This leaves open how well current models can sustain real-time interaction throughout an evolving multimodal stream. To address this gap, we introduce DIVE, a benchmark for evaluating full-duplex interaction with visual engagement under continuous and realistic multimodal streams. It organizes six tasks into three complementary interaction scenarios. (1) Proactive Intervention evaluates whether models can identify emerging visual or semantic cues and intervene at the appropriate moment. (2) Continuous Engagement evaluates sustained participation through continuous visual tracking and visually grounded query interaction. (3) Interaction Coordination evaluates how models coordinate listening and speaking under overlapping and multi-party interactions. DIVE contains 525 audio/video samples with fine-grained human annotations and precise temporal metadata. Its evaluation protocol separates response quality, participation decisions, and response timing. We evaluate representative real-time models using response quality, window response rates, and reaction latency. Experiments show that leading real-time models achieve sub-second average response latency, yet their average window response rates remain at or below 53.77%. Response quality varies substantially across models and tasks, with adapting ongoing responses to newly arriving information remaining particularly challenging. We hope DIVE facilitates progress toward more natural and capable real-time multimodal interactive systems.