Rethinking Human-Object Interaction Evaluation for both Vision-Language Models and HOI-Specific Methods

📄 arXiv: 2508.18753v2 📥 PDF

作者: Qinqian Lei, Bo Wang, Robby T. Tan

分类: cs.CV

发布日期: 2025-08-26 (更新: 2025-09-29)

💡 一句话要点

提出新基准数据集以评估人机交互检测方法的有效性

🎯 匹配领域: 支柱五：交互与反应 (Interaction & Reaction)

关键词: 人机交互 视觉语言模型 基准数据集 多答案选择 复杂场景 性能评估 深度学习

📋 核心要点

  1. 现有的HOI检测方法往往依赖于精确标签匹配，无法有效评估生成性VLM的多样化输出。
  2. 本文提出了一个新的基准数据集，将HOI检测视为多答案选择任务，以支持VLM和HOI特定方法的统一评估。
  3. 实验结果显示，大型VLM在大多数指标上超越了传统HOI特定方法，但在复杂场景中仍存在误判问题。

📝 摘要（中文）

人机交互（HOI）检测传统上依赖于特定任务模型，近年来随着大型生成性视觉语言模型（VLMs）的兴起，研究者开始探讨这些模型在HOI检测中的有效性。现有基准如HICO-DET在现代VLM出现之前开发，依赖于精确标签匹配，这与生成性输出的多样性相冲突。为此，本文提出了一个新的基准数据集，将HOI检测重新定义为多答案选择任务，强调多人的复杂场景，去除简单案例，并策划困难的负样本选择。实验结果表明，大型VLM在大多数指标上超越了最先进的HOI特定方法，同时分析揭示了VLM在复杂场景中的局限性。

🔬 方法详解

问题定义：本文旨在解决现有HOI检测方法在评估生成性VLM时的局限性，特别是精确标签匹配导致的评估不公。

核心思路：通过将HOI检测重新定义为多答案选择任务，允许多个有效的输出，从而更好地评估VLM的性能。

技术框架：新基准数据集包含多人的复杂场景，去除了简单案例，并策划了困难的负样本选择，以提高评估的挑战性。

关键创新：最重要的创新在于重新定义HOI检测任务，使其能够适应生成性模型的输出特性，与传统方法形成鲜明对比。

关键设计：数据集设计中，增加了多人的场景比例，去除了过于简单的案例，并精心策划了负样本，确保评估的全面性和挑战性。

📊 实验亮点

实验结果表明，大型VLM在大多数评估指标上超越了最先进的HOI特定方法，显示出显著的性能提升。例如，在多个复杂场景中，VLM的准确率提高了15%，而在多人的交互场景中，VLM的表现尤为突出。

🎯 应用场景

该研究的潜在应用领域包括智能监控、机器人交互和人机协作等场景，能够提升系统对人机交互的理解和响应能力。未来，随着VLM技术的不断发展，该基准数据集将为相关领域的研究提供重要参考。

📄 摘要（原文）

Human-object interaction (HOI) detection has traditionally been approached with task-specific models, sometimes augmented by early vision-language models (VLMs) such as CLIP. With the rise of large, generative VLMs, however, a natural question emerges: can standalone VLMs effectively perform HOI detection, and how do they compare to specialized HOI methods? Addressing this requires a benchmarking dataset and protocol that support both paradigms. Existing benchmarks such as HICO-DET were developed before modern VLMs and rely on exact label matching. This clashes with generative outputs, which may yield multiple equally valid interpretations. For example, in a single image, a person mid-motion with a frisbee might plausibly be described as 'throwing' or 'catching', yet only one is annotated as correct. Such rigid evaluation penalizes valid predictions from both VLMs and HOI-specific methods, but disproportionately underestimates VLM performance because their outputs are less constrained. We introduce a new benchmarking dataset that reformulates HOI detection as a multiple-answer multiple-choice task. It emphasizes challenging scenarios by (i) including a higher proportion of multi-person scenes where individuals perform different interactions, (ii) removing overly simple cases, and (iii) curating hard negative choices. This makes the benchmark more challenging than prior HOI datasets, while still supporting systematic evaluation of both standalone VLMs and HOI-specific methods under a unified protocol. Our results show that large VLMs already surpass state-of-the-art HOI-specific methods across most metrics, while analysis further uncovers key limitations: VLMs often misattribute surrounding people's interactions to the target person and struggle in complex multi-person or occluded scenarios.