CountVideoBench: A Benchmark to Evaluate Prediction Biases on the Object Counting task by Video-Language Models
作者: Marc Chapus, Carlos Crispim-Junior, Véronique Eglin, Atilla Baskurt
分类: cs.CV, cs.AI, cs.LG
发布日期: 2026-10-20
备注: Accepted at BMVC 2026. Project page: https://countvideobench.github.io
💡 一句话要点
提出CountVideoBench以解决视频语言模型在物体计数任务中的偏差问题
🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)
关键词: 视频理解 物体计数 多模态模型 基准评估 计数偏差 合成视频 视频问答 模型评估
📋 核心要点
- 现有的视频问答基准在物体计数任务中，真实计数分布严重偏向小值，导致模型评估不全面。
- 本文提出CountVideoBench基准，包含2600个合成视频，覆盖26个计数值，旨在提供更均衡的评估。
- 实验结果显示，16个视频语言模型的准确率未超过27.6%，并且模型在计数值上的表现差异显著，揭示了计数任务的复杂性。
📝 摘要（中文）
物体计数在视频中的能力仍然是多模态大语言模型中一个未充分表征的能力。现有的视频问答基准中，计数问题的真实分布严重偏向小值，导致模型在计数任务中的表现受到影响。为此，本文提出CountVideoBench，一个包含2600个合成视频的基准，覆盖26个计数值，旨在评估视频语言模型在物体计数任务中的偏差。实验结果显示，16个视频语言模型在零-shot设置下的准确率均未超过27.6%，并且模型的准确性与计数值的依赖关系各异，突显了建立平衡的计数基准的必要性。
🔬 方法详解
问题定义：本文旨在解决视频语言模型在物体计数任务中的预测偏差问题。现有基准中计数值分布偏向小值，导致模型在较大计数值上的表现被忽视。
核心思路：提出CountVideoBench基准，通过合成视频覆盖更广泛的计数值范围，以便更全面地评估模型的计数能力。设计上避免了物体运动的干扰，确保了计数任务的纯粹性。
技术框架：CountVideoBench包含2600个合成视频，分为六个连续计数的集群，每个集群包含100个视频，涵盖从5到50的26个计数值。每个视频展示单一类静态物体，控制相机运动。
关键创新：最重要的创新在于构建了一个均衡的计数基准，能够有效评估模型在不同计数值上的表现，尤其是对较大计数值的评估，填补了现有基准的空白。
关键设计：在实验中，评估了16个视频语言模型，采用零-shot设置，未发现任何模型的准确率超过27.6%。模型的准确性与计数值的依赖关系表现出多样性，部分模型在特定整数上表现优异，而在相邻计数值上则表现不佳。
🖼️ 关键图片
📊 实验亮点
实验结果显示，16个视频语言模型在零-shot设置下的最高准确率仅为27.6%。模型在计数值上的表现差异显著，部分模型在特定整数上表现优越，强调了建立均衡计数基准的重要性。
🎯 应用场景
该研究的潜在应用领域包括视频监控、自动驾驶、智能家居等场景，能够提升多模态模型在物体计数任务中的准确性和可靠性。未来，CountVideoBench可能成为评估视频理解能力的重要标准，推动相关技术的发展。
📄 摘要（原文）
Object counting in videos remains a poorly characterized capability of multimodal large language models. Counting questions are included in many existing VideoQA benchmarks, but the ground-truth count distributions in these benchmarks are heavily skewed toward small values. Across the nine benchmarks we survey, the median ground-truth count is at most five, and the three most frequent values together account for between 45% and 93% of the videos. The resulting mean accuracy is then dominated by the most frequent counts; a value-dependent bias at a rare count contributes negligibly to the average and remains invisible. We introduce CountVideoBench, a benchmark of 2,600 synthetic videos covering 26 cardinalities in the interval [5, 50] (100 videos each), grouped into six clusters of consecutive counts that span the range while enabling near-neighbor comparisons. Each video contains a single class of static objects under a controlled camera motion, removing object motion and reducing recognition as confounds. We evaluate sixteen video-language models from nine architecture families in a zero-shot setting. No model exceeds 27.6% Exact Match. The dependence of accuracy on cardinality varies across models, ranging from monotonic decline to a non-monotonic peak-and-collapse pattern, which rules out increasing difficulty with cardinality as the dominant explanation of counting failure. Every model further concentrates its predictions on its own set of preferred integers, with accuracy peaks coinciding with these values and collapsing at adjacent counts. These findings highlight the need for balanced, diagnostic counting benchmarks to reliably assess this fundamental visual capability.