MM-BigBench: Evaluating Multimodal Models on Multimodal Content Comprehension Tasks
作者: Xiaocui Yang, Wenfang Wu, Shi Feng, Ming Wang, Daling Wang, Yang Li, Qi Sun, Yifei Zhang, Xiaoming Fu, Soujanya Poria
分类: cs.CL, cs.MM
发布日期: 2023-10-13
备注: Underview
🔗 代码/项目: GITHUB
💡 一句话要点
提出MM-BigBench以解决多模态内容理解评估问题
🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)
📋 核心要点
- 现有的多模态大型语言模型评估主要集中在单模态内容，缺乏对多模态内容理解的全面评估。
- 本文提出MM-BigBench评估框架，结合多种指标，全面评估多模态内容理解任务中的模型表现。
- 在14个多模态数据集上评估20种语言模型，提出适应性指标，揭示模型与指令之间的适应性。
- method_zh
📝 摘要（中文）
多模态大型语言模型（MLLMs）的流行引发了对这些模型评估的研究热潮。然而，现有的评估研究主要集中在单模态（视觉）内容的理解与推理上，忽视了多模态（视觉-语言）内容理解的性能评估。本文提出了一种名为MM-BigBench的综合评估框架，涵盖多种指标，以全面评估不同模型和指令在多样化多模态内容理解任务中的表现。我们的工作补充了对MLLMs在多模态理解任务中的研究，实现了更全面的评估。我们对20种语言模型（其中14种为MLLMs）在14个多模态数据集上的6个任务进行了评估，并提出了适应性指标来量化模型与指令之间的适应性。
🖼️ 关键图片
📄 摘要（原文）
The popularity of multimodal large language models (MLLMs) has triggered a recent surge in research efforts dedicated to evaluating these models. Nevertheless, existing evaluation studies of MLLMs primarily focus on the comprehension and reasoning of unimodal (vision) content, neglecting performance evaluations in the domain of multimodal (vision-language) content understanding. Beyond multimodal reasoning, tasks related to multimodal content comprehension necessitate a profound understanding of multimodal contexts, achieved through the multimodal interaction to obtain a final answer. In this paper, we introduce a comprehensive assessment framework called MM-BigBench, which incorporates a diverse range of metrics to offer an extensive evaluation of the performance of various models and instructions across a wide spectrum of diverse multimodal content comprehension tasks. Consequently, our work complements research on the performance of MLLMs in multimodal comprehension tasks, achieving a more comprehensive and holistic evaluation of MLLMs. To begin, we employ the Best Performance metric to ascertain each model's performance upper bound on different datasets. Subsequently, the Mean Relative Gain metric offers an assessment of the overall performance of various models and instructions, while the Stability metric measures their sensitivity. Furthermore, previous research centers on evaluating models independently or solely assessing instructions, neglecting the adaptability between models and instructions. We propose the Adaptability metric to quantify the adaptability between models and instructions. Our paper evaluates a total of 20 language models (14 MLLMs) on 14 multimodal datasets spanning 6 tasks, with 10 instructions for each task, and derives novel insights. Our code will be released at https://github.com/declare-lab/MM-BigBench.