Rephrase, Augment, Reason: Visual Grounding of Questions for Vision-Language Models

📄 arXiv: 2310.05861v2 📥 PDF

作者: Archiki Prasad, Elias Stengel-Eskin, Mohit Bansal

分类: cs.CL, cs.AI, cs.CV, cs.LG

发布日期: 2023-10-09 (更新: 2024-04-02)

备注: ICLR 2024 camera-ready (23 pages), Code: https://github.com/archiki/RepARe

💡 一句话要点

提出RepARe以解决视觉语言模型中的问题表述不足

🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 视觉语言模型 问题重述 视觉问答 多模态学习 零样本学习

📋 核心要点

  1. 现有的视觉语言模型在处理输入时，表述不明确可能导致错误答案，影响模型性能。
  2. 论文提出的RepARe框架通过提取图像信息并修改问题表述，旨在减少输入的不明确性。
  3. 实验表明，RepARe在VQAv2、A-OKVQA和VizWiz等任务上分别提高了3.85%、6.41%和7.94%的零样本准确率。

📝 摘要（中文）

随着视觉语言任务的增多，结合大型语言模型与视觉编码器的视觉语言模型（LVLM）能够以零样本或少样本的方式处理这些任务。然而，输入的表述方式对模型的零样本性能影响显著，尤其是当输入表述不明确时，可能导致错误答案。为此，本文提出了Rephrase, Augment and Reason（RepARe）框架，通过提取图像的显著细节并对原始问题进行修改，旨在减少不明确性，从而提升模型性能。实验结果显示，RepARe在多个视觉问答任务上显著提高了零样本准确率，验证了其有效性。

🔬 方法详解

问题定义：本文旨在解决视觉语言模型在处理不明确输入时导致的性能下降问题，现有方法在这一方面存在明显不足，尤其是在复杂推理和语言歧义的情况下。

核心思路：RepARe框架的核心思想是通过提取图像的显著信息，利用LVLM作为图像描述生成器和推理器，来修改原始问题，从而提升模型的回答准确性。

技术框架：该框架主要包括三个模块：图像特征提取、问题重述生成和答案评分。首先提取图像中的关键信息，然后生成修改后的问题，最后通过LVLM的信心评分来选择最优问题。

关键创新：RepARe的创新之处在于其无梯度的框架设计，能够有效利用视觉信息来增强问题的表述，显著改善了传统方法在零样本任务中的表现。

关键设计：在设计中，使用了LVLM的信心评分作为无监督的选择标准，并通过对比不同问题重述的效果来优化模型的回答能力。

📊 实验亮点

实验结果显示，RepARe在VQAv2任务上提高了3.85%的零样本准确率，在A-OKVQA和VizWiz任务上分别提高了6.41%和7.94%。此外，使用金标准答案进行候选问题选择时，VQA准确率提升可达14.41%。

🎯 应用场景

该研究具有广泛的应用潜力，尤其是在需要视觉理解和语言推理的任务中，如智能问答系统、图像描述生成和人机交互等领域。通过提升视觉语言模型的性能，未来可以更好地支持复杂的多模态应用，推动相关技术的发展。

📄 摘要（原文）

An increasing number of vision-language tasks can be handled with little to no training, i.e., in a zero and few-shot manner, by marrying large language models (LLMs) to vision encoders, resulting in large vision-language models (LVLMs). While this has huge upsides, such as not requiring training data or custom architectures, how an input is presented to an LVLM can have a major impact on zero-shot model performance. In particular, inputs phrased in an underspecified way can result in incorrect answers due to factors like missing visual information, complex implicit reasoning, or linguistic ambiguity. Therefore, adding visually-grounded information to the input as a preemptive clarification should improve model performance by reducing underspecification, e.g., by localizing objects and disambiguating references. Similarly, in the VQA setting, changing the way questions are framed can make them easier for models to answer. To this end, we present Rephrase, Augment and Reason (RepARe), a gradient-free framework that extracts salient details about the image using the underlying LVLM as a captioner and reasoner, in order to propose modifications to the original question. We then use the LVLM's confidence over a generated answer as an unsupervised scoring function to select the rephrased question most likely to improve zero-shot performance. Focusing on three visual question answering tasks, we show that RepARe can result in a 3.85% (absolute) increase in zero-shot accuracy on VQAv2, 6.41%, and 7.94% points increase on A-OKVQA, and VizWiz respectively. Additionally, we find that using gold answers for oracle question candidate selection achieves a substantial gain in VQA accuracy by up to 14.41%. Through extensive analysis, we demonstrate that outputs from RepARe increase syntactic complexity, and effectively utilize vision-language interaction and the frozen LLM.