Progressive Evidence Refinement for Open-domain Multimodal Retrieval Question Answering

📄 arXiv: 2310.09696v1 📥 PDF

作者: Shuwen Yang, Anran Wu, Xingjiao Wu, Luwei Xiao, Tianlong Ma, Cheng Jin, Liang He

分类: cs.AI

发布日期: 2023-10-15

💡 一句话要点

提出渐进证据精炼策略以解决开放域多模态检索问答问题

🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture) 支柱五：交互与反应 (Interaction & Reaction) 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 多模态检索 问答系统 证据精炼 对比学习 特征提取 信息检索 深度学习

📋 核心要点

  1. 现有多模态检索问答模型在细粒度信息保留和证据与问题特征提取上存在显著不足。
  2. 提出的渐进证据精炼策略通过迭代检索和过滤优化证据选择，结合半监督对比学习扩展问题域。
  3. 在WebQA和MultimodalQA基准测试中，模型表现出色，验证了方法的有效性和优越性。

📝 摘要（中文）

预训练的多模态模型在基于检索的问答中取得了显著成功。然而，现有模型面临两个主要挑战：首先，压缩的证据特征输入导致细粒度信息的丢失；其次，证据与问题之间的特征提取存在差距，影响了模型从证据中有效提取关键特征。为此，本文提出了一个两阶段的框架，通过渐进证据精炼策略和半监督对比学习策略来优化证据选择和问题域扩展，最终通过多轮检索和问答策略来处理多模态输入。实验结果表明，该模型在WebQA和MultimodalQA基准测试中表现优异。

🔬 方法详解

问题定义：本文旨在解决开放域多模态检索问答中的证据特征压缩导致的信息丢失及证据与问题特征提取的差距问题。

核心思路：通过渐进证据精炼策略和半监督对比学习，优化证据选择和问题域扩展，以提高问答的准确性和效率。

技术框架：整体框架分为两个阶段：第一阶段为证据检索与精炼，第二阶段为基于多模态输入的问答生成，包含迭代检索、过滤和多轮问答模块。

关键创新：提出的渐进证据精炼策略和半监督对比学习是本研究的核心创新，显著提升了模型在特征提取和信息保留方面的能力。

关键设计：采用交叉模态注意力机制捕捉证据与问题之间的关联，设计了多轮检索问答策略以处理多模态输入，确保信息的有效利用。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

在WebQA和MultimodalQA基准测试中，模型表现出色，显著提升了问答准确率，相较于基线模型，性能提升幅度达到XX%。这些结果验证了所提出方法的有效性和实用性。

🎯 应用场景

该研究的潜在应用领域包括智能问答系统、信息检索、教育技术等。通过提高多模态问答的准确性和效率，能够为用户提供更为精准的信息服务，推动相关领域的技术进步和应用落地。

📄 摘要（原文）

Pre-trained multimodal models have achieved significant success in retrieval-based question answering. However, current multimodal retrieval question-answering models face two main challenges. Firstly, utilizing compressed evidence features as input to the model results in the loss of fine-grained information within the evidence. Secondly, a gap exists between the feature extraction of evidence and the question, which hinders the model from effectively extracting critical features from the evidence based on the given question. We propose a two-stage framework for evidence retrieval and question-answering to alleviate these issues. First and foremost, we propose a progressive evidence refinement strategy for selecting crucial evidence. This strategy employs an iterative evidence retrieval approach to uncover the logical sequence among the evidence pieces. It incorporates two rounds of filtering to optimize the solution space, thus further ensuring temporal efficiency. Subsequently, we introduce a semi-supervised contrastive learning training strategy based on negative samples to expand the scope of the question domain, allowing for a more thorough exploration of latent knowledge within known samples. Finally, in order to mitigate the loss of fine-grained information, we devise a multi-turn retrieval and question-answering strategy to handle multimodal inputs. This strategy involves incorporating multimodal evidence directly into the model as part of the historical dialogue and question. Meanwhile, we leverage a cross-modal attention mechanism to capture the underlying connections between the evidence and the question, and the answer is generated through a decoding generation approach. We validate the model's effectiveness through extensive experiments, achieving outstanding performance on WebQA and MultimodelQA benchmark tests.