FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation

📄 arXiv: 2310.03214v2 📥 PDF

作者: Tu Vu, Mohit Iyyer, Xuezhi Wang, Noah Constant, Jerry Wei, Jason Wei, Chris Tar, Yun-Hsuan Sung, Denny Zhou, Quoc Le, Thang Luong

分类: cs.CL

发布日期: 2023-10-05 (更新: 2023-11-22)

备注: Preprint, 26 pages, 10 figures, 5 tables; Added FreshEval

💡 一句话要点

提出FreshQA与FreshPrompt以解决大型语言模型动态适应性不足问题

🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 大型语言模型 动态问答 知识更新 搜索引擎增强 FreshQA FreshPrompt 事实性评估

📋 核心要点

  1. 现有的大型语言模型在面对快速变化的知识和虚假前提时表现不佳，缺乏动态更新能力。
  2. 本文提出FreshQA基准和FreshPrompt方法，通过搜索引擎获取最新信息来增强LLM的回答能力。
  3. 实验结果显示，FreshPrompt在多个基准上超越了竞争对手，显著提升了模型的正确性和减少了幻觉现象。

📝 摘要（中文）

大多数大型语言模型（LLMs）在训练后不会更新，因此缺乏动态适应不断变化世界的能力。本文详细研究了LLM生成文本的事实性，特别是在回答测试当前世界知识的问题时。我们引入了FreshQA，一个新颖的动态问答基准，涵盖多种问题和答案类型，包括需要快速变化的世界知识的问题。通过对多种LLM的评估，我们揭示了这些模型的局限性，并提出了FreshPrompt，一种简单的少量示例提示方法，显著提升了LLM在FreshQA上的表现。实验结果表明，FreshPrompt在性能上优于现有的搜索引擎增强提示方法。

🔬 方法详解

问题定义：本文旨在解决大型语言模型在动态知识更新方面的不足，尤其是在回答涉及快速变化信息的问题时，现有模型普遍存在事实性错误和幻觉现象。

核心思路：通过引入FreshQA基准和FreshPrompt方法，利用搜索引擎实时获取相关信息，增强LLM的回答能力，从而提高其在快速变化知识领域的表现。

技术框架：整体架构包括两个主要模块：FreshQA基准用于评估模型的回答能力，FreshPrompt方法用于生成增强的提示，结合检索到的信息来提升LLM的输出质量。

关键创新：最重要的创新在于FreshPrompt方法，它通过整合最新的检索信息，显著提升了LLM在FreshQA上的表现，与传统的静态训练模型形成鲜明对比。

关键设计：在FreshPrompt中，检索到的证据数量和顺序对模型生成答案的正确性有重要影响。此外，指导LLM生成简洁直接的答案有助于减少幻觉现象，相较于鼓励冗长的回答更为有效。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

实验结果显示，FreshPrompt在FreshQA基准上显著提升了LLM的表现，超越了包括Self-Ask在内的竞争方法，以及商业系统Perplexity.AI。具体而言，FreshPrompt在处理快速变化知识和虚假前提问题时，模型的正确性得到了显著改善。

🎯 应用场景

该研究的潜在应用领域包括智能问答系统、信息检索和知识管理等。通过提升大型语言模型的动态适应能力，能够更好地满足用户对实时信息的需求，具有重要的实际价值和未来影响。

📄 摘要（原文）

Most large language models (LLMs) are trained once and never updated; thus, they lack the ability to dynamically adapt to our ever-changing world. In this work, we perform a detailed study of the factuality of LLM-generated text in the context of answering questions that test current world knowledge. Specifically, we introduce FreshQA, a novel dynamic QA benchmark encompassing a diverse range of question and answer types, including questions that require fast-changing world knowledge as well as questions with false premises that need to be debunked. We benchmark a diverse array of both closed and open-source LLMs under a two-mode evaluation procedure that allows us to measure both correctness and hallucination. Through human evaluations involving more than 50K judgments, we shed light on limitations of these models and demonstrate significant room for improvement: for instance, all models (regardless of model size) struggle on questions that involve fast-changing knowledge and false premises. Motivated by these results, we present FreshPrompt, a simple few-shot prompting method that substantially boosts the performance of an LLM on FreshQA by incorporating relevant and up-to-date information retrieved from a search engine into the prompt. Our experiments show that FreshPrompt outperforms both competing search engine-augmented prompting methods such as Self-Ask (Press et al., 2022) as well as commercial systems such as Perplexity.AI. Further analysis of FreshPrompt reveals that both the number of retrieved evidences and their order play a key role in influencing the correctness of LLM-generated answers. Additionally, instructing the LLM to generate concise and direct answers helps reduce hallucination compared to encouraging more verbose answers. To facilitate future work, we release FreshQA at github.com/freshllms/freshqa and commit to updating it at regular intervals.