LLMaAA: Making Large Language Models as Active Annotators

📄 arXiv: 2310.19596v2 📥 PDF

作者: Ruoyu Zhang, Yanzeng Li, Yongliang Ma, Ming Zhou, Lei Zou

分类: cs.CL, cs.AI

发布日期: 2023-10-30 (更新: 2023-10-31)

备注: Findings of EMNLP 2023 camera ready

💡 一句话要点

提出LLMaAA以提高大语言模型的主动标注效率

🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 主动学习 大语言模型 伪标签 自然语言处理 命名实体识别 关系提取 数据生成

📋 核心要点

  1. 现有的监督学习方法对高质量标注数据的需求极高，获取这些数据的成本高昂且效率低下。
  2. LLMaAA通过将大语言模型作为主动标注者，利用未标注数据进行高效标注，优化了标注和训练过程。
  3. 在命名实体识别和关系提取任务中，LLMaAA的模型在仅有数百个标注样本的情况下超越了传统教师模型，展现出更高的成本效益。

📝 摘要（中文）

在自然语言处理（NLP）中，现有的监督学习方法对高质量标注数据的需求极高，获取这些数据的成本也很高。尽管大语言模型（LLMs）在少量样本学习中表现优异，但其生成的数据质量往往较低，且需要大量标注数据才能达到满意的性能。为此，本文提出LLMaAA，将LLMs作为主动标注者，利用未标注数据进行高效标注。通过优化标注和训练过程，LLMaAA在命名实体识别和关系提取等任务中表现出色，能够在仅有数百个标注样本的情况下超越传统方法，具有更高的成本效益。

🔬 方法详解

问题定义：本文旨在解决现有监督学习方法对大量高质量标注数据的依赖问题，现有方法在数据获取上成本高且效率低下。

核心思路：LLMaAA的核心思路是将大语言模型作为主动标注者，利用未标注数据进行高效标注，从而减少对标注数据的需求。

技术框架：LLMaAA的整体架构包括两个主要模块：标注模块和训练模块。标注模块利用LLMs生成伪标签，训练模块则对这些伪标签进行优化。

关键创新：LLMaAA的创新在于将LLMs置于主动学习循环中，利用k-NN示例作为上下文示例，并采用可学习的样本权重进行训练，从而提高了标注的效率和可靠性。

关键设计：在参数设置上，LLMaAA通过从小型示例池中抽取k-NN示例，并对训练样本分配可学习的权重，优化了标注和训练过程。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

在实验中，LLMaAA在命名实体识别和关系提取任务上表现出色，能够在仅有数百个标注样本的情况下超越传统教师模型，展现出显著的性能提升，具有更高的成本效益。

🎯 应用场景

LLMaAA的研究成果在自然语言处理领域具有广泛的应用潜力，尤其是在需要大量标注数据的任务中，如命名实体识别和关系提取。通过降低标注成本，LLMaAA能够帮助研究人员和企业更高效地构建高质量的数据集，推动相关技术的发展。

📄 摘要（原文）

Prevalent supervised learning methods in natural language processing (NLP) are notoriously data-hungry, which demand large amounts of high-quality annotated data. In practice, acquiring such data is a costly endeavor. Recently, the superior few-shot performance of large language models (LLMs) has propelled the development of dataset generation, where the training data are solely synthesized from LLMs. However, such an approach usually suffers from low-quality issues, and requires orders of magnitude more labeled data to achieve satisfactory performance. To fully exploit the potential of LLMs and make use of massive unlabeled data, we propose LLMaAA, which takes LLMs as annotators and puts them into an active learning loop to determine what to annotate efficiently. To learn robustly with pseudo labels, we optimize both the annotation and training processes: (1) we draw k-NN examples from a small demonstration pool as in-context examples, and (2) we adopt the example reweighting technique to assign training samples with learnable weights. Compared with previous approaches, LLMaAA features both efficiency and reliability. We conduct experiments and analysis on two classic NLP tasks, named entity recognition and relation extraction. With LLMaAA, task-specific models trained from LLM-generated labels can outperform the teacher within only hundreds of annotated examples, which is much more cost-effective than other baselines.