RAIM: Robust Aggregation of Inexpensive Models for Hallucination Detection

📄 arXiv: 2609.39229v1 📥 PDF

作者: Elia Onofri, Roberto Di Pietro

分类: cs.CL, cs.AI, cs.LG

发布日期: 2026-09-30

备注: 49 pages, 23 tables, 10 figures. Code and data: https://github.com/eOnofri04/raim-analysis and https://github.com/eOnofri04/raim-verdicts

💡 一句话要点

提出RAIM以实现低成本模型的聚合用于幻觉检测

🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 幻觉检测 模型聚合 逻辑回归 开放权重模型 高通量监测

📋 核心要点

  1. 现有方法依赖昂贵的前沿模型进行幻觉检测，难以实现高效监测。
  2. RAIM通过聚合多个廉价模型，结合逻辑回归和可接受性测试，提升检测性能。
  3. 实验结果显示，RAIM在多个基准上保持93%的Cohen's κ，且与专用检测器的准确率相近。

📝 摘要（中文）

自动评估文本的真实性日益依赖大型语言模型作为评判者，但最可靠的评判者往往是昂贵的专有前沿模型，不适合高通量监测。本文研究了是否可以通过聚合多个廉价的开放权重模型（4-9B）来替代前沿模型，并提出了RAIM聚合方案，该方案对成员的相关错误具有鲁棒性。RAIM结合了交叉拟合的堆叠逻辑回归和可接受性测试，能够识别何时聚合模型能够超越其最佳成员并接近前沿模型。实验证明，RAIM在多个基准测试中表现出色，且成本仅为前沿模型推理价格的六十四分之一。

🔬 方法详解

问题定义：本文旨在解决依赖昂贵前沿模型进行幻觉检测的问题，现有方法在高通量监测中效率低下且成本高昂。

核心思路：RAIM通过聚合多个廉价的开放权重模型，利用交叉拟合的堆叠逻辑回归和可接受性测试，识别何时聚合能够提升性能。

技术框架：RAIM的整体架构包括多个廉价模型的输出聚合、交叉拟合逻辑回归的训练过程，以及基于模型输出的可接受性测试模块。

关键创新：RAIM的创新在于其对成员相关错误的鲁棒性，能够在多个模型之间有效聚合，超越单一最佳模型的性能。

关键设计：RAIM的设计包括对50-100个标注记录的单次领域内校准，使用逻辑回归作为聚合方法，并在多个基准测试中进行验证。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

RAIM在与Claude Sonnet的对比中，保持了93%的Cohen's κ，且仅平均损失2.9个平衡准确率点。此外，RAIM在多个基准测试中表现优于专用检测器，显示出其强大的竞争力。

🎯 应用场景

该研究的潜在应用领域包括文本生成、内容审核和信息真实性检测等。通过使用RAIM，组织可以在降低成本的同时，保持对文本真实性的高效监测，具有重要的实际价值和广泛的应用前景。

📄 摘要（原文）

Automatic evaluation of faithfulness increasingly relies on a large language model acting as a judge, yet the most reliable judges are proprietary frontier models, costly and ill-suited to high-throughput monitoring. We investigate whether a panel of cheap open-weight judges (4--9B) can be aggregated to stand in for a frontier one, what the substitution sacrifices, and when it is worth making. We propose RAIM, an aggregation scheme robust to the members' correlated errors, coupling a cross-fitted stacked logistic regression with an admissibility test that, read from the members' own outputs, identifies when aggregating them improves on their best member and stays within reach of the frontier judge. We instantiate RAIM with ten judges from disjoint families across eight faithfulness benchmarks. Against Claude Sonnet, the panel retains a median 93% of its Cohen's $κ$ and gives up only 2.9 points of balanced accuracy on average; read as paired differences, it clearly improves on one benchmark and clearly worsens on three (only two by a non-negligible margin), leaving four unresolved. At a sixty-fourth of the frontier's inference price, the operative expense is a one-time in-domain calibration on 50--100 labelled records. The panel is also competitive with purpose-trained detectors on their home benchmarks (within 1.3 accuracy points of GPT-4o and 1.9 of the LLM-AggreFact leader), and beats the strongest one we reran by 6 points on our grounded sets. Whether aggregation pays depends on the members themselves: where several capable members err on different items, the panel improves on its best judge and approaches the frontier; where one dominates, the stacker recovers the leader, and only there does the frontier remain materially ahead. Both conditions are read off the calibration set at no further cost, so a cheap panel can stand in for a frontier one wherever this audit admits it.