BITEM at the NTCIR-19 R2C2 Task: Predicting Confidence from Agentic RAG Pipeline Signals
作者: Julien Knafou, Luc Mottin, Alexandre Flament, Paul van Rijen, Esteban Gaillac, Patrick Ruch
分类: cs.CL, cs.AI, cs.IR
发布日期: 2026-09-29
备注: 8 pages. Participant paper for the NTCIR-19 R2C2 task
💡 一句话要点
提出基于代理RAG管道信号的置信度预测方法
🎯 匹配领域: 支柱六：视频提取与匹配 (Video Extraction)
关键词: 多跳推理 置信度评估 信息检索 问答系统 自动化知识获取
📋 核心要点
- 现有方法在多跳推理和证据检索中面临挑战，尤其是在如何有效评估答案的置信度方面。
- 论文提出了一种基于代理RAG管道的解决方案，通过协调器管理证据检索和答案提交，确保高效性和准确性。
- 实验结果显示，系统在准确率和HMR评分上均有显著提升，提出的accHMR指标为未来研究提供了新的方向。
📝 摘要（中文）
BITEM团队在NTCIR-19 R2C2任务中，采用单一的代理管道进行证据搜索、阅读和记录。该管道通过一个协调器管理记录，并在经过蕴涵级联检查后，决定何时提交答案。每个问题运行三到四次，从未见过的语料库中检索信息。答案的置信度由协调器计算，而非模型自身。实验结果显示，该方法在多跳和后处理问题上表现优异，最终获得了较高的准确率和HMR评分，提出了accHMR作为新的评估指标，未来可通过模型拟合进一步提升性能。
🔬 方法详解
问题定义：本论文旨在解决在多跳推理任务中，如何有效评估和提升答案置信度的问题。现有方法往往依赖模型自评，导致置信度评估不准确。
核心思路：论文的核心思路是通过一个协调器来管理证据的检索和答案的提交，而不是让模型自行评估置信度。这种设计确保了置信度的计算基于实际检索到的证据，避免了模型自评的偏差。
技术框架：整体架构包括一个代理管道，负责搜索和读取电影语料库中的证据，协调器则负责记录和判断何时提交答案。每个问题会被多次运行，以确保从未见过的语料中检索信息。
关键创新：论文的关键创新在于引入了accHMR指标，结合准确率和HMR评分，提供了一种新的评估方式，能够更好地反映系统的实际表现。与现有方法相比，该指标更能体现置信度与答案正确性的关系。
关键设计：在设计上，系统通过设置特定的规则来提升置信度评估的准确性，避免了额外的模型调用和检索，优化了整体性能。
🖼️ 关键图片
📊 实验亮点
实验结果显示，系统在准确率上达到了0.9219，排名第六，而通过优化规则后，准确率提升至0.9375，排名第五。HMR评分也从0.4915提升至0.6985，显示出显著的性能改进。
🎯 应用场景
该研究的潜在应用领域包括信息检索、问答系统和自动化知识获取等。通过提升答案的置信度评估，能够在实际应用中提高系统的可靠性和用户体验，未来可在更复杂的推理任务中发挥重要作用。
📄 摘要（原文）
The BITEM team entered both subtasks of the NTCIR-19 R2C2 task with a single agentic pipeline, in which a model searches, reads and records evidence over a movie corpus while an orchestrator holds the record and rules on what may be submitted. A claim is admitted only once an entailment cascade has checked it against the passage it cites, and an answer is released only once enough checked evidence stands behind it. Each question is run three or four times, every pass retrieving from a corpus stripped of what the earlier passes have already seen. The confidence filed with each answer is computed by the orchestrator from what the run leaves behind and is never asked of the model, which is offered no way to rate itself. The two retrieval runs placed 4th and 5th of 22, pooling the passes was worth 0.0709 nDCG@20, and the gain was largest on the multi-hop and post-processing-heavy questions, where the organisers rank the pooled run top of the field. Sixteen of the 25 answer runs were built on passages these two runs supplied, 12 of them filed by other teams. HMR rewards a system whose confidence is high where it answers right and low where it answers wrong. The pipeline reached an accuracy of 0.9219, 6th of 25, while the confidence filed with those answers gave an HMR of 0.4915, 13th. A few rules crafted over those same recorded signals, with no further model call and no further retrieval, raise that to an accuracy of 0.9375, 5th, and an HMR of 0.6985, 9th. Ranking on HMR alone can reward a system for answering wrongly with low confidence, so we propose accHMR, the accuracy multiplied by HMR, which reports the reward in proportion to the accuracy, and on which the revised rules would have scored 0.6549, 5th. For future work, fitting a model on the numbers the pipeline already produces, rather than writing such rules by hand, would be a real step forward.