JuryFlow: Disagreement-Guided Human-in-the-Loop Multi-Agent Evaluation
作者: Mufeng Yang, Junwei Yu, Yepeng Ding
分类: cs.CL, cs.HC
发布日期: 2026-09-30
备注: 9 pages, 5 figures, 5 tables. To appear in Proceedings of the 14th International Conference on Human-Agent Interaction (HAI '26), November 16-19, 2026, Osaka, Japan
💡 一句话要点
提出JuryFlow框架以解决评估中评审者分歧问题
🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)
关键词: 评估框架 人机协作 分歧引导 多代理评估 自动化评审 大型语言模型 自我优化
📋 核心要点
- 现有评估方法在评审者之间存在分歧时，往往通过多数投票来处理，导致信息损失和不准确性。
- JuryFlow框架通过构建分歧图，允许评审者对每个主张进行独立评判，并通过人类引导选择解决分歧。
- 实验结果显示，JuryFlow在MT-Bench和LLMBar上显著提高了评估一致性，相较于传统方法有明显提升。
📝 摘要（中文）
大型语言模型（LLMs）越来越多地被用作AI生成内容的自动评审者，但单一评审者的可靠性不足，多个评审者的投票机制在存在分歧时也无法有效解决问题。本文提出了JuryFlow，一个基于分歧引导的、人机协作的多代理评估框架，旨在将评审者之间的分歧视为重要信号，而非噪声。JuryFlow将候选响应分解为原子性主张，采用异构评审者对每个主张进行评判，并构建分歧图。人类评审者作为结构引导者，通过最小干预选择需要解决的分歧，从而实现逐步自我优化的评估过程。实验表明，JuryFlow在MT-Bench和LLMBar上相较于单评审和多数投票基线显著提高了与金标准标签的一致性。
🔬 方法详解
问题定义：本文旨在解决评审者之间分歧导致的评估不准确问题。现有方法如单一评审和多数投票无法有效处理评审者的意见不一致，导致信息损失和评估结果的不可靠性。
核心思路：JuryFlow框架将评审者之间的分歧视为重要信号，通过构建分歧图来引导评审过程。人类评审者不再仅仅是标签提供者，而是作为结构引导者，选择需要解决的分歧，从而提高评估的准确性和一致性。
技术框架：JuryFlow的整体架构包括分解候选响应为原子性主张、异构评审者对每个主张进行评判、构建分歧图、以及通过人类引导进行分歧解决。分歧图的节点由评判熵评分，边则表示主张之间的结构相似性。
关键创新：JuryFlow的主要创新在于将评审者的角色重新定义为结构引导者，并通过分歧图和焦点重评估机制实现自我优化。这一方法与传统的多数投票机制本质上不同，能够有效利用评审者的分歧信息。
关键设计：在设计上，JuryFlow采用了熵排名来选择焦点主张，确保评审者的分歧能够被有效解决。通过最小干预的方式，评审者可以对特定主张进行重新评估，进而将修正结果传播到相关的历史案例中。该框架还实现了封闭循环的评分标准引导。
🖼️ 关键图片
📊 实验亮点
实验结果表明，JuryFlow在MT-Bench和LLMBar数据集上，相较于单评审和多数投票基线，显著提高了与金标准标签的一致性，具体提升幅度未知。通过消融实验，进一步验证了分歧引导重评估、传播和评分标准引导的贡献。
🎯 应用场景
JuryFlow框架具有广泛的应用潜力，特别是在需要高质量评估的领域，如内容审核、AI生成文本的质量评估等。通过有效利用评审者的分歧信息，该框架能够提高评估的一致性和准确性，未来可能推动自动化评估系统的发展。
📄 摘要（原文）
Large language models (LLMs) are increasingly deployed as automated judges for AI-generated content, yet a single judge is unreliable and even a panel of judges leaves a hard residue: when judges disagree, majority voting discards the conflict instead of resolving it. We present JuryFlow, a disagreement-guided, human-in-the-loop multi-agent evaluation framework that treats inter-judge disagreement not as noise to be averaged away, but as a precise, claim-level signal indicating where an evaluation is uncertain. JuryFlow decomposes each candidate response into atomic claims, has a panel of heterogeneous judges assign per-claim verdicts, and builds a disagreement graph whose nodes are scored by verdict entropy and whose edges encode structural similarity between claims. A human acts as a structural guide, selecting which disagreement to resolve through a single, minimal intervention rather than re-labeling the response, after which the focal claim is re-evaluated, the correction propagates along graph edges and to historically similar cases, and is crystallized into reusable rubric entries that all judges inherit, making the evaluator progressively self-refining. To enable large-scale, reproducible benchmarking without human studies, we evaluate JuryFlow in an automatic configuration in which focal selection is made by entropy ranking. On MT-Bench and LLMBar, JuryFlow improves agreement with gold labels over single-judge and majority-vote panel baselines, and ablations isolate the contributions of disagreement-targeted re-evaluation, propagation, and rubric induction. We contribute (1) a human-in-the-loop paradigm that recasts the human from labeler to structural guide, (2) the JuryFlow framework operationalizing it through a disagreement graph, focal re-evaluation, and closed-loop rubric induction, and (3) an evaluation protocol with ablations that isolate where the gains originate.