AusClinBench: a deterministically scored benchmark of large language models against current Australian clinical policy

📄 arXiv: 2610.31993v1 📥 PDF

作者: Daniel S. Epstein

分类: cs.CY, cs.CL

发布日期: 2026-10-22

备注: 19 pages, 3 figures. Technical report, version 1.4.1 of the benchmark. Not peer reviewed. Answer keys not yet independently reviewed. Results and methods: https://ausclinbench.vercel.app

💡 一句话要点

提出AusClinBench基准以评估澳大利亚临床政策下的语言模型

🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 语言模型 临床决策 医学基准 澳大利亚政策 模型评估 AI应用 医疗软件

📋 核心要点

  1. 现有医学基准主要针对其他地区，缺乏针对澳大利亚临床政策的评估工具，导致模型应用的透明度不足。
  2. 本文提出了AusClinBench基准，通过142个问题评估28个语言模型在澳大利亚临床政策下的表现，确保模型的应用符合当地规定。
  3. 实验结果显示，模型的盲测准确率在51.5%到92.7%之间，且86%的错误回答在提供指导条款后变为正确，显示出模型的潜在应用价值。

📝 摘要（中文）

AI模型已在澳大利亚临床软件中用于辅助临床决策和草拟记录，但通常未告知从业者所使用的模型。现有医学基准主要针对其他地区。本文基于澳大利亚指南和日程编写了142个问题，涵盖五个领域，并通过精确匹配进行评分。我们在2026年7月至10月间测试了28个模型，盲测准确率从51.5%到92.7%不等。结果显示，86%的错误回答在提供指导条款后变为正确。我们建议将模型视为临床产品的组成部分，并提出最低标准以满足本地规则。

🔬 方法详解

问题定义：本文旨在解决现有医学基准缺乏针对澳大利亚临床政策的评估工具的问题，现有方法未能有效评估模型在特定地区的适用性和准确性。

核心思路：通过编写142个基于澳大利亚临床指南的问题，结合精确匹配评分机制，评估语言模型在特定政策下的表现，从而提高模型的透明度和适用性。

技术框架：整体架构包括问题设计、模型测试和结果分析三个主要模块。首先设计问题，随后对28个模型进行盲测，最后分析模型在不同问题上的表现。

关键创新：最重要的创新在于将模型视为临床产品的组成部分，并提出最低标准要求，确保模型在实际应用中符合当地临床政策。

关键设计：在问题设计中，确保每个问题有一个可验证的答案，并通过提供指导条款来区分模型的记忆能力与应用能力。

📊 实验亮点

实验结果显示，28个模型的盲测准确率从51.5%到92.7%不等，且在791个模型-问题对中，86%的错误回答在提供指导条款后变为正确。这表明模型在特定政策下的表现具有显著的改进潜力。

🎯 应用场景

该研究的潜在应用领域包括临床决策支持系统和医疗软件开发。通过提供针对澳大利亚临床政策的评估基准，能够提升AI模型在实际医疗场景中的应用效果，确保其符合当地法规和标准，进而提高医疗服务的质量和安全性。

📄 摘要（原文）

AI models already inform clinical decisions and draft notes in Australian clinical software, usually without the practitioner being told which model is used. Existing medical benchmarks are written for other jurisdictions. We wrote 142 questions from Australian guidelines and schedules across five domains (PBS authority, MBS billing, Ahpra obligations, triage red flags, immunisation and screening), each with one verifiable answer scored by exact match. Each question ran blind and with the governing clause supplied, which separates failure to recall the current rule from failure to apply it. We tested 28 models between July and October 2026. Blind accuracy ranged from 51.5% to 92.7%, and the highest-scoring models could not be separated. Of 791 failed model-question pairs, 684 (86%) were wrong blind and right with the clause. Confident-error rates ranged from 6.6% to 47.8%. On five immunisation questions where the guideline gives no directive, 82% of answers stated one. On 20 Aboriginal and Torres Strait Islander health questions, models did no worse overall, but on a few the wrong answers gave the rule for the general population, an older rule or another country's rule. We propose treating the model as an ingredient of a clinical product, with a minimum standard on local rules that sits beside product evaluation. The keys are one general practitioner's and are not yet independently reviewed. The benchmark measures base models, not clinical products, and does not license any model for clinical use.