Making every stage of medical device document assessment with large language models deterministic
作者: Ryosuke Suzuki
分类: cs.CL, cs.IR
发布日期: 2026-10-15
备注: 30 pages, 3 figures, 3 tables; supplementary tables S1-S15 and two supplementary notes included
💡 一句话要点
提出确定性评估框架以解决医疗设备文档评估不一致问题
🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)
关键词: 医疗设备 文档评估 确定性评估 大型语言模型 合规性判断 光学字符识别 数据处理管道
📋 核心要点
- 现有的医疗设备文档评估方法存在评估结果不一致的问题，影响合规性判断的可靠性。
- 本文提出了一种确定性评估框架，通过构建稳定的处理管道，确保每个阶段的输出一致性。
- 实验结果表明，构建的管道在不同条件下均能返回相同的输出，验证了端到端确定性的可行性。
📝 摘要（中文）
为了证明医疗设备符合规定，专家需要判断文档和记录是否满足每个要求，并记录判断结果以供他人验证。本文展示了大型语言模型在初步评估中的应用，包括标签、理由和引用证据。通过对十四个商业部署的模型进行重复请求，发现每个部署在至少一个请求中返回了不同的评估结果。尽管减少推理并未降低评估结果的变化，构建的确定性管道在每个阶段均返回字节相同的输出，证明了端到端的确定性是可实现的，为审计和变更控制提供了可测试的基础。
🔬 方法详解
问题定义：本文旨在解决医疗设备文档评估中存在的结果不一致性问题。现有方法在不同模型和请求下，评估结果常常不一致，影响合规性判断的可靠性。
核心思路：论文提出通过构建一个确定性评估管道，确保在文本提取、光学字符识别（OCR）、嵌入、索引、检索和推理等每个阶段都能返回一致的输出，从而提高评估的可靠性。
技术框架：整体架构包括多个模块：文本提取、OCR、数据分块、嵌入、索引、检索和推理。每个模块都经过精心设计，以确保在不同条件下的输出一致性。
关键创新：最重要的创新点在于构建了一个在每个阶段都能保持确定性的处理管道，解决了现有方法中因模型和请求变化导致的评估结果不一致的问题。
关键设计：在设计中，特别关注了OCR、嵌入和近似检索的服务一致性，确保在不同的GPU模型和推理引擎调度设置下，输出结果保持不变。
🖼️ 关键图片
📊 实验亮点
实验结果显示，构建的确定性管道在不同的运行条件下均能返回字节相同的输出，验证了端到端确定性的可行性。这一成果为医疗设备文档评估提供了一个可靠的基础，确保了合规性判断的一致性。
🎯 应用场景
该研究的潜在应用领域包括医疗设备的合规性评估、监管审计和质量控制等。通过提供一个确定性的评估框架，能够提高医疗设备文档评估的可靠性和透明度，促进医疗行业的标准化和合规性管理。
📄 摘要（原文）
To show that a medical device complies with regulation, experts judge whether documents and records meet each requirement and record the judgment for others to verify. For each assessment item that experts define, large language models could prepare a preliminary assessment: a label such as Satisfied or Not satisfied, a reason and quoted evidence. We sent fourteen commercial deployments of such models (seven models from three providers, through six routes) the same request repeatedly, with the deterministic sampling settings each accepted; every deployment returned differing assessments for at least one request, and in twelve the label changed. Reducing reasoning where the interface allows it did not reduce the variation, and prompts of retrieved passages removed it only for the smallest models. The same open-weight model was reproducible or not depending only on how it was served. A pipeline we built to be deterministic at every stage -- text extraction and optical character recognition (OCR), chunking, embedding, indexing, retrieval and inference -- returned byte-identical output at each stage across reruns, interruptions, edits, machines of one graphics processing unit (GPU) model and days, for blood glucose meter and infusion pump documents. OCR, embedding and approximate retrieval varied when their own serving was not made invariant, and outputs changed with the GPU model and with the inference engine's scheduling settings, which therefore belong in the recorded configuration. End-to-end determinism is achievable and gives a testable basis for audit and change control.