A Lie Detector Test for Language Models: Reading Knowledge a Model Won't Reveal
作者: Hiskias Dingeto
分类: cs.AI
发布日期: 2026-09-18
💡 一句话要点
提出内部识别探测器以解决语言模型知识隐藏问题
🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)
关键词: 语言模型 知识隐藏 内部识别 隐蔽信息测试 模型评估 沙袋审计 知识去除
📋 核心要点
- 现有语言模型的评估方法无法有效判断模型是否隐藏知识，导致对其能力的误解。
- 本文提出的内部识别探测器（PIR）通过分析模型内部状态，识别其认可的答案，解决了知识隐藏问题。
- 在多个模型上，PIR实现了0.70至0.87的准确率，明显高于传统方法的基线，表现出色。
📝 摘要（中文）
大型语言模型可能持有未报告的知识。模型可能在能力评估中表现不佳，或根据其内部知识作出反应，而仅凭输出无法判断其是否隐藏答案。本文借鉴了隐蔽信息测试，通过向模型提出问题及候选答案，读取其内部状态以识别正确答案。提出的内部识别探测器（PIR）在多个模型中实现了0.70至0.87的准确率，显著高于基线水平，能够有效区分模型是否隐藏已知答案，支持沙袋审计和知识去除验证。
🔬 方法详解
问题定义：本文旨在解决大型语言模型在能力评估中可能隐藏知识的问题。现有方法无法有效区分模型是故意不回答还是确实不知道答案。
核心思路：提出的内部识别探测器（PIR）通过向模型展示问题及候选答案，读取其内部状态以识别模型认可的答案，从而揭示其隐藏的知识。
技术框架：PIR的整体架构包括问题呈现、候选答案生成和内部状态读取三个主要模块。模型根据输入的问题和候选答案进行内部处理，最终输出识别结果。
关键创新：PIR的最大创新在于其无参考性，不需要诚实的参考模型或标注的真相语料，能够独立评估模型的知识状态。这一方法与现有的黑箱行为分析方法本质上不同。
关键设计：在设计中，PIR采用了多种形式的隐蔽测试，包括提示性欺骗和训练沙袋等，确保在不同的隐蔽形式下仍能保持高识别率。
🖼️ 关键图片
📊 实验亮点
实验结果显示，PIR在八个不同模型上实现了0.70至0.87的平衡准确率，远高于0.28至0.40的未知项基线和0.25的随机率。即使在模型隐藏已知答案的情况下，识别率仍保持在0.85至0.93之间，显示出其强大的鲁棒性。
🎯 应用场景
该研究的潜在应用领域包括语言模型的能力评估、知识审计和模型透明性提升。通过有效识别模型的隐藏知识，PIR可以帮助开发者更好地理解和优化模型性能，确保其在实际应用中的可靠性和安全性。
📄 摘要（原文）
Large language models can hold knowledge they do not report. A model may sandbag on a capability evaluation, or answer against what it internally knows, and its outputs alone cannot tell whether it is hiding an answer or simply does not have one. We borrow the Concealed Information Test, a forensic method that identifies guilty knowledge by presenting a suspect with the true detail among plausible decoys and measuring a stronger response to the item they recognize. Our method, Probe of Internal Recognition (PIR), does the same inside a model. It presents a question with its candidate answers and reads, from the model's internal states, which candidate the model recognizes as correct. PIR is reference-free, needing no honest reference model and no labeled truth corpus. Across eight models from five families (Gemma, Qwen, Llama, Mistral, and Phi), PIR recovers the recognized answer at 0.70 to 0.87 balanced accuracy, well above the 0.28 to 0.40 unknown-item baseline and the 0.25 chance rate. It stays readable across every form of concealment we test, from prompted deception and trained sandbagging to external password-locked and circuit-broken checkpoints, with recognition between 0.85 and 0.93. When the model hides a known answer, recognition stays high. When unlearning removes the knowledge, recognition drops to the level of a question the model never knew. PIR therefore separates a model that will not answer from one that cannot, which supports sandbagging audits and unlearning verification. The signal is causal, adds information beyond black-box behavioral cues, and extends from multiple-choice questions to free-form generation.