Trusting the Tool: Asymmetric Deference of LLM Vulnerability Auditors to Static-Analysis Evidence

📄 arXiv: 2610.32876v1 📥 PDF

作者: ZhiHong Liu, Shuyao Gao

分类: cs.SE, cs.CR, cs.LG

发布日期: 2026-10-23

备注: 33 pages, 9 figures, 5 tables. Submitted to IEEE Access. Code: https://github.com/lzh0082-cloud/CODE-DETECTION

💡 一句话要点

提出静态分析与大型语言模型结合的审计方法以解决证据可信度问题

🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 大型语言模型 静态分析 脆弱性审计 证据可信度 软件安全

📋 核心要点

  1. 现有方法未能有效区分真实与虚假证据，导致模型在审计脆弱性时的准确性受到影响。
  2. 论文提出了一种新的审计方法，通过构建内部分析器并在多种证据条件下进行实验，验证模型对证据的依赖性。
  3. 实验结果表明，虚假保证显著降低了真实脆弱性的检测率，而真实证据的提升效果微弱，提示需要在模型使用前验证工具的保证。

📝 摘要（中文）

本研究探讨了将大型语言模型（LLMs）与静态分析结合的管道，分析模型是否能够区分真实与虚假证据，以及其审查是否依赖于证据的声明。我们构建了一个针对C/C++的内部分析器，生成证明义务，并在六种不同证据条件下审计了PrimeVul中的脆弱和修补函数。研究涵盖了916对数据，结果显示真实证据对准确率的提升微乎其微，而虚假证据的影响则表现出不对称性，虚假保证显著降低了真实脆弱性的检测率。最后，建议在模型依赖工具保证之前，先验证其有效性。

🔬 方法详解

问题定义：本研究旨在解决大型语言模型在静态分析中对证据的信任问题，现有方法未能有效评估模型对真实与虚假证据的区分能力。

核心思路：通过构建一个针对C/C++的内部分析器，生成不同条件下的证据，评估模型在审计脆弱性时的表现，特别关注虚假保证与虚假警报的影响。

技术框架：整体流程包括数据生成、模型配置、证据条件设定和审计结果分析。分析器生成证明义务，模型根据不同证据进行审计，最终评估准确性。

关键创新：本研究的创新在于系统性地评估了虚假证据对模型判断的影响，尤其是虚假保证与虚假警报的非对称性，揭示了模型对证据的依赖性。

关键设计：实验中使用了916对数据，涵盖11种模型配置和16,572次审计，采用了多种证据条件，确保结果的可靠性和有效性。

📊 实验亮点

实验结果显示，真实证据仅提升了0.02的准确率，而虚假保证则使真实脆弱性检测率下降了多达19个百分点。相较之下，虚假警报对准确率的影响不显著，且被质疑的频率更高。这些发现强调了在模型依赖工具保证之前进行验证的重要性。

🎯 应用场景

该研究的潜在应用领域包括软件安全性审计、漏洞检测和代码分析等。通过提高模型对证据的判断能力，可以有效提升静态分析工具的可靠性，减少误报和漏报，进而增强软件系统的安全性和稳定性。

📄 摘要（原文）

Pipelines that combine large language models (LLMs) with static analysis let the model consume the analyser's output as evidence, but whether the model distinguishes true from false evidence, and whether its scrutiny depends on what the evidence claims, has not been measured. We built an intraprocedural analyser for C/C++ that produces proof obligations and audited vulnerable and patched functions from PrimeVul under six evidence conditions that differ only in the notes, including fabricated assurances that a missing check exists and fabricated alarms that an added check is missing. The study covers 916 pairs in four disjoint sets, eleven model configurations from four vendors and 16,572 audits; no model is trained or fine-tuned. True evidence barely improved accuracy (+0.02). False evidence acted asymmetrically: relative to true notes about the same operations, false assurances lowered the detection of real vulnerabilities by up to 19 points, whereas false alarms never significantly lowered accuracy and were disputed far more often. An internally registered replication on seven new models confirmed that false assurances move verdicts towards their claim more than false alarms do (mean asymmetry +0.16, 95% CI +0.10 to +0.24). Instructions in the audit prompt did not reliably reduce deference; the best one failed to replicate. Verifying each claim in a separate call and showing the refuted claims raised accuracy under false assurances by 12 points in a pre-specified test (p=0.002) without loss under true evidence; for a second model the gain was smaller and not significant. Pipelines should therefore verify a tool's guarantees before the model relies on them, and studies should measure deference in a direction-aware way.