Metrics Failure in LLM-Based Code Vulnerability Repair: An Empirical Study and a Change-Aware Screen

📄 arXiv: 2609.26749v1 📥 PDF

作者: Om Nepal, Sushant Aryal, Oluseyi Olukola, Nick Rahimi

分类: cs.SE, cs.AI, cs.CR

发布日期: 2026-09-22

备注: 23 pages, 4 figures, 11 tables. Code and data: https://github.com/OmNepal/llm-vulnrepair-metrics

💡 一句话要点

提出变更感知评估以解决LLM代码漏洞修复中的指标失效问题

🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 大型语言模型 代码修复 安全漏洞 评估指标 变更感知 自动化测试 机器学习

📋 核心要点

  1. 现有的编译率指标在评估LLM生成的代码修复效果时存在显著不足，无法真实反映修复质量。
  2. 论文提出了diff_F1作为一种变更感知的筛选方法，旨在更准确地评估代码编辑的有效性。
  3. 通过对203个漏洞函数的实验，发现编译率与模型质量无关，而diff_F1能有效识别真实的部分编辑。

📝 摘要（中文）

大型语言模型（LLMs）在自动修复C/C++安全漏洞中应用日益广泛，而编译率常被用作进展的代理指标。本文认为编译率对于单函数漏洞修复是一个科学上不可靠的指标，并通过对203个来自Big-Vul的漏洞函数进行的五项控制实验来支持这一观点。研究发现编译率对改善生成代码的干预反应微弱，且受到评估工具和数据集伪影的主导影响。我们提出了diff_F1作为一种变更感知的筛选方法，能够更准确地评估编辑区域的质量，尽管它并不是修复质量的最终指标。我们的研究呼吁对LLM基础的漏洞修复进行变更感知和执行基础的评估。

🔬 方法详解

问题定义：本文旨在解决LLM在代码漏洞修复中使用编译率作为评估指标的失效问题。现有方法未能准确反映生成代码的修复质量，导致评估结果不可靠。

核心思路：论文提出了diff_F1作为一种变更感知的评估方法，专注于编辑区域的评分，以此来提高评估的准确性和可靠性。

技术框架：整体流程包括数据集准备、模型训练、生成代码修复、编译率评估和diff_F1评估。主要模块包括LLM模型、评估工具和数据集。

关键创新：最重要的创新在于引入了变更感知的评估指标diff_F1，能够有效区分真实修复和无效修复，克服了编译率的局限性。

关键设计：在实验中，diff_F1对无操作（no-op）给予零分，对某些基于删除的无效修复给予近零分，而对真实的部分编辑则给予相应的评分。

🖼️ 关键图片

img_0
img_1
img_2

📊 实验亮点

实验结果显示，编译率对模型质量的反应微弱，约64%的编译失败与模型无关。相比之下，diff_F1能够有效识别真实的部分编辑，提供更准确的评估，显示出其在漏洞修复中的应用潜力。

🎯 应用场景

该研究的潜在应用领域包括自动化代码审查、安全漏洞检测与修复等。通过提供更可靠的评估指标，能够提升LLM在实际应用中的有效性和安全性，推动软件开发的自动化进程。

📄 摘要（原文）

Large language models (LLMs) are increasingly applied to the automated repair of C/C++ security vulnerabilities, and compile rate is a commonly reported proxy for progress: whether the generated patch compiles. We argue that compile rate is a scientifically unreliable metric for single-function vulnerability repair, and we support this with five controlled experiments over 203 vulnerable functions from Big-Vul, three open-source code LLMs (350M to 6.7B parameters), and three prompting strategies. Compile rate (i) barely responds to an intervention that substantially improves the generated code; (ii) is dominated by evaluation-harness and dataset artifacts rather than model quality, with about 64% of compile failures not attributable to the model, a share that is nearly invariant across models; (iii) shifts by 1.8 to 2.7 times on identical patches under a single compiler-standard flag, with zero regressions; (iv) ranks the three models in the opposite order to reference-similarity metrics; and (v) rewards non-repairs when used as an optimization target, since a compiler-feedback loop raises compile rate while similarity to the human fix falls, with manual inspection finding deletion- and placeholder-style non-repairs among the newly compiling outputs. The natural fallback, whole-function CodeBLEU, also fails: an unchanged copy of the vulnerable input outscores every model. We also examine diff_F1, a change-aware screen that scores only the edited region. It gives exactly zero credit to a no-op and near-zero credit to some, though not all, of the deletion-based gaming patches we observed, while still crediting genuine partial edits, so it may serve as a cheap screen before deeper, execution-based analysis. It is not a repair-quality metric, and we report where it falls short. Our findings argue for change-aware, execution-grounded evaluation of LLM-based vulnerability repair.