Retrieve, Reproduce, Reveal: Dissecting Retrieval-Augmented Software Vulnerability Detection

📄 arXiv: 2609.37669v1 📥 PDF

作者: Sabrina Kaniewski, Tim Krämer, Julius Bächle, Markus Enzweiler, Michael Menth, Tobias Heer

分类: cs.SE, cs.AI

发布日期: 2026-09-29

💡 一句话要点

提出统一基准以解决RAG4SVD系统可重复性与可比性问题

🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 软件漏洞检测 检索增强生成 可重复性 开源系统 性能评估 组件分析 统一基准

📋 核心要点

  1. 现有的RAG4SVD系统多采用专有模型，导致可重复性和可比性不足，影响研究的开放性。
  2. 本文通过重现实验设置和建立统一基准，解决了RAG4SVD系统的可重复性和可比性问题。
  3. 实验结果显示，RAG4SVD的性能在开放权重评估下显著下降，且检索效果不足以保证检测的可靠性。

📝 摘要（中文）

检索增强生成（RAG）技术在基于大语言模型的软件漏洞检测中越来越受到重视，但现有的RAG4SVD系统多采用专有模型，影响了开放科学和可重复性。本文研究了六个开源RAG4SVD系统，通过在开放权重设置下重现实验设置，并使用统一基准进行评估，解决了可重复性和可比性挑战。结果表明，RAG4SVD系统的可重复性差异显著，性能依赖于所用模型，且仅依赖检索效果不足以确保可靠检测。这些发现促使我们在评估RAG4SVD时不仅考虑整体系统，还需关注各组件的表现。

🔬 方法详解

问题定义：本文旨在解决现有RAG4SVD系统在可重复性和可比性方面的挑战，现有方法多依赖专有模型，缺乏统一的评估标准。

核心思路：通过重现实验设置和建立统一基准，提供开放权重的评估环境，以便更好地比较不同RAG4SVD系统的性能。

技术框架：研究分析了六个开源RAG4SVD系统，采用统一的数据集、指标和开放权重模型进行评估，并对系统进行组件级分析，分解为输入抽象、知识检索和检测三个阶段。

关键创新：提出了在组件级别进行分析的思路，强调了各个阶段之间的对齐对RAG4SVD系统性能的重要性，突破了以往仅评估整体系统的局限。

关键设计：在实验中使用了统一的数据集和指标套件，确保了评估的一致性，同时分析了不同模型对系统性能的影响，揭示了检索效果与检测可靠性之间的关系。

🖼️ 关键图片

img_0
img_1
img_2

📊 实验亮点

实验结果表明，在统一基准下，已发布的RAG4SVD系统性能无法转移，且在开放权重评估中，性能显著依赖于所用模型。此外，尽管检索效果接近最优，系统的配对准确率仍然较低，仅为0.51，显示出检索效果不足以保证检测的可靠性。

🎯 应用场景

该研究为软件漏洞检测领域提供了新的评估框架，能够促进不同RAG4SVD系统之间的比较与改进。未来，该方法可以应用于其他基于检索的生成模型，推动相关领域的研究进展和技术应用。

📄 摘要（原文）

Retrieval-Augmented Generation (RAG) is increasingly used to enhance Large Language Model (LLM)-based software vulnerability detection by grounding predictions in retrieved vulnerability knowledge, such as vulnerability reports. However, existing RAG-based software vulnerability detection (RAG4SVD) systems are often evaluated using proprietary models, which challenges open science and reproducibility. Further, studies use different datasets, custom knowledge bases, different backbone models, and diverse metrics, which hinders meaningful cross-system comparison. In this work, we study six open-source RAG4SVD systems and address these reproducibility and comparability challenges through (i) reproduction of their experimental settings under an open-weight setting, and (ii) a unified benchmark using a common dataset, metric suite, and pool of open-weight models. Further, RAG4SVD systems typically consist of multiple components, yet are often evaluated only as a whole system, i.e., end-to-end. Therefore, we perform (iii) a component-level analysis that decomposes representative RAG4SVD pipelines into input abstraction, knowledge retrieval, and detection. Our results demonstrate that reproducibility varies substantially across systems. Under the presented unified benchmark, published RAG4SVD performance does not transfer under a controlled open-weight evaluation and depends strongly on the used model. The component analysis shows that effective RAG4SVD depends on the alignment between pipeline stages. For example, oracle knowledge raises retrieval to near-optimal, yet performance remains low (0.51 pairwise accuracy), demonstrating that retrieval effectiveness alone is insufficient for reliable detection. These findings motivate evaluating RAG4SVD not only end-to-end, but at the level of pipeline components, and provide a basis for more standardized, RAG-aware evaluation practices.