Better Understanding, Better Fixes? A Study of Hallucination in LLM-based Automated Program Repair
作者: Xuemeng Cai, Jiakun Liu, Linhan Yang, Wei Ma, Lingxiao Jiang
分类: cs.SE, cs.AI
发布日期: 2026-09-04
💡 一句话要点
深入分析LLM在自动程序修复中的幻觉现象
🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)
关键词: 自动程序修复 大型语言模型 幻觉现象 修复效果 中间工件分析
📋 核心要点
- 现有的自动程序修复方法在评估上过于关注结果，缺乏对幻觉现象的深入分析，导致修复效果不理想。
- 本文通过多层次分析修复过程中的幻觉现象，提出了对最终补丁和中间工件的理解与评估方法。
- 实验结果显示，修复和理解幻觉普遍存在，生成补丁的有效性较低，且中间工件的准确性与修复成功率并不总是正相关。
📝 摘要（中文）
大型语言模型（LLMs）在自动程序修复（APR）领域取得了显著进展，但现有评估主要集中于结果，缺乏对修复过程中幻觉现象的深入理解。本文通过多层次分析，探讨了幻觉在最终补丁和中间工件中的表现，定义幻觉为未能忠实反映可用修复证据的补丁或中间工件的生成。研究表明，修复和理解幻觉普遍存在，生成补丁通过开发者编写的测试套件的比例仅为21.0%-55.9%。手动分析发现，72.7%的修复案例存在修复幻觉，错误的因果定位和修复策略分别占45.9%和18.5%。
🔬 方法详解
问题定义：本文旨在解决自动程序修复过程中幻觉现象的识别与分析问题。现有方法往往忽视了中间工件的质量，导致最终补丁的有效性不足。
核心思路：通过对修复过程中的幻觉进行多层次分析，本文定义了修复幻觉和理解幻觉，并通过具体任务进行评估，以揭示其对修复效果的影响。
技术框架：研究分为三个主要任务：触发测试用例识别、行覆盖预测和额外测试用例生成。通过这三个任务，评估不同LLM在修复过程中的表现。
关键创新：本文的创新点在于系统性地分析了幻觉现象在修复过程中的各个阶段，尤其是中间工件的影响，这在现有研究中尚属首次。
关键设计：在实验中，使用了832个Defects4J的缺陷进行评估，采用自动评估与手动分析相结合的方法，重点关注生成补丁的有效性和中间工件的准确性。
🖼️ 关键图片
📊 实验亮点
实验结果显示，生成补丁通过开发者编写的测试套件的比例仅为21.0%-55.9%。手动分析发现，72.7%的修复案例存在修复幻觉，其中错误的因果定位和修复策略分别占45.9%和18.5%。
🎯 应用场景
该研究为自动程序修复领域提供了新的视角，揭示了幻觉现象对修复效果的影响，未来可应用于改进LLM在程序修复中的表现，提高软件开发的自动化水平和效率。
📄 摘要（原文）
Large language models (LLMs) have significantly advanced automated program repair (APR), yet existing evaluations remain largely result-centric and provide limited insight into hallucination during repair. In APR, hallucination may arise not only in final patches but also in the intermediate artifacts that guide patch generation. To address this gap, we perform a multi-layered analysis of hallucination throughout the APR process. Specifically, we characterize hallucination as the production of patches or intermediate artifacts that are not faithfully grounded in the available repair evidence. We examine repair hallucination in final patches and understanding hallucination in intermediate artifacts through three tasks, namely triggering testcase identification, line coverage prediction, and additional testcase generation.We then evaluate three representative LLMs on 832 Defects4J bugs through automatic evaluation and manual analysis. Our results show that both repair and understanding hallucinations remain prevalent. Across models and settings, only 21.0%-55.9% of generated patches pass the developer-written test suite. Moreover, although more accurate intermediate artifacts are generally associated with successful repairs, this relationship does not always hold. Manual analysis of 812 sampled repairs identifies repair hallucinations in 72.7% of cases, including patches that pass all available tests; incorrect causal localization and incorrect repair strategies account for 45.9% and 18.5% of these hallucinations, respectively. Meanwhile, models frequently misidentify triggering testcases, mispredict line coverage involving branching control flow, and generate additional testcases with missing bug-triggering conditions or incorrect expected behavior.