Thought Anchors: Which LLM Reasoning Steps Matter?

📄 arXiv: 2506.19143v4 📥 PDF

作者: Paul C. Bogdan, Uzay Macar, Neel Nanda, Arthur Conmy

分类: cs.LG, cs.AI, cs.CL

发布日期: 2025-06-23 (更新: 2025-10-27)

备注: Paul C. Bogdan and Uzay Macar contributed equally to this work, and their listed order was determined by coinflip. Neel Nanda and Arthur Conmy contributed equally to this work as senior authors, and their listed order was determined by coinflip

💡 一句话要点

提出思维锚点方法以解析大型语言模型的推理过程

🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 大型语言模型 推理过程 可解释性 思维锚点 因果关系分析 数学问题求解 黑箱方法

📋 核心要点

  1. 现有的可解释性方法主要关注模型的单次前向传递，无法有效分析推理过程中的多标记计算步骤。
  2. 论文提出了一种黑箱方法，通过分析句子的反事实重要性，识别出对推理轨迹影响显著的句子，即思维锚点。
  3. 通过案例研究，论文展示了该方法在解决复杂数学问题时的一致性和有效性，提供了深入理解推理模型的工具。

📝 摘要（中文）

当前前沿的大型语言模型依赖推理以实现最先进的性能。然而，现有的可解释性方法在这一领域存在局限，因为标准方法主要研究模型的单次前向传递，而非推理过程中展开的多标记计算步骤。本文提出了一种黑箱方法，通过反复从模型中抽样替换句子，过滤出语义上不同的句子，并从该点继续推理，以量化句子对最终答案分布的影响。我们发现某些句子对推理轨迹和最终答案有显著影响，称之为“思维锚点”。这些句子通常涉及规划或不确定性管理，后续句子的专门注意力头会持续关注思维锚点。我们进一步展示了在推理轨迹中检查句子间因果关系可以深入理解模型行为，并为预测问题难度提供了有价值的信息。我们提供了一个开源工具（thought-anchors.com）用于可视化我们方法的输出。

🔬 方法详解

问题定义：本论文旨在解决现有可解释性方法无法有效分析大型语言模型推理过程中的多标记计算步骤的问题。现有方法通常只关注单次前向传递，导致对推理过程的理解不足。

核心思路：论文的核心思路是通过分析句子的反事实重要性，识别出对推理轨迹和最终答案有显著影响的句子。这种方法能够揭示推理过程中的关键步骤，帮助理解模型的决策机制。

技术框架：整体架构包括句子替换、语义过滤和推理链延续三个主要模块。首先，从模型中抽样替换句子；其次，过滤出语义上不同的句子；最后，从替换句子开始继续推理，以量化其对最终答案的影响。

关键创新：最重要的技术创新点在于提出了“思维锚点”的概念，这些句子在推理过程中起到关键作用，且后续句子会专门关注这些锚点。这一发现与现有方法的单一分析视角形成鲜明对比。

关键设计：在技术细节上，论文设计了特定的句子替换策略和语义过滤机制，以确保替换句子在语义上具有显著差异。此外，模型的注意力机制被用于追踪句子间的因果关系，进一步增强了对推理过程的理解。

📊 实验亮点

实验结果表明，所提出的方法能够有效识别出对推理轨迹影响显著的思维锚点，且在解决复杂数学问题时，模型的推理结构表现出一致性。具体而言，模型在处理不同问题时，思维锚点的识别率显著提高，展示了方法的有效性和实用性。

🎯 应用场景

该研究的潜在应用领域包括教育、自动化推理系统和复杂问题求解等。通过深入理解大型语言模型的推理过程，研究者和开发者可以优化模型性能，提高其在特定任务中的表现，进而推动人工智能在实际应用中的发展。

📄 摘要（原文）

Current frontier large-language models rely on reasoning to achieve state-of-the-art performance. Many existing interpretability are limited in this area, as standard methods have been designed to study single forward passes of a model rather than the multi-token computational steps that unfold during reasoning. We argue that analyzing reasoning traces at the sentence level is a promising approach to understanding reasoning processes. We introduce a black-box method that measures each sentence's counterfactual importance by repeatedly sampling replacement sentences from the model, filtering for semantically different ones, and continuing the chain of thought from that point onwards to quantify the sentence's impact on the distribution of final answers. We discover that certain sentences can have an outsized impact on the trajectory of the reasoning trace and final answer. We term these sentences \textit{thought anchors}. These are generally planning or uncertainty management sentences, and specialized attention heads consistently attend from subsequent sentences to thought anchors. We further show that examining sentence-sentence causal links within a reasoning trace gives insight into a model's behavior. Such information can be used to predict a problem's difficulty and the extent different question domains involve sequential or diffuse reasoning. As a proof-of-concept, we demonstrate that our techniques together provide a practical toolkit for analyzing reasoning models by conducting a detailed case study of how the model solves a difficult math problem, finding that our techniques yield a consistent picture of the reasoning trace's structure. We provide an open-source tool (thought-anchors.com) for visualizing the outputs of our methods on further problems. The convergence across our methods shows the potential of sentence-level analysis for a deeper understanding of reasoning models.