SparseDecoding: Decoding-Aware Pruning for Accurate and Efficient LLM Inference

📄 arXiv: 2610.12327v1 📥 PDF

作者: Qitong Wang, Xinwei Niu, Mingluo Su, Shanwei Zhao, Shiai Zhu, Huan Wang

分类: cs.LG, cs.CL

发布日期: 2026-10-08

💡 一句话要点

提出SparseDecoding以解决大语言模型推理中的延迟问题

🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 大语言模型 网络剪枝 解码优化 Hessian 稀疏矩阵 自回归生成 性能提升

📋 核心要点

  1. 现有的基于Hessian的网络剪枝方法在解码阶段存在分布偏移问题，影响模型性能。
  2. SparseDecoding通过在自回归生成过程中收集激活数据，构建校准矩阵，解决了剪枝与解码激活不一致的问题。
  3. 在多个代表性LLM上进行的实验表明，该方法在长文本生成基准上表现优异，解码速度提升可达1.48倍。

📝 摘要（中文）

大语言模型（LLM）推理的解码阶段受限于内存，导致显著延迟。现有的基于Hessian的逐层无训练网络剪枝方法虽然能减少解码时从内存读取的非零参数数量，但其计算的Hessian基于预先收集的自然序列，而解码时使用的是自生成的令牌，导致分布偏移，从而影响剪枝效果。为了解决这一问题，本文提出了SparseDecoding，一个针对解码的剪枝框架，旨在提高LLM解码的准确性和效率。通过在密集模型自回归生成过程中收集层级激活构建校准矩阵，SparseDecoding使剪枝目标与解码激活对齐，并开发了优化的N:M稀疏矩阵-向量内核，显著提升了解码速度。

🔬 方法详解

问题定义：本文旨在解决大语言模型推理中解码阶段的延迟问题，现有方法在剪枝时计算的Hessian基于自然序列，导致与生成序列之间存在分布偏移，影响剪枝效果。

核心思路：SparseDecoding的核心思路是通过在密集模型自回归生成过程中收集层级激活，构建校准矩阵，从而使剪枝目标与解码激活对齐，提升剪枝后的模型性能。

技术框架：该方法的整体架构包括两个主要模块：一是算法模块，通过收集激活数据构建校准矩阵；二是系统模块，开发优化的N:M稀疏矩阵-向量内核，采用位掩码索引和固定步长遍历。

关键创新：SparseDecoding的创新在于其解码感知的剪枝框架，能够有效对齐剪枝目标与解码激活，解决了传统方法中存在的分布偏移问题。

关键设计：在参数设置上，SparseDecoding采用了层级激活数据构建校准矩阵，并在系统层面优化了稀疏矩阵-向量乘法的实现，确保在解码过程中高效利用内存。

🖼️ 关键图片

img_0
img_1
img_2

📊 实验亮点

实验结果显示，SparseDecoding在长文本生成基准上优于标准固定文本校准，且在A100 GPU上实现了高达1.48倍的端到端解码速度提升，展现了其在实际应用中的显著优势。

🎯 应用场景

该研究的潜在应用领域包括自然语言处理、对话系统和文本生成等场景，能够显著提高大语言模型在实际应用中的解码效率和响应速度，具有重要的实际价值和未来影响。

📄 摘要（原文）

The memory-bound nature of the decoding stage of large language model (LLM) inference incurs significant latency. Layer-wise training-free network pruning approaches guided by the Hessian have been a prominent solution to this problem, as pruning reduces the number of nonzero parameters read from memory during decoding. Nevertheless, typical methods in this line compute the Hessian using pre-collected natural sequences, whereas the model is fed self-generated tokens during decoding, creating a distribution shift between the two sequences. The Hessian calculated on the natural sequence is different from that calculated on the generated sequence. We observe that this discrepancy causes the activation distribution during generation to deviate from that used for pruning, further hurting the pruned model performance. Moreover, most existing LLM pruning methods that bring actual speedup primarily target the sparse matrix-matrix (SpMM) multiplication, providing limited support for the sparse matrix-vector (SpMV) operations, which dominate decoding. To solve these problems, we introduce SparseDecoding, a principled decoding-aware pruning framework tailored for accurate and efficient LLM decoding. Specifically, at the algorithmic axis, SparseDecoding constructs calibration matrices from layer-wise activations collected during the dense-model autoregressive generation, excluding prefill, thereby aligning the pruning objective with the decoding activations. At the system axis, we develop an optimized N:M sparse matrix-vector kernel with bitmask indexing and fixed-step traversal. Substantial empirical results on representative LLMs (Llama-3.1-8B, Llama-3.3-70B, Qwen3-14B / 32B) demonstrate that our method consistently outperforms standard fixed-text calibration on the long-form generation benchmarks while achieving up to 1.48x end-to-end wall-clock decoding speedup on A100 GPUs.