Compresso: Structured Pruning with Collaborative Prompting Learns Compact Large Language Models

📄 arXiv: 2310.05015v2 📥 PDF

作者: Song Guo, Jiahang Xu, Li Lyna Zhang, Mao Yang

分类: cs.AI

发布日期: 2023-10-08 (更新: 2023-10-11)

💡 一句话要点

提出Compresso以解决大语言模型压缩问题

🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 大语言模型 模型压缩 结构化剪枝 低秩适应 协作提示 自然语言处理 机器学习

📋 核心要点

  1. 现有的大语言模型压缩方法主要集中在量化，剪枝方法因训练成本高和数据收集困难而未被充分探索。
  2. Compresso通过结合资源高效的剪枝算法与LLM本身，在训练过程中学习最佳剪枝决策，提升了剪枝效果。
  3. 实验结果显示，Compresso在多个基准测试中显著优于一次性剪枝基线，提升幅度可达2.21%至11.43%。

📝 摘要（中文）

尽管大语言模型（LLMs）取得了显著成功，但其庞大的体积在资源受限的硬件上部署时面临重大挑战。现有的LLM压缩方法主要集中在量化上，而剪枝方法相对较少探索。本文提出了一种新的结构化剪枝范式Compresso，通过资源高效的剪枝算法与LLM的协作，在训练过程中学习最佳剪枝决策。Compresso结合低秩适应（LoRA）与$L_0$正则化，解决了训练成本高和数据收集难的问题。实验表明，Compresso在多个基准测试中显著超越了一次性剪枝基线，保持了原有性能并在阅读理解上超越LLaMA-7B。

🔬 方法详解

问题定义：本文旨在解决大语言模型在资源受限硬件上的部署挑战，现有剪枝方法因训练成本高和数据收集困难而效果有限。

核心思路：Compresso通过在训练过程中结合剪枝算法与LLM的协作，学习最佳剪枝决策，从而实现高效的结构化剪枝。

技术框架：整体流程包括两个主要模块：资源高效的剪枝算法和LLM的协作提示。首先，利用低秩适应（LoRA）与$L_0$正则化进行指令调优，然后通过协作提示增强剪枝算法的效果。

关键创新：Compresso的创新在于通过协作提示实现了剪枝算法与LLM的协同工作，显著提升了剪枝效果，与传统的一次性剪枝方法相比具有本质区别。

关键设计：在参数设置上，采用了低秩适应（LoRA）与$L_0$正则化的结合，损失函数设计上注重剪枝决策的优化，确保在保持性能的同时实现有效的模型压缩。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

Compresso在多个基准测试中表现优异，相较于一次性剪枝基线，在常识推理、阅读理解、MMLU和BBH基准上分别提升了2.21%、11.43%、7.04%和4.81%的得分，显示出其在保持性能的同时实现有效压缩的能力。

🎯 应用场景

该研究的潜在应用领域包括自然语言处理、智能助手和对话系统等，能够在资源受限的环境中高效部署大语言模型，提升其实际应用价值。未来，Compresso可能推动更多高效模型压缩技术的发展，促进AI技术的普及与应用。

📄 摘要（原文）

Despite the remarkable success of Large Language Models (LLMs), the massive size poses significant deployment challenges, particularly on resource-constrained hardware. While existing LLM compression methods focus on quantization, pruning remains relatively unexplored due to the high cost of training-based approaches and data collection challenges. One-shot pruning methods, although cost-effective and data-free, have become dominant in LLM pruning, but lead to performance decline under the structured pruning setting. In this work, we introduce a new paradigm for structurally pruning LLMs, called Compresso. Our approach, through the collaboration of the proposed resource-efficient pruning algorithm and the LLM itself, learns optimal pruning decisions during the training process. Compresso addresses the challenges of expensive training costs and data collection by incorporating Low-Rank Adaptation (LoRA) into the $L_0$ regularization during the instruction tuning process. Then, we further augment the pruning algorithm by introducing a collaborative prompt that fosters collaboration between the LLM and the pruning algorithm, significantly boosting the overall performance. To this end, Compresso prunes LLaMA-7B to 5.4B, maintaining original performance and even surpassing LLaMA-7B in reading comprehension by 2.62%. Extensive experiments demonstrate that Compresso significantly outperforms one-shot pruning baselines across various sparsity ratios, achieving up to 2.21%, 11.43%, 7.04%, and 4.81% higher scores on the commonsense reasoning, reading comprehension, MMLU, and BBH benchmarks, respectively.