D2 Pruning: Message Passing for Balancing Diversity and Difficulty in Data Pruning

📄 arXiv: 2310.07931v1 📥 PDF

作者: Adyasha Maharana, Prateek Yadav, Mohit Bansal

分类: cs.LG, cs.AI, cs.CL, cs.CV

发布日期: 2023-10-11

备注: 17 pages (Our code is available at https://github.com/adymaharana/d2pruning)

💡 一句话要点

提出D2 Pruning以平衡数据剪枝中的多样性与难度问题

🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 数据剪枝 多样性与难度 图结构 消息传递 核心集选择 深度学习 模型泛化

📋 核心要点

  1. 现有数据剪枝方法往往偏向于选择简单样本，导致模型训练效果不佳。
  2. D2 Pruning通过图结构表示数据集，利用消息传递机制更新样本难度评分，平衡多样性与难度。
  3. 实验表明，D2 Pruning在多个数据集上提升了剪枝效果，最高可达70%的剪枝率，且增强了模型的泛化能力。

📝 摘要（中文）

分析理论表明，高质量数据可以降低模型的测试误差。数据剪枝旨在选择训练数据的子集，以最大化在该子集上训练模型的性能。现有方法主要有基于几何的数据选择和基于难度评分的选择，但这两者各有不足。本文提出了一种新的剪枝算法D2 Pruning，通过在数据集图上进行前向和反向消息传递，更新样本的难度评分，从而实现同时考虑数据多样性和难度的核心选择。实验结果表明，D2 Pruning在多种视觉和语言数据集上显著提升了剪枝效果，最高可达70%的剪枝率，同时提高了预训练模型的泛化能力。

🔬 方法详解

问题定义：本文解决的是在数据剪枝过程中如何平衡数据的多样性与样本的难度。现有方法往往只关注其中一方面，导致最终选择的子集无法有效支持模型训练。

核心思路：D2 Pruning的核心思路是将数据集表示为无向图，通过前向和反向消息传递机制，更新每个样本的难度评分，确保选择的子集既包含多样性又具备适当的难度。

技术框架：整体流程包括将数据集构建为图结构，进行消息传递以更新样本难度评分，最后基于更新后的评分进行图基采样以选择核心集。主要模块包括图构建、消息传递和核心集选择。

关键创新：D2 Pruning的创新在于结合了数据多样性与样本难度的双重考虑，通过图结构和消息传递机制实现了更为精确的核心集选择。这与传统方法的单一优化目标形成了鲜明对比。

关键设计：在设计中，难度评分的更新依赖于邻近样本的难度信息，确保了评分的动态性。此外，采用图基采样方法选择核心集，能够有效捕捉数据空间中的多样性和难度特征。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

实验结果显示，D2 Pruning在多个视觉和语言数据集上显著提升了核心集选择的效果，剪枝率最高可达70%。与现有最先进的方法相比，D2 Pruning不仅提高了数据集的多样性，还增强了预训练模型的泛化能力，显示出其优越性。

🎯 应用场景

该研究的潜在应用领域包括计算机视觉、自然语言处理等多模态任务，尤其适用于需要处理大规模数据集的场景。通过优化数据选择，D2 Pruning能够提升模型的训练效率和泛化能力，具有重要的实际价值和未来影响。

📄 摘要（原文）

Analytical theories suggest that higher-quality data can lead to lower test errors in models trained on a fixed data budget. Moreover, a model can be trained on a lower compute budget without compromising performance if a dataset can be stripped of its redundancies. Coreset selection (or data pruning) seeks to select a subset of the training data so as to maximize the performance of models trained on this subset, also referred to as coreset. There are two dominant approaches: (1) geometry-based data selection for maximizing data diversity in the coreset, and (2) functions that assign difficulty scores to samples based on training dynamics. Optimizing for data diversity leads to a coreset that is biased towards easier samples, whereas, selection by difficulty ranking omits easy samples that are necessary for the training of deep learning models. This demonstrates that data diversity and importance scores are two complementary factors that need to be jointly considered during coreset selection. We represent a dataset as an undirected graph and propose a novel pruning algorithm, D2 Pruning, that uses forward and reverse message passing over this dataset graph for coreset selection. D2 Pruning updates the difficulty scores of each example by incorporating the difficulty of its neighboring examples in the dataset graph. Then, these updated difficulty scores direct a graph-based sampling method to select a coreset that encapsulates both diverse and difficult regions of the dataset space. We evaluate supervised and self-supervised versions of our method on various vision and language datasets. Results show that D2 Pruning improves coreset selection over previous state-of-the-art methods for up to 70% pruning rates. Additionally, we find that using D2 Pruning for filtering large multimodal datasets leads to increased diversity in the dataset and improved generalization of pretrained models.