Towards Lossless Dataset Distillation via Difficulty-Aligned Trajectory Matching
作者: Ziyao Guo, Kai Wang, George Cazenavette, Hui Li, Kaipeng Zhang, Yang You
分类: cs.CV
发布日期: 2023-10-09 (更新: 2024-03-18)
备注: First lossless dataset distillation method, accepted by ICLR 2024
🔗 代码/项目: PROJECT_PAGE
💡 一句话要点
提出基于难度对齐轨迹匹配的无损数据集蒸馏方法
🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture)
关键词: 数据集蒸馏 无损蒸馏 轨迹匹配 合成数据集 深度学习
📋 核心要点
- 现有数据集蒸馏方法在合成样本数量极少时有效，难以扩展到更大规模的合成数据集。
- 论文提出通过对齐生成模式的难度与合成数据集大小，来提高数据集蒸馏的有效性。
- 通过实验证明，新的方法在合成数据集上实现了无损蒸馏，首次达到这一目标。
📝 摘要（中文）
数据集蒸馏的最终目标是合成一个小型合成数据集，使得在该数据集上训练的模型性能与在完整真实数据集上训练的模型相当。迄今为止，尚无任何数据集蒸馏方法实现完全无损的目标，部分原因在于现有方法在合成样本总数极少时才有效。为了实现真正的无损数据集蒸馏，必须开发一种在合成数据集规模增大时仍然有效的蒸馏方法。本文提出了一种新算法，并阐明了现有方法在生成更大高质量合成集时的失败原因。我们发现选择匹配的轨迹训练阶段（早期或晚期）对蒸馏数据集的有效性有显著影响。基于这些发现，我们提出将生成模式的难度与合成数据集的大小对齐，从而首次成功实现无损数据集蒸馏。
🔬 方法详解
问题定义：本文旨在解决现有数据集蒸馏方法在合成样本数量极少时有效的问题，导致无法生成更大规模的高质量合成数据集。
核心思路：论文的核心思路是通过对齐生成模式的难度与合成数据集的大小，优化轨迹匹配过程，以提高蒸馏效果。这样的设计使得在合成数据集规模增大时，仍能保持有效的学习信号。
技术框架：整体架构包括选择合适的轨迹进行匹配，分为早期和晚期轨迹匹配两个阶段。早期轨迹用于小规模合成集，晚期轨迹则用于大规模合成集。
关键创新：最重要的技术创新点在于首次实现了无损数据集蒸馏，突破了现有方法在合成样本数量上的限制，能够有效生成更大规模的合成数据集。
关键设计：在参数设置上，选择合适的损失函数以优化轨迹匹配效果，网络结构上采用了教师网络与学生网络的配合，确保信息的有效传递。
🖼️ 关键图片
📊 实验亮点
实验结果表明，新的蒸馏方法在合成数据集上实现了无损蒸馏，较现有基线方法在性能上提升了显著的效果，具体性能数据和提升幅度在论文中详细列出，展示了该方法的有效性和优越性。
🎯 应用场景
该研究的潜在应用场景包括计算机视觉、自然语言处理等领域的数据集生成，尤其是在数据稀缺的情况下，能够有效降低数据收集成本，提高模型训练效率。未来，该方法可能推动更多领域的无损数据集蒸馏技术发展。
📄 摘要（原文）
The ultimate goal of Dataset Distillation is to synthesize a small synthetic dataset such that a model trained on this synthetic set will perform equally well as a model trained on the full, real dataset. Until now, no method of Dataset Distillation has reached this completely lossless goal, in part due to the fact that previous methods only remain effective when the total number of synthetic samples is extremely small. Since only so much information can be contained in such a small number of samples, it seems that to achieve truly loss dataset distillation, we must develop a distillation method that remains effective as the size of the synthetic dataset grows. In this work, we present such an algorithm and elucidate why existing methods fail to generate larger, high-quality synthetic sets. Current state-of-the-art methods rely on trajectory-matching, or optimizing the synthetic data to induce similar long-term training dynamics as the real data. We empirically find that the training stage of the trajectories we choose to match (i.e., early or late) greatly affects the effectiveness of the distilled dataset. Specifically, early trajectories (where the teacher network learns easy patterns) work well for a low-cardinality synthetic set since there are fewer examples wherein to distribute the necessary information. Conversely, late trajectories (where the teacher network learns hard patterns) provide better signals for larger synthetic sets since there are now enough samples to represent the necessary complex patterns. Based on our findings, we propose to align the difficulty of the generated patterns with the size of the synthetic dataset. In doing so, we successfully scale trajectory matching-based methods to larger synthetic datasets, achieving lossless dataset distillation for the very first time. Code and distilled datasets are available at https://gzyaftermath.github.io/DATM.