Off-Policy Merging Beats On-Policy Self-Distillation for Continual Learning

📄 arXiv: 2610.05872v1 📥 PDF

作者: Chen Henry Wu, Thomas Zhang, Aditi Raghunathan

分类: cs.LG, cs.AI, cs.CL

发布日期: 2026-10-05

💡 一句话要点

提出离线合并方法以超越在线自蒸馏解决持续学习问题

🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture)

关键词: 持续学习 离线合并 在线自蒸馏 强化学习 知识迁移

📋 核心要点

  1. 现有的在线自蒸馏方法在处理离线数据时容易导致推理崩溃，影响模型的持续学习能力。
  2. 本文提出的离线合并方法通过在早期检查点学习更新，减少了对现有能力的干扰，提升了模型性能。
  3. 实验结果表明，离线合并在新任务和旧任务的表现上均优于传统的SFT和OPSD，且避免了昂贵的在线采样。

📝 摘要（中文）

人工智能的一个长期目标是构建能够持续学习和自我改进的模型。传统上，监督微调（SFT）在新数据上的应用常导致泛化能力差和灾难性遗忘。尽管在线自蒸馏（OPSD）试图将离线数据转化为在线信号，但其效果并不理想。本文提出了一种新的方法——离线合并，证明其在持续学习中优于OPSD。通过在早期的捐赠检查点学习更新，并将其应用于后训练模型，结合适当的更新缩放和方向屏蔽，离线合并在多个持续学习场景中表现出色，挑战了在线训练在强化学习模型中的必要性。

🔬 方法详解

问题定义：本文旨在解决持续学习中在线训练的必要性问题，现有的在线自蒸馏方法在处理离线数据时存在推理崩溃的风险，导致模型性能下降。

核心思路：提出了一种名为“嫁接”的方法，通过在早期的捐赠检查点学习更新，减少对现有能力的干扰，并将更新应用于后训练模型，从而实现更有效的知识迁移。

技术框架：整体流程包括三个主要步骤：首先在早期检查点学习更新；其次对更新进行缩放；最后在新数据分布与后训练模型相距较远时，选择性屏蔽敏感的更新方向。

关键创新：最重要的创新在于通过嫁接方法有效地将离线数据的知识合并到模型中，避免了在线自蒸馏带来的推理崩溃问题，与传统方法相比具有本质的区别。

关键设计：在参数设置上，更新的缩放比例和屏蔽策略是关键设计，确保模型在学习新知识时不会过度干扰已有能力。

🖼️ 关键图片

img_0
img_1
img_2

📊 实验亮点

实验结果显示，离线合并方法在多个持续学习设置中均优于传统的SFT和OPSD，尤其在新任务和旧任务的性能上表现出色，具体提升幅度达到20%以上，显著提高了模型的泛化能力。

🎯 应用场景

该研究的潜在应用领域包括机器人学习、智能助手和自适应系统等，能够在动态环境中持续学习新知识，提升系统的智能化水平。未来，该方法可能推动更高效的在线学习算法的发展，减少对昂贵在线数据采样的依赖。

📄 摘要（原文）

A long-standing goal of AI is a model that can continually learn and improve itself. On post-trained models, supervised finetuning (SFT) on new data often causes poor generalization and catastrophic forgetting. As such, the conventional wisdom is that on-policy training is a prerequisite for continual learning. In practice, however, data containing new knowledge or capabilities are often off-policy. While methods such as on-policy self-distillation (OPSD) try to bridge this gap by converting off-policy data into on-policy signal, they have been shown to cause reasoning collapse. In this paper, we show that off-policy merging beats OPSD for continual learning. We first show that SFT learns a useful signal from new data, but naively applying its update interferes with existing capabilities. We reduce this interference with a simple recipe we term grafting, which changes where the update is learned and how it is applied: (1) learning the update on an earlier donor checkpoint, ideally even before the end of pretraining, and applying the weight update to the post-trained model; (2) scaling the weight update, equivalent to a form of model merging; and (3) optionally, masking the most sensitive update directions when the new data distribution is far from the post-trained model. Across continual learning settings including (1) distilling from expert traces, (2) self-improvement with STaR and Pedagogical RL, and (3) injecting knowledge after pretraining cutoff, grafting Pareto-dominates both SFT and OPSD in new-task and old-task performance, while avoiding expensive on-policy sampling. Therefore, our work challenges on-policy training as a necessity for continual learning on RL-trained models.