CARD: Closing the Teacher-Student Loop for Adversarial Robustness Distillation

📄 arXiv: 2610.19911v1 📥 PDF

作者: Zhi Li, Haowei Liu, Hongchen Yang, Xiaoxuan Wang, Song Gao, Zhenli He, Wei Zhou, Shaowen Yao

分类: cs.LG, cs.CV

发布日期: 2026-10-14

💡 一句话要点

提出CARD框架以解决对抗鲁棒性蒸馏中的反馈问题

🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture)

关键词: 对抗鲁棒性 蒸馏训练 闭环反馈 深度学习 模型优化 教师-学生框架 对抗攻击 机器学习

📋 核心要点

  1. 现有对抗鲁棒性蒸馏方法忽视了教师与学生之间的反馈互动，导致潜在信息的丢失。
  2. CARD框架通过将蒸馏学生作为反馈源，形成闭环，优化鲁棒教师并将改进反馈回学生。
  3. 在CIFAR-10、CIFAR-100和TinyImageNet-10等数据集上，CARD在大多数设置中提高了对抗鲁棒性，同时保持了清晰准确性。

📝 摘要（中文）

对抗鲁棒性蒸馏通常将鲁棒教师视为固定知识源，以单向方式将鲁棒性转移给紧凑的学生。然而，这种方法在蒸馏后终止了教师与学生的互动，忽视了蒸馏学生可能提供的有用反馈。为了解决这一问题，本文提出了循环对抗鲁棒性蒸馏（CARD）框架，将蒸馏学生转变为反馈源，以改进鲁棒教师，并将教师的改进反馈回同一学生。CARD首先进行强到弱的对抗蒸馏，然后利用增强可靠性的学生反馈来优化鲁棒教师。实验结果表明，CARD在多个评估设置中提高了对抗鲁棒性，同时通常保持了清晰准确性。

🔬 方法详解

问题定义：本文旨在解决对抗鲁棒性蒸馏中教师与学生之间缺乏有效反馈的问题。现有方法在蒸馏后终止了互动，导致学生的潜在反馈未被利用，可能影响鲁棒性提升。

核心思路：CARD框架通过将蒸馏学生转化为反馈源，形成一个闭环，允许教师根据学生的反馈进行优化，从而实现双向知识传递。

技术框架：CARD的整体架构包括两个主要阶段：首先进行强到弱的对抗蒸馏，然后利用增强的学生反馈来优化鲁棒教师，最后将改进后的教师用于继续训练同一学生。

关键创新：CARD的创新在于引入了教师与学生之间的反馈闭环机制，突破了传统单向蒸馏的局限，使得鲁棒性转移能够持续进行。

关键设计：在实现过程中，CARD结合了逆对抗学生反馈与教师-学生差异最大化的策略，并建立了可靠的弱到强反馈转移的样本级充分条件，确保了反馈的有效性。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

实验结果显示，CARD在CIFAR-10、CIFAR-100和TinyImageNet-10数据集上，相较于基线方法，显著提高了对抗鲁棒性，且在大多数情况下保持了清晰准确性，验证了蒸馏学生对教师的反馈价值。

🎯 应用场景

该研究的潜在应用领域包括安全性要求高的深度学习系统，如自动驾驶、金融欺诈检测和网络安全等。通过提高模型的对抗鲁棒性，CARD能够增强这些系统在面对恶意攻击时的稳定性和可靠性，具有重要的实际价值和未来影响。

📄 摘要（原文）

Adversarial robustness distillation typically treats a robust teacher as a fixed knowledge source and transfers robustness to a compact student in a one-way manner. This paradigm terminates teacher-student interaction once the student is distilled, thereby discarding potentially useful feedback from the distilled student. However, exploiting such feedback is nontrivial because indiscriminate weak-to-strong transfer may propagate student errors to the teacher. To address this issue, we propose Cyclic Adversarial Robustness Distillation (CARD), a closed-loop framework that turns the distilled student into a feedback source for refining the robust teacher and subsequently transfers the teacher gains back to the same student. CARD first performs strong-to-weak adversarial distillation and then uses reliability-enhanced student feedback to refine the robust teacher. The refinement combines inverse adversarial student feedback with teacher-student discrepancy maximization, after which the refined teacher is used to continue training the same student under the unchanged base distillation objective. We further establish a sample-level sufficient condition for reliable weak-to-strong feedback transfer and derive a distribution-level upper bound on the refined teacher's risk. Experiments on CIFAR-10, CIFAR-100, and TinyImageNet-10 across multiple teacher-student architectures and base distillation methods show that CARD improves adversarial robustness in most evaluated settings while generally preserving clean accuracy. These results show that the distilled student can provide useful feedback to the teacher, enabling robustness transfer to proceed beyond the conventional one-way teacher-to-student paradigm.