ADS-C: Antidistillation Sampling for Classification

📄 arXiv: 2607.15467 📥 PDF

作者: Khawaja Abaid Ullah, Mohammad Javad Khojasteh

分类: cs.LG, cs.CR

发布日期: 2026-07-20

💡 一句话要点

提出ADS-C以解决分类任务中的蒸馏攻击问题

🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture) 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 抗蒸馏 知识蒸馏 分类任务 模型防御 深度学习

📋 核心要点

  1. 现有的知识蒸馏方法容易受到攻击者的蒸馏攻击，导致专有分类器的知识被复制。
  2. 提出的ADS-C方法通过输入依赖的扰动来防御蒸馏攻击，确保教师模型的准确率不变，同时降低学生模型的性能。
  3. 在CIFAR-100、CIFAR-10和Tiny-ImageNet数据集上，ADS-C分别使蒸馏学生模型性能下降17.4、29.6和13.3个百分点，显示出其有效性。

📝 摘要（中文）

知识蒸馏使得对手能够通过查询分类器的预测接口来复制专有分类器，并在返回的概率向量上训练替代模型。为应对这一威胁，提出了抗蒸馏采样（ADS-C），该方法通过输入依赖的梯度引导扰动来调整输出分布。本文将该防御方法适配于分类任务，发现其行为受教师模型每个输入的置信度边际分布控制。实验结果表明，ADS-C在保持教师模型准确率不变的情况下，显著降低了蒸馏学生模型的性能，且其效用成本为零。

🔬 方法详解

问题定义：论文要解决的问题是知识蒸馏过程中，攻击者通过查询教师模型的输出概率来复制其知识，现有防御方法未能有效应对这一挑战。

核心思路：ADS-C方法通过对输出分布进行输入依赖的梯度引导扰动，来降低蒸馏攻击的效果，同时保持教师模型的准确性。

技术框架：该方法的整体架构包括输入扰动生成、教师模型输出调整和学生模型训练三个主要模块。输入扰动根据教师模型的置信度边际进行设计，以确保输出的有效性。

关键创新：ADS-C是首个在分类任务中实现的抗蒸馏防御方法，其创新之处在于通过闭式形式的边际预算来构建扰动，确保教师模型的准确率与未防御时相同。

关键设计：该方法的关键设计包括温度软化机制，用于调整输出分布的过渡行为，以及确保每个输入的顶级预测不变的参数设置。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

实验结果显示，ADS-C在CIFAR-100、CIFAR-10和Tiny-ImageNet数据集上，蒸馏学生模型的性能分别下降17.4、29.6和13.3个百分点，而未修改的防御方法则导致教师模型准确率下降27.5、32.9和22.2个百分点，显示出ADS-C的有效性和优势。

🎯 应用场景

ADS-C方法在保护专有分类器的知识不被复制方面具有重要应用价值，尤其适用于金融、医疗等对数据隐私要求高的领域。未来，该方法有望推广到其他类型的模型防御中，提升整体安全性。

📄 摘要（原文）

Knowledge distillation enables an adversary to replicate a proprietary classifier by querying its prediction interface and training a surrogate on the returned probability vectors. Antidistillation sampling, proposed for large language models, counters this threat with an input-dependent, gradient-directed perturbation of the served distribution; its transfer to classification has not been studied. Adapting the defense to classification, we show its behavior is governed by the distribution of the teacher's per-input confidence margins. Because well-trained classifiers are severely overconfident, the direct transfer exhibits an inert window: below a closed-form-predictable threshold, it affects neither attacker nor defender; beyond it, the defense undergoes a phase transition and degrades the teacher faster than the attacker's student. Temperature softening rescales the transition in closed form, and every temperature configuration lies on the same unfavorable trade-off curve. Our method, ADS-C, composes the perturbation under a closed-form, per-input margin budget that provably preserves every served top-1 prediction, so the defended teacher's accuracy equals the undefended teacher's identically. Under this guarantee the distilled student still loses 17.4 percentage points on CIFAR-100, 29.6 on CIFAR-10, and 13.3 on Tiny-ImageNet; matching this degradation with the unmodified defense costs 27.5, 32.9, and 22.2 points of teacher accuracy. Because served labels are unchanged, a hard-label attacker gains nothing, while the defended soft output trains a student up to 29.7 points below that floor: the incentive to distill served probabilities is not merely removed but reversed. To our knowledge, ADS-C is the first antidistillation defense for classification whose utility cost is exactly zero.