Interleaved Noise Injection Improves Clean, Corrupted, and OOD Performance

📄 arXiv: 2607.14466v1 📥 PDF

作者: Matt L. Wiemann, Peter Melchior, Andrew K. Saydjari

分类: cs.LG

发布日期: 2026-07-16

备注: 24 pages, 11 figures

💡 一句话要点

提出交错噪声注入以提升模型对干净、受损及OOD数据的性能

🎯 匹配领域: 支柱八：物理动画 (Physics-based Animation)

关键词: 噪声注入 模型鲁棒性 随机优化 数据增强 深度学习

📋 核心要点

  1. 现有的噪声注入方法通常采用单调衰减调度，导致模型在优化过程中容易陷入局部最小值，探索性不足。
  2. 本文提出交错噪声注入方法，通过在训练中交替使用干净和噪声数据，增强模型的探索能力，避免灾难性遗忘。
  3. 实验结果表明，交错噪声注入在CIFAR-100-C、ImageNet-C和ImageNet-R数据集上显著提升了模型的鲁棒性和对分布变化的适应性。

📝 摘要（中文）

噪声注入是一种在随机优化中广泛应用的技术。本文报告了交错（开-关-开-关）噪声注入的意外有效性，优于传统的单调衰减调度。我们提供了噪声注入的理论分析，确认冲击噪声的干扰近似于雅可比正则化，而高斯噪声则作为曲率惩罚。交错调度的特性使得优化器能够逃离局部最小值，增加探索，而不会灾难性地遗忘干净数据的重要特征。为稳定这一训练方案，我们引入了基于干净梯度幅度的梯度范数稳定化技术。实验结果显示，该方法在CIFAR-100-C、ImageNet-C和ImageNet-R上显著提高了模型的鲁棒性和对真实世界分布变化的耐受性，尤其是在与其他增强方法结合使用时效果最佳。

🔬 方法详解

问题定义：本文旨在解决现有噪声注入方法在优化过程中容易陷入局部最小值的问题，导致模型对干净和噪声数据的鲁棒性不足。

核心思路：提出交错噪声注入策略，通过在训练过程中交替使用干净和噪声数据，增强模型的探索能力，同时保持对重要特征的记忆。

技术框架：整体框架包括交替使用干净和噪声数据的训练过程，结合梯度范数稳定化技术，以应对损失快速变化的问题。

关键创新：交错噪声注入方法的核心创新在于其调度方式，允许模型在不同噪声条件下进行训练，从而提高了对各种数据分布的适应性。

关键设计：在训练过程中，采用基于干净数据梯度幅度的噪声更新缩放策略，以确保模型在切换数据类型时的稳定性。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

实验结果显示，交错噪声注入方法在CIFAR-100-C、ImageNet-C和ImageNet-R上相比于其他常见增强方法，显著提高了模型的鲁棒性，尤其是在与其他增强方法结合使用时，性能提升幅度达到XX%。

🎯 应用场景

该研究的潜在应用领域包括计算机视觉、自然语言处理等需要处理多种数据分布的任务。通过提高模型对干净、噪声和OOD数据的鲁棒性，能够在实际应用中显著提升模型的可靠性和适应性，尤其是在面对真实世界复杂数据时。

📄 摘要（原文）

Noise injection is a well-known technique in stochastic optimization. We report its surprising effectiveness with an interleaved (on-off-on-off...) rather than the usual monotonic decay schedule. We present a theoretical analysis of noise injection, which confirms that corruption by impulse noise approximates a Jacobian regularization, whereas Gaussian noise acts as a curvature penalty. This regularization behavior has been invoked to explain why noise injection increases model robustness. But the interleaved nature of our proposed schedule produces superior results even for the optimization objective: mixing phases of noisy data permits the optimizer to escape local minima and increase exploration without the risk of catastrophically forgetting the important features from the clean data. To stabilize this training scheme against the rapid changes of the loss when switching between clean and noisy data, we introduce a gradient-norm stabilization technique that scales noisy updates based on clean gradient magnitudes. We compare this method with other common augmentation methods and find substantial improvements in corruption tolerance and robustness to real-world distribution shifts on CIFAR-100-C, ImageNet-C, and ImageNet-R for ResNet and ViT architectures, with the best results being achieved by stacking our method on top of other augmentations. Through saliency and attention maps we show that the effect of interleaved noise injection stems from penalizing the failure modes encouraged by the inductive bias of the models: impulse noise works against the locality bias of convolutional (ResNet) architectures, and Gaussian noise reduces the tendency of attention-based models to pick up large-scale spurious features. Interleaved noise injection is therefore an effective tool to improve the test performance on clean, noisy, and out-of-distribution data at essentially zero computational cost.