A Unified Evaluation Framework of Physical Adversarial Robustness in the Edge VLM Era

📄 arXiv: 2610.29196v1 📥 PDF

作者: Eric Liang, Alan Luo, Kaiwen Yuan

分类: cs.CV, cs.CR, cs.LG

发布日期: 2026-10-21

备注: Accepted at the NeurIPS 2026 Workshop on Grounded and Faithful Vision-Language Models for Real-World Deployment (VLM4RWD). 16 pages, 6 figures, 5 tables. Code: https://github.com/ericliang1/physical-adversarial-robustness

🔗 代码/项目: GITHUB

💡 一句话要点

提出统一评估框架以研究边缘VLM的物理对抗鲁棒性

🎯 匹配领域: 支柱三：空间感知与语义 (Perception & Semantics) 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 物理对抗攻击 视觉-语言模型 鲁棒性评估 条件攻击成功率 目标检测 边缘计算 安全系统

📋 核心要点

  1. 现有的目标检测方法在面对物理对抗攻击时缺乏系统性评估，尤其是边缘VLM的鲁棒性尚未被深入研究。
  2. 本文提出了一种统一的评估框架，通过条件攻击成功率（cASR）来比较VLM与传统目标检测器的物理对抗脆弱性。
  3. 实验结果表明，VLM在物理对抗攻击中的脆弱性与目标检测器相当，并且其失败模式与模型输出格式密切相关。

📝 摘要（中文）

近年来，视觉-语言模型（VLM）在语言提示和生成解码方面取得了显著进展，提升了零-shot和开放词汇检测能力。然而，边缘VLM的物理对抗鲁棒性尚未得到系统研究，且与现代目标检测器的输出存在根本差异，给公平评估带来了方法论挑战。本文提出了一种统一的指标——条件攻击成功率（cASR），并通过对比不同模型在特定条件下的表现，揭示了VLM在物理对抗攻击中的脆弱性及其潜在的危险失败模式。

🔬 方法详解

问题定义：本文旨在解决边缘VLM在物理对抗攻击中的鲁棒性评估问题。现有方法未能系统性地比较VLM与传统目标检测器在此类攻击下的表现，导致评估结果缺乏公正性。

核心思路：论文提出了一种新的评估指标——条件攻击成功率（cASR），该指标专注于在特定条件下（如匹配的灰色遮挡物下）模型的检测能力，从而提供更公平的比较。

技术框架：整体架构包括模型选择、条件攻击设计和结果评估三个主要模块。首先选择代表性模型，然后设计对抗补丁进行攻击，最后通过cASR评估模型的鲁棒性。

关键创新：最重要的创新在于引入了cASR这一评估指标，并将失败模式分为遗漏、替代和位移，揭示了VLM在物理对抗攻击中的潜在危险性。

关键设计：在实验中，模型的输出格式被认为是影响失败模式的关键因素，尤其是位移和替代模式可能导致更危险的后果。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

实验结果显示，VLM在物理对抗攻击中的脆弱性与传统目标检测器相当，且在分布转移下，VLM的失败模式从遗漏转向位移和替代，提示了更危险的失败风险。这一发现为未来的模型设计和评估提供了重要参考。

🎯 应用场景

该研究的潜在应用领域包括安全关键的边缘计算系统，如自动驾驶、监控和人机交互等场景。通过提高VLM在物理对抗攻击下的鲁棒性，可以增强这些系统的安全性和可靠性，降低潜在风险。

📄 摘要（原文）

Recent advancements in Vision-Language Models (VLMs) enable end-to-end visual grounding via language prompts and generative decoding, demonstrating largely improved zero-shot and open-vocabulary detection capabilities. As a result, VLMs are rapidly being deployed at the edge in safety-critical real-world systems. However, the physical adversarial robustness of edge VLMs has not been systematically studied, and the fundamental differences in output with modern object detectors create methodological challenges for fair evaluation. To compare the two families directly, we evaluate every selected representative model on one outcome: whether an adversarial patch worn on a person evades detection of the model. We report this by introducing a unified metric, conditional attack success rate (cASR), counted only on images the model still localizes under a matched gray occluder, and partition every failure into omission (no box), substitution (a box on the patch), and displacement (a box on neither). We benchmark the physical adversarial vulnerability of compact grounding VLMs against a range of object detectors, and find that the VLMs can be as vulnerable as the detectors. We disclose that failure mode is largely associated with the output format of a model rather than its architecture family. Of the three, only omission is visible to a downstream system; displacement and substitution return confident, well-formed boxes, making them potentially more dangerous failures. Further, we find that, under distribution shift, VLMs that primarily failed by omission instead shift to failing more by displacement/substitution. Thus, a grounding VLM can inherit the detector's vulnerability while showing the possibility to adopt more dangerous failure modes. Our code and data are available at https://github.com/ericliang1/physical-adversarial-robustness.