JevAdvBench: A Benchmark and Black-Box Attacks for Reinforcement Learning for Calibrated Decisions Models
作者: Jianyi Hu, Hangtao Zhang, Yi Liu, Yeqi Zeng, Li Zeng, Xianlong Wang, Rui Wang, Leo Yu Zhang
分类: cs.CR, cs.AI, cs.CL
发布日期: 2026-09-25
备注: 33 pages, 13 figures, 19 tables. Project website: https://JevAdvBench.github.io/JevAdvBench/
🔗 代码/项目: PROJECT_PAGE
💡 一句话要点
提出JevAdvBench以评估强化学习决策模型的鲁棒性
🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture)
关键词: 强化学习 决策模型 对抗性基准 鲁棒性评估 输入操控
📋 核心要点
- 现有强化学习决策模型的鲁棒性尚未得到有效评估，传统对抗基准无法适用于生成类型答案的模型。
- 论文提出通过比较模型干净决策与操控决策的差异，构建JevAdvBench以评估RLCD模型的鲁棒性。
- 实验结果表明，某些输入操控能显著影响模型决策，最高可导致12.1%的决策翻转，显示出模型在面对操控时的脆弱性。
📝 摘要（中文）
本文提出了JevAdvBench，这是首个针对强化学习决策模型（RLCD）的对抗性基准，旨在评估其在面对输入操控时的鲁棒性。RLCD模型如Jev能够对输入状态做出概率、选择或评分的响应，但其鲁棒性尚未得到有效测量。现有的对抗基准主要评估模型生成的输出，而JevAdvBench通过比较模型的干净决策与被操控决策的差异，提供了一种新的评估方法。该基准包含812个类型问题和9744个单编辑变体，实验结果显示，某些输入操控可以显著影响模型决策，提示应用RLCD模型时需谨慎对待输入状态。
🔬 方法详解
问题定义：本文旨在解决强化学习决策模型在面对输入操控时的鲁棒性评估问题。现有方法主要依赖于模型生成的输出进行评估，无法有效反映模型的真实表现。
核心思路：论文提出的核心思路是通过比较模型的干净决策与被操控决策的差异来进行评估，而不是依赖于外部标签。这种方法能够更准确地反映模型在不同输入下的表现。
技术框架：JevAdvBench的整体架构包括812个类型问题和9744个单编辑变体。每个变体通过编辑请求的某一部分来测试模型的鲁棒性，确保编辑能够被模型接收。
关键创新：该研究的主要创新在于首次提出了针对RLCD模型的对抗性基准，能够有效评估模型在面对输入操控时的脆弱性，与传统方法相比具有本质区别。
关键设计：在实验中，模型的干净决策与操控决策的比较是通过统计分析实现的，确保了评估的准确性。同时，使用的输入令牌计费机制确保了每次编辑都能被模型正确处理。
🖼️ 关键图片
📊 实验亮点
实验结果显示，通过对输入进行操控，模型的决策翻转率可达12.1%，而最强的注入命令翻转率为10.1%。此外，38%的自信回答被推至0.8以下的置信阈值，需人工审核，显示出模型在处理不可信输入时的脆弱性。
🎯 应用场景
JevAdvBench的研究成果可广泛应用于基于强化学习的决策系统，尤其是在自动化决策和智能助手等领域。通过评估模型的鲁棒性，开发者可以更好地理解和改进模型，提升其在实际应用中的可靠性和安全性。未来，该基准可能推动更多针对决策模型的安全性研究。
📄 摘要（原文）
Models trained with reinforcement learning for calibrated decisions (RLCD), such as Jev, answer a typed question about an input, the state, with a probability, a choice, or a score, and software acts on the answer without a person reading it. Their robustness has not been measured: adversarial benchmarks score what a model generates or executes, whereas a typed model generates nothing and returns a well-formed answer even when manipulated. Measurement is also hard, because identical requests can return different answers, most available labels come from the model itself, and the API preprocesses each request out of view. Our key idea is to score each attacked decision against the model's own clean decision rather than against labels, and to read it against the change caused by an identical re-run. Building on this, we introduce JevAdvBench, to our knowledge the first adversarial benchmark for RLCD models, with 812 typed questions over 66 scenarios, and a black-box attack suite of 9,744 single-edit variants that each edit one part of a request, with billed input tokens confirming that the edit reached the model. On jev-1.13.0, rewording stays within 1.2 percentage points of the re-run baseline, and fields outside the schema never reach the model. In contrast, one unverified opinion appended to the state flips 12.1% of decisions, statistically tied with the strongest injected command (10.1%), and pushes 38% of confident answers below the 0.8 confidence threshold that routes them to human review. Applications built on RLCD models should therefore treat the state as untrusted, argued input. Project website: https://JevAdvBench.github.io/JevAdvBench/