TenetBench: A Clause-Anchored Russian Corpus for Evaluating Language Models against Professional Codes of Ethics
作者: Yegor Anichkov, Valery Vinogradov, Andrey Furov, Marina Polukoshko, Sergey Bolovtsov
分类: cs.CL
发布日期: 2026-10-21
备注: Preprint under review. 39 pages, 2 figures, 5 tables; Online Resource 1 appended (9 pages). Dataset and code: https://doi.org/10.5281/zenodo.22678484
💡 一句话要点
提出TenetBench以评估语言模型对职业伦理规范的遵守情况
🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)
关键词: 职业伦理 语言模型 合规性评估 自动化生成 数据集构建 人类验证 模型评估
📋 核心要点
- 现有方法无法准确评估语言模型在实际应用中可能违反的职业伦理规范，导致潜在风险难以识别。
- 论文提出了TenetBench数据集，通过自动化生成与职业伦理条款相冲突的请求，评估语言模型的合规性。
- 实验结果显示，十四种模型中大多数在处理冲突请求时未能发出警告，且模型的拒绝率在启用推理时显著降低。
📝 摘要（中文）
职业伦理规范指导和限制专业人士的行为，尤其是在与大型语言模型助手互动时。TenetBench构建了一个包含3433个请求的语料库，这些请求与十五个俄语职业伦理规范的条款相冲突，并与合法请求形成对比。通过自动化流程生成任务，语料库可随着新规范的加入而扩展。人类验证结果显示，冲突任务的合理性评分高达0.995，模型评判者的协议一致性达到0.894。研究评估了十四种模型和推理模式的表现，发现大多数模型在处理冲突请求时未能发出警告。该数据集和流程旨在帮助评估语言模型对职业规范的遵守情况，并为特定职业和组织构建诊断数据集。
🔬 方法详解
问题定义：本研究旨在解决语言模型在与用户交互时可能违反职业伦理规范的问题。现有评估方法无法具体识别哪些条款被违反，以及这种情况发生的频率。
核心思路：TenetBench通过构建一个包含冲突请求和合法请求的语料库，提供了一种系统化的方法来评估语言模型的合规性。该方法允许随着新职业伦理规范的加入而扩展数据集。
技术框架：整体流程包括从职业伦理文本中自动生成任务，形成冲突请求与合法请求的配对，并通过人类专家进行验证。数据集的自动化生成使得每次运行都能产生新的任务。
关键创新：TenetBench的创新在于其自动化生成与职业伦理条款相冲突的请求，填补了现有方法在具体评估语言模型合规性方面的空白。
关键设计：在实验中，使用了多种模型和推理模式的组合，评估其在处理冲突请求时的表现。关键参数包括冲突请求的生成规则和人类验证的标准，确保了数据集的高质量和可靠性。
📊 实验亮点
实验结果显示，在评估的十四种模型中，有十一种在处理冲突请求时产生了冲突内容，且大多数未发出警告。启用推理功能后，两个模型的拒绝率减少了一半，表明推理能力在合规性方面的重要性。
🎯 应用场景
该研究的潜在应用领域包括法律、医疗、教育等专业领域，能够帮助机构评估和监控语言模型在实际应用中的合规性，从而减少潜在的伦理风险。未来，TenetBench可为特定职业和组织构建定制化的诊断数据集，提升语言模型的安全性和可靠性。
📄 摘要（原文）
Codes of professional ethics guide and restrict specialists, who increasingly work with assistants built on large language models. Such an assistant can breach the user's code, or help the user breach it, without the user noticing. Evaluations reporting broad categories of harm cannot show how often this happens or which provision is breached. TenetBench pairs 3,433 requests conflicting with clauses of fifteen Russian-language professional codes with benign twins that differ only in the legitimacy of what is asked. An automated pipeline generates the tasks from the text of a code, so the corpus grows with every code added and a provision yields new tasks on every run. It also produced tasks and human-validated verdicts from two English codes of other jurisdictions. In human validation of the frozen pipeline, specialists rated 0.995 of the conflicting tasks plausible, and the model judge agreed with the adjudicated label at Gwet's AC1 of at least 0.894 on 1,860 elements. We evaluated fourteen model and reasoning-mode combinations on whether they produce the conflicting text, decline or warn. Eleven of the fourteen produce conflicting content on most such requests, mostly without a warning, and nearly three warnings in four accompany conflicting content. The models that conform more do so by declining while still serving the legitimate twin, and enabling reasoning halved the refusals of two models. We hope the dataset and pipeline will help evaluate how language models observe professional norms and let diagnostic datasets be built for particular professions and organisations.