OpenTWBench (Open Taiwan Benchmark): Taiwan-Domain Knowledge Evaluation for Large Language Models, and How the Measurement Apparatus Shapes the Score
作者: Liang-Hsun Huang, Min-Yi Chen, Chao-Chun Chuang, Yu-Wei Liu, Che-Yu Wu, Po-Feng Lee
分类: cs.CL
发布日期: 2026-10-14
备注: 19 pages, no figures. Datasets, per-item difficulty data and a public leaderboard at https://opentwbench.ai
💡 一句话要点
提出OpenTWBench以评估大型语言模型的台湾领域知识
🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)
关键词: 大型语言模型 知识评估 台湾领域 OpenTWBench 教育评估 数据清洗 模型优化
📋 核心要点
- 现有大型语言模型在特定领域知识评估中存在不足，尤其是针对台湾的知识缺乏系统性评估工具。
- 论文提出OpenTWBench，通过tw-exam-bench和tw-formosa-bench两个基准，系统评估大型语言模型在台湾领域的知识掌握情况。
- 实验结果显示，模型在相同问题上的表现受测量工具设计影响显著，某些设置可使准确率从20.8%提升至93.9%。
📝 摘要（中文）
我们提出了OpenTWBench，这是一个针对大型语言模型在台湾领域知识评估的开放评估套件，包含两个基准。tw-exam-bench包含来自台湾国家考试的255,854道四选一问题，涵盖148个学科，经过严格的审查和数据清洗，确保了高达99.91%的答案一致性。tw-formosa-bench则包含1,696道关于当代台湾的难度精心策划的问题，旨在填补现有考试未覆盖的领域。我们还记录了评分结果中来自测量工具而非模型的影响，展示了不同设置对模型表现的显著影响。所有数据集、难度数据和排行榜均已公开发布。
🔬 方法详解
问题定义：本研究旨在解决大型语言模型在台湾领域知识评估中的不足，现有方法缺乏针对特定文化和社会背景的系统性评估工具。
核心思路：通过构建OpenTWBench，提供两个基准来全面评估模型在台湾领域的知识，确保评估的科学性和准确性。
技术框架：整体架构包括tw-exam-bench和tw-formosa-bench两个模块，前者基于国家考试数据，后者基于社区策划的日常问题，确保覆盖广泛的知识领域。
关键创新：最重要的创新在于通过精心设计的评估工具，揭示了测量工具对模型评分的重大影响，强调了评估设计的重要性。
关键设计：在数据清洗过程中，确保了高达99.91%的答案一致性，且通过多轮审查剔除不合格问题，确保最终使用的253,674道题目具有高质量。
🖼️ 关键图片
📊 实验亮点
实验结果显示，模型在相同问题上的准确率因测量工具的不同而变化显著，例如从模式匹配到显式答案提取，准确率提升至93.9%。此外，模型在排行榜上的表现也显示出显著的提升，22个模型的得分超过了原有的上限91.6%。
🎯 应用场景
该研究的潜在应用领域包括教育评估、语言模型的训练与优化，以及针对特定文化背景的知识图谱构建。OpenTWBench的发布将推动台湾领域知识的研究和应用，提升语言模型在特定领域的表现。
📄 摘要（原文）
We present OpenTWBench, an open evaluation suite for Taiwan-domain knowledge in large language models, built around two benchmarks. tw-exam-bench contains 255,854 four-option questions across 148 academic subjects, from fifteen years of Taiwan's national examinations published as open data by the Ministry of Examination; answer keys agree across 31,312 duplicated question groups at 99.91%, with all 27 disagreements accounted for. A reviewed second pass withdrew 2,180 items whose stems depend on a figure the text does not carry; all scores use the remaining 253,674. tw-formosa-bench contains 1,696 difficulty-curated questions about everyday and contemporary Taiwan - the Taiwan that licensing examinations never ask about - generated from a community-curated corpus under a discipline in which no model writes a ground truth, and curated against a frozen seven-model pool so that the pool scored at the passing margin (46.9-74.6%, median ~54%); a leave-one-out experiment shows the selection preserves model ranking (near-zero bias). Within six weeks the frontier had cleared that band: of 108 models now on the public board, 22 score above its ceiling, the best 91.6%. We also document how much of a score comes from the apparatus rather than the model. On identical questions, switching answer extraction from pattern matching to an explicit \boxed{} answer moves accuracy from 20.8% to 93.9%; a single placeholder character in a system prompt moved one model from last (58.2%) to first (83.9%) on the same benchmark; doubling the thinking budget from 8k to 16k tokens bought 0.2% while shuffling which questions failed; and a three-judge validation panel proved to be one judge in disguise, two members approving 92-99% of everything shown. We release all datasets, per-item difficulty data, curation manifests and a leaderboard at https://opentwbench.ai