SyntaxBench: A Statistical Diagnostic Framework for Character-Level Reasoning in Large Language Models

📄 arXiv: 2610.03329v1 📥 PDF

作者: Mohsen Larni, Sobhan Ebrahimi Azar, Pouyan Nahed, Kazem Taghva

分类: cs.CL, cs.AI, cs.LG

发布日期: 2026-10-02

备注: 32 pages, 17 figures. The first two authors contributed equally. The code will be released soon

💡 一句话要点

提出SyntaxBench框架以解决大语言模型字符级推理评估问题

🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 字符级推理 语言模型评估 统计框架 自然语言处理 推理模式

📋 核心要点

  1. 现有方法在评估字符级推理时多依赖聚合准确率，缺乏对输入控制和推理模式的分析。
  2. 本文提出SyntaxBench框架，通过设计多个核心任务和统计评估方法，系统性地评估字符级推理能力。
  3. 实验结果显示，tokenization显著影响准确率，推理模式效果不一，index_to_span任务的最佳准确率仅为6.75%。

📝 摘要（中文）

随着大语言模型在小语法错误场景中的应用增加，字符级推理的评估仍主要依赖孤立探针和聚合准确率。本文提出SyntaxBench，一个用于字符级推理的诊断基准和统计评估框架。该框架包含五个核心任务：字符计数、字母包含、回文检测、编辑距离和最长字符串选择，以及一个更具挑战性的子字符串提取测试index_to_span。通过对八个开放权重模型的评估，发现tokenization对准确率的影响显著，推理模式的效果不均匀，且index_to_span任务仍未得到有效解决。

🔬 方法详解

问题定义：本文旨在解决大语言模型在字符级推理评估中的不足，现有方法缺乏对输入控制和推理模式的深入分析，导致评估结果不够全面和准确。

核心思路：提出SyntaxBench框架，通过设计五个核心任务和一个挑战性任务，结合多种统计评估方法，全面评估模型的字符级推理能力。

技术框架：SyntaxBench框架包括五个核心任务（字符计数、字母包含、回文检测、编辑距离、最长字符串选择）和一个更难的index_to_span任务，使用零、一个和四个示例的提示进行评估。

关键创新：SyntaxBench的创新在于其系统化的评估方法，强调了tokenization和推理模式对准确率的影响，提供了比传统聚合准确率更全面的评估视角。

关键设计：在任务设计中，核心任务使用配对的英语和随机字符串输入，index_to_span任务则使用200-500字的文档，且不匹配字符长度。评估中采用了Cohen's kappa、McNemar测试等多种统计方法。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

实验结果表明，tokenization对准确率的影响显著，随机字符串的字符可见性高于英语字符串，字符计数准确率随着英语单词占用更多tokens而下降。此外，推理模式的效果不均匀，某些模型在特定任务上表现不佳，index_to_span任务的最佳四-shot准确率仅为6.75%。

🎯 应用场景

该研究的潜在应用领域包括自然语言处理、文本分析和机器翻译等，能够帮助研究人员更好地理解和评估大语言模型在字符级推理方面的能力，推动相关技术的进步与应用。未来，SyntaxBench框架可能成为评估语言模型推理能力的标准工具。

📄 摘要（原文）

Large language models are increasingly used where small syntactic errors matter, yet character-level reasoning is still evaluated mostly through isolated probes and aggregate accuracy. We introduce SyntaxBench, a diagnostic benchmark and statistical evaluation framework for character-level reasoning. It contains five core tasks, character counting, letter containment, palindrome detection, edit distance, and longest-string selection, plus index_to_span, a harder substring-extraction stress test. The five core tasks use paired English and character-length-matched random-string inputs. index_to_span documents share a 200-500 word band and are not character-length matched. All six tasks use zero-, one-, and four-shot prompts. We evaluate eight open-weight models from 2B to 32B parameters across 11 reasoning-mode configurations. The framework reports exact-match and relaxed accuracy, Cohen's kappa, paired McNemar tests with odds ratios, bootstrap confidence intervals, Kendall's tau, class-conditional metrics, tokenization analysis, and multiple-comparison-corrected tests. Three findings stand out. First, tokenization shapes accuracy: random strings are more character-visible than English strings (1.892 vs. 3.169 characters per token), and character-counting accuracy falls as English words occupy more tokens. Second, reasoning mode is not uniformly helpful: Gemma4-31B is nearly unchanged across modes on the near-saturated tasks, while Qwen3.6-27B is worse with thinking on palindrome detection (0.952 non-thinking vs. 0.886 thinking at four-shot). Third, index_to_span remains largely unsolved; the best four-shot exact-match accuracy is 6.75%. Character-level evaluation needs controlled inputs, paired tests, and analyses of tokenization and reasoning mode rather than aggregate accuracy alone.