GDP.pdf: Benchmarking Grounded Multimodal Reasoning over Professional PDF Documents
作者: Suhaas Garre, Emily Ritchie, Sushant Mehta, Edwin Chen
分类: cs.CV
发布日期: 2026-07-13
备注: 9 pages. Accepted at the 2nd Workshop on Knowledge-Intensive Multimodal Reasoning (KnowledgeMR) at CVPR 2026 (non-archival), under the paper's former title "PDFParse: A Benchmark for Grounded Multimodal Reasoning over Professional PDF Documents". Dataset: https://huggingface.co/datasets/surgeai/GDP.pdf ; Code: https://github.com/surge-ai/gdp-pdf
🔗 代码/项目: HUGGINGFACE
💡 一句话要点
提出GDP.pdf基准以解决专业PDF文档多模态推理问题
🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)
关键词: 多模态推理 文档AI PDF分析 专业领域 基准测试
📋 核心要点
- 现有文档AI基准往往孤立测量各项能力，无法反映模型在实际PDF文档中的综合表现。
- GDP.pdf基准通过专业人士撰写的问题-文档对，直接测量多模态推理能力，确保问题的实际应用价值。
- 在对七个前沿模型的评估中，最佳模型仅通过15%的项目，揭示了当前技术的局限性和改进空间。
📝 摘要（中文）
在专业领域，日常工作大多发生在PDF文件中，如福利包、租赁协议、数据表等。现有文档AI基准通常孤立地测量OCR、布局分析、图表推理等能力，无法真实反映模型在实际应用中的表现。GDP.pdf基准旨在直接测量这一能力，包含由十个领域的专业人士撰写的问题-文档对，确保问题具有实际意义。研究评估了七个前沿模型在100个项目上的表现，发现最佳模型仅通过15%的项目，错误主要源于表格对齐、图表读取等常见问题。该基准现已公开可用。
🔬 方法详解
问题定义：论文旨在解决现有文档AI基准无法真实反映模型在专业PDF文档中推理能力的问题。现有方法往往孤立测量各项能力，导致无法评估模型在实际应用中的有效性。
核心思路：论文提出GDP.pdf基准，通过专业人士撰写的问题-文档对，确保所测问题具有实际意义，从而直接评估模型的多模态推理能力。
技术框架：该基准包含100个问题-文档对，问题经过严格筛选，确保至少有两个前沿模型未能正确回答。每个项目配有评分标准和能力标签，涵盖文本提取、表格理解、空间推理等多个维度。
关键创新：最重要的创新点在于通过真实场景中的问题-文档对来评估模型，避免了以往基准的片面性，提供了更全面的评估标准。
关键设计：每个问题-文档对都有明确的评分标准，采用了分级评分机制，确保评估的客观性和准确性。模型评估时，重点关注模型在实际应用中可能遇到的错误类型，如表格对齐、图表读取等。
🖼️ 关键图片
📊 实验亮点
实验结果显示，评估的七个前沿模型在100个项目中表现不佳，最佳模型仅通过15%的项目，最差模型仅通过1%。这些结果揭示了当前技术在处理复杂文档时的局限性，尤其是在表格和图表的理解方面。
🎯 应用场景
该研究的潜在应用领域包括法律文档分析、医疗指南解读、工程图纸审核等，能够显著提升文档AI系统在专业领域的实用性和准确性。未来，GDP.pdf基准有望推动多模态推理技术的发展，促进更智能的文档处理工具的出现。
📄 摘要（原文）
A large share of day-to-day work in professional domains happens inside PDF files: benefits packets, leases, datasheets, clinical guidelines, construction plans. Benchmarks for document AI have generally measured the required capabilities in isolation: OCR, layout analysis, chart reasoning, table QA, document VQA. A high score on any one of them does not necessarily reveal whether a model can answer a realistic question that someone in the field would actually ask about a specific PDF. GDP.pdf is a benchmark built to measure this directly. It consists of question-document pairs authored by working professionals in ten fields, and a candidate question was kept only when at least two frontier multimodal models failed it in a way that mattered: a wrong answer, missed decisive evidence, or a fabricated claim, rather than a superficial difference such as style. Each item comes with a rubric of atomic criteria, so we can report a graded rubric score as well as a strict task-level pass rate, and each item is tagged against a taxonomy of eleven capabilities in three tiers, spanning text extraction and grounding, table and chart comprehension, cross-referencing, spatial reasoning, and abstention on unsupported queries. We evaluated seven frontier models on the 100-item benchmark. The best model passed only 15% of the items and the worst passed 1%. Most errors trace back to a small set of recurring loss patterns: misaligned tables, misread charts, skipped footnotes and exclusions, miscounted floor-plan symbols, scan noise, and amendments that supersede earlier text. The full 100-item benchmark is publicly available at https://huggingface.co/datasets/surgeai/GDP.pdf