OmniaBench: Benchmarking General AI Agents Across Diverse Scenarios
作者: Chengyu Shen, Yujie Fu, Gangtao Xin, Yanheng Hou, Wenlong Fei, Guojie Zhu, Jiawei Li, Hongcheng Gao, Runming He, Zhen Hao Wong, Meiyi Qiang, Hao Liang, Zhao Cao, Hao Jiang, Chong Chen, Wentao Zhang
分类: cs.CL, cs.AI
发布日期: 2026-07-16
💡 一句话要点
提出OmniaBench以解决通用AI代理评估的局限性问题
🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)
关键词: 通用AI代理 基准评估 层次化分类法 任务设计 能力评估
📋 核心要点
- 现有的代理基准集中于有限的场景和工具生态，难以全面评估模型能力。
- OmniaBench通过构建层次化分类法和可执行环境，提供多样化的评估任务。
- 实验结果显示，当前模型在OmniaBench上表现不佳，揭示了多方面的能力限制。
📝 摘要（中文）
随着大型语言模型从文本生成器演变为能够理解用户请求、调用外部工具并通过交互完成复杂任务的通用代理，现有的代理基准往往集中于有限的场景、工具生态或交互格式，难以系统性地表征模型在异构应用环境中的能力。为此，本文提出OmniaBench，一个用于评估通用代理在多样场景下的基准，构建了一个层次化的分类法，涵盖90个一级和354个二级领域，并设计了1,431个任务集。实验结果显示，当前前沿模型在OmniaBench上表现不佳，揭示了规划、约束维护和自适应纠正等方面的持续局限性。
🔬 方法详解
问题定义：本文旨在解决现有通用AI代理评估基准的局限性，现有方法往往集中于特定场景，无法全面表征模型在多样化应用中的能力。
核心思路：通过构建OmniaBench基准，利用应用导向的场景知识，形成层次化分类法，设计多样化的任务以全面评估代理能力。
技术框架：OmniaBench的整体架构包括四个主要模块：DAG、DAG-S、Solver和Program，支持单轮和多轮任务的执行。
关键创新：引入十维能力分类法和八个组成原子难度因子，支持细粒度的评估与分析，显著提升了评估的全面性与准确性。
关键设计：数据集中包含1,431个任务，其中644个任务为挑战性子集，旨在降低评估成本并减少潜在的污染风险。
🖼️ 关键图片
📊 实验亮点
在OmniaBench的评估中，Claude-Sonnet-5和GPT-5.6-Sol的Overall Pass@1得分仅为58.54和57.14，显示出当前前沿模型在多样化任务中的显著挑战，揭示了规划和适应性纠正等能力的不足。
🎯 应用场景
OmniaBench的研究成果可广泛应用于AI代理的性能评估与优化，尤其在智能助手、自动化工具和人机交互等领域。通过提供全面的能力评估，帮助开发者理解和改进AI模型的局限性，推动智能系统的进一步发展。
📄 摘要（原文）
Large language models are increasingly evolving from text generators into general agents capable of understanding user requests, invoking external tools, and completing complex tasks through interaction. However, existing agent benchmarks often focus on limited scenarios, tool ecosystems, or interaction formats, making it difficult to systematically characterize model capabilities across heterogeneous application settings. We introduce OmniaBench, a benchmark for evaluating general agents across diverse scenarios with explicit state spaces. We derive application-oriented scenario knowledge from app stores, product documents, industry resources, Web retrieval, and human refinement, forming a hierarchical taxonomy that spans ToC, ToB and ToE with 90 level-1 and 354 level-2 domains. Based on this taxonomy, we construct executable environments and synthesize single-turn and multi-turn tasks through four complementary routes: DAG, DAG-S, Solver, and Program. OmniaBench further introduces a ten-dimensional capability taxonomy and eight compositional atomic difficulty factors to support fine-grained evaluation and analysis. The resulting dataset contains 1,431 tasks, together with a challenging subset of 644 tasks designed to reduce evaluation cost and mitigate potential contamination of the full set after public release. The bench presents substantial challenges to current frontier models, with even Claude-Sonnet-5 and GPT-5.6-Sol achieving Overall Pass@1 scores of only 58.54 and 57.14, respectively. Further analyses reveal clear differences across domains and capabilities, as well as persistent limitations in planning, constraint maintenance, and adaptive correction. OmniaBench provides a broad and diagnostic benchmark for characterizing the capability boundaries of general agents.