JevAgentHarness: Benchmarking, Deploying and Training Decision Models That Share Control with LLMs

📄 arXiv: 2610.21412v1 📥 PDF

作者: Kunlun Zhu, Tao Feng, Xu Han, Heng Wang, Ling Yang, Xiangru Tang, Jiaxuan You, Heng Ji

分类: cs.AI

发布日期: 2026-10-15

备注: Code: https://github.com/JevAgentHarness/JevAgentHarness

🔗 代码/项目: GITHUB

💡 一句话要点

提出JevAgentHarness以解决LLM决策效率低下问题

🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 大型语言模型 决策模型 代理任务 效率优化 成本降低 自适应调整

📋 核心要点

  1. 现有的LLM代理在执行长任务时效率低下，频繁调用LLM导致时间和成本增加。
  2. 本文提出JevAgentHarness，通过将决策任务委托给决策模型，减少LLM调用次数，从而提高效率。
  3. 实验显示，使用新模型AgentDecision-4B的代理在不同任务中成功率高达91.5%，且成本降低了81.1-89.0%。

📝 摘要（中文）

大型语言模型（LLM）代理现在能够修复软件、操作计算机和服务客户，但其运行程序在每一步都调用LLM，导致长任务变得缓慢且成本高昂。Jev等决策模型能够直接回答多项选择问题，显著降低了调用LLM的时间和费用。本文构建了AgentDecisionBench来评估决策模型在代理任务中的表现，并设计了JevAgentHarness，通过在代理自身轨迹上拟合信心阈值，优化了决策过程。实验结果表明，使用AgentDecision-4B模型的代理在与训练数据不同的任务中成功率达到91.5%，在相似任务中则显著降低了成本和提高了速度。相关模型和数据将公开发布。

🔬 方法详解

问题定义：本文旨在解决大型语言模型（LLM）在执行长任务时效率低下的问题。现有方法频繁调用LLM，导致时间和成本的显著增加，尤其是在多项选择决策时。

核心思路：论文提出的核心思路是通过引入决策模型（如Jev）来直接回答多项选择问题，从而减少对LLM的调用次数。通过在代理的轨迹上拟合信心阈值，优化决策过程，确保在不影响成功率的情况下提高效率。

技术框架：整体架构包括AgentDecisionBench用于评估决策模型的表现，DecisionGym环境用于训练和测试决策模型。JevAgentHarness则负责在代理执行任务时动态调整决策模型的使用。

关键创新：最重要的技术创新在于设计了JevAgentHarness，使得决策模型能够在代理的实际运行轨迹中进行自适应调整，解决了现有模型无法处理代理错误后的状态的问题。

关键设计：在模型训练中，使用了AgentDecision-4B模型，并通过Jev的答案作为标签进行训练。关键参数包括信心阈值的动态调整，以及在不同任务中的性能优化策略。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

实验结果显示，使用AgentDecision-4B模型的代理在与训练数据不同的任务中成功率达到91.5%，而在相似任务中则使代理的成本降低了81.1-89.0%，并且速度提高了最多2.5倍，且任务成功率未下降超过3个百分点。

🎯 应用场景

该研究的潜在应用领域包括智能客服、自动化软件修复和智能助手等。通过提高决策效率，JevAgentHarness能够显著降低运营成本，并提升用户体验，具有广泛的实际价值和未来影响。

📄 摘要（原文）

Large language model (LLM) agents now fix software, operate computers and serve customers, but their harnesses, the programs that run them, call a large LLM at every step, which makes long tasks slow and costly. Most of these calls end in a multiple-choice decision, such as which tool to call next. Decision models such as Jev answer these questions directly, at a fraction of an LLM call's time and price. We ask how a harness can delegate these decisions to a decision model with little loss of success. To evaluate decision models in agentic tasks, we build AgentDecisionBench, which tests them from isolated questions up to control of a running agent, on public benchmarks and our DecisionGym environment. The decision model most accurate on isolated questions solved only 69.5% of agent tasks, and without a fitted confidence threshold, delegating tool choices lowered success by 19.0 points. Building on these findings, we then design JevAgentHarness, which fits this threshold for each choice on the agent's own trajectories and lets the LLM decide below it. Because existing decision models never see the states an agent reaches after its own mistakes, we train AgentDecision-4B, an open 4B model, on such states with Jev's answers as labels. On tasks unlike its training data, the agent solved 91.5% of them with AgentDecision-4B and 82.0% with Jev. On tasks like their training data, 4B models trained this way made agents 81.1-89.0% cheaper and up to 2.5 times faster, without losing 3 or more points of task success. The models, data, and harness will be released at https://github.com/JevAgentHarness/JevAgentHarness.