OpenForgeRL: Train Harness-native Agents in Any Environment
作者: Xiao Yu, Baolin Peng, Ruize Xu, Hao Zou, Qianhui Wu, Hao Cheng, Wenlin Yao, Nikhil Singh, Zhou Yu, Jianfeng Gao
分类: cs.AI, cs.CL
发布日期: 2026-07-23
💡 一句话要点
提出OpenForgeRL以解决复杂环境中代理训练难题
🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)
关键词: 代理训练 开源框架 多模态交互 Kubernetes 强化学习
📋 核心要点
- 现有的AI代理训练方法依赖复杂的推理工具，导致端到端训练困难，尤其是在多进程和状态管理方面。
- OpenForgeRL通过轻量级代理和Kubernetes调度器，解耦训练与推理，支持在多样化环境中进行大规模训练。
- 实验结果显示，OpenForgeRL在多个基准测试中表现优异，超越了同类开放基线，验证了其有效性和可靠性。
📝 摘要（中文）
现代AI代理依赖复杂的推理工具来实现多轮推理、工具使用和外部系统访问。然而，这些复杂的工具使得代理的端到端训练变得困难。为此，本文提出了OpenForgeRL，一个开源框架，旨在多样化环境中训练基于工具的代理。OpenForgeRL通过轻量级代理记录模型调用作为训练数据，并利用Kubernetes调度器在独立容器中运行每次训练，支持在任何环境中进行大规模训练。实验结果表明，该框架在多个基准测试中超越了现有的开放基线，验证了其有效性。
🔬 方法详解
问题定义：本文旨在解决现有AI代理训练中因复杂推理工具导致的端到端训练困难，尤其是在多进程和状态管理方面的挑战。
核心思路：OpenForgeRL通过引入轻量级代理来记录模型调用，并利用Kubernetes调度器在独立容器中运行训练，解耦训练与推理，从而实现灵活的环境适应性。
技术框架：该框架包括两个主要模块：轻量级代理用于捕捉模型调用并记录训练数据，Kubernetes调度器用于管理训练过程中的容器化环境，确保每次训练的独立性和可扩展性。
关键创新：OpenForgeRL的核心创新在于其解耦训练与推理的能力，使得研究人员能够在真实环境中直接训练和改进代理，这与现有方法的紧耦合设计形成鲜明对比。
关键设计：在实现中，框架采用标准RL代码库（如veRL）作为基础，轻量级代理负责模型调用的记录，Kubernetes调度器则确保每个训练回合在独立的远程容器中运行，支持大规模的训练任务。具体的参数设置和损失函数设计尚未详细披露。
🖼️ 关键图片
📊 实验亮点
在多个基准测试中，OpenForgeClaw在ClawEval上达到31.7 pass^3和55.9 pass@3，OpenForgeGUI在OSWorld-Verified上达到37.7，在线测试中表现优于同类开放基线，且在GUI设置中与更大模型的表现相当或更优。
🎯 应用场景
OpenForgeRL的潜在应用领域包括智能助手、自动化工具和多模态交互系统等。通过提高代理的训练效率和可靠性，该框架能够推动AI在复杂环境中的应用，提升用户体验和系统性能。未来，OpenForgeRL可能会在更多实际场景中得到应用，促进AI技术的进一步发展。
📄 摘要（原文）
Modern AI agents rely on elaborate inference harnesses such as Claude Code, Codex, and OpenClaw to drive multi-turn reasoning, tool use, and access to external systems. While powerful, these complex harnesses also make agents hard to train end-to-end with open infrastructure, whose SFT/RL stacks cannot natively express stateful, multi-process harness inference. To address this, we present OpenForgeRL, an open-source framework for training harness-based agents end-to-end in diverse environments. OpenForgeRL achieves this with a lightweight proxy that serves the harness's model calls while recording them as training data for a standard RL codebase (e.g., veRL), and a Kubernetes orchestrator that runs each rollout in its own remote container, together enabling training on any harness in any environment at scale. By decoupling training and inference, OpenForgeRL allows researchers to easily train, study, and improve agents directly in the real harnesses and environments they are deployed with. We validate our framework across diverse, complex harnesses and environments, spanning tool/claw-based agents and multimodal GUI browser- and computer-use agents. Using only hundreds to a few thousand tasks, OpenForgeClaw reaches 31.7 pass^3 and 55.9 pass@3 on ClawEval and 33.7 on QwenClawBench. OpenForgeGUI reaches 37.7 on OSWorld-Verified, 63.0 on Online-Mind2Web, and 72.3 on WebVoyager. Both outperform open baselines of similar size on nearly all benchmarks, and in the GUI setting match or surpass models several times larger. Beyond benchmarks, we analyze how harness choice (e.g., ZeroClaw, OpenClaw, Codex) and RL shape agent behavior. We find that some harnesses are substantially harder to learn than others, and that RL improves agentic reliability, such as self-verification, tool coverage, and completing multi-step plans, though critical abilities such as error recovery remain weak.