OpenForgeRL: Train Harness-native Agents in Any Environment
作者: Xiao Yu, Baolin Peng, Ruize Xu, Hao Zou, Qianhui Wu, Hao Cheng, Wenlin Yao, Nikhil Singh, Zhou Yu, Jianfeng Gao
分类: cs.AI, cs.CL
发布日期: 2026-07-23 (更新: 2026-07-24)
备注: updated the paper header to show ICLR2027 instead of ICLR2026 (already past)
💡 一句话要点
提出OpenForgeRL以解决复杂环境中代理训练问题
🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)
关键词: 强化学习 代理训练 多模态推理 开源框架 Kubernetes 工具使用 复杂环境
📋 核心要点
- 现有的推理工具使得AI代理的端到端训练变得复杂，难以在开放基础设施上实现。
- OpenForgeRL通过轻量级代理和Kubernetes调度器解耦训练与推理，支持在多样化环境中进行代理训练。
- 实验结果显示，OpenForgeRL在多个基准测试中超越了同类模型，且在GUI设置中表现优于更大规模的模型。
📝 摘要（中文）
现代AI代理依赖复杂的推理工具进行多轮推理和工具使用，但这些工具使得代理的端到端训练变得困难。为此，本文提出了OpenForgeRL，一个开源框架，旨在多样化环境中训练基于工具的代理。OpenForgeRL通过轻量级代理记录模型调用作为训练数据，并利用Kubernetes调度器在远程容器中运行每次训练，使得在任何环境中进行大规模训练成为可能。实验结果表明，OpenForgeRL在多个基准测试中表现优异，超越了同类开放基线。
🔬 方法详解
问题定义：本文解决的问题是如何在复杂环境中有效训练基于推理工具的AI代理。现有方法在多进程推理和状态管理方面存在局限，导致训练过程不够灵活和高效。
核心思路：OpenForgeRL的核心思路是通过轻量级代理记录模型调用，并利用Kubernetes进行容器化管理，从而实现训练与推理的解耦。这种设计使得研究人员能够在真实环境中直接训练和改进代理。
技术框架：OpenForgeRL的整体架构包括一个轻量级代理模块，用于捕获模型调用和生成训练数据，以及一个Kubernetes调度器，负责在独立的远程容器中运行每次训练。这种架构支持大规模的多样化环境训练。
关键创新：OpenForgeRL的主要创新在于其解耦训练与推理的能力，使得代理能够在真实环境中进行高效训练。这与现有方法的紧耦合设计形成鲜明对比，显著提升了训练的灵活性和可扩展性。
关键设计：在关键设计方面，OpenForgeRL采用了标准的强化学习代码库（如veRL），并通过记录模型调用的方式生成训练数据。此外，Kubernetes的使用确保了每次训练的隔离性和资源管理的高效性。
🖼️ 关键图片
📊 实验亮点
在多个基准测试中，OpenForgeRL的表现显著优于同类开放基线，例如在ClawEval上达到31.7 pass^3和55.9 pass@3，在GUI设置中，OpenForgeGUI在OSWorld-Verified上达到37.7，在线测试中表现更佳，显示出其在复杂环境中的训练能力。
🎯 应用场景
OpenForgeRL的研究成果具有广泛的应用潜力，特别是在需要复杂推理和工具使用的AI代理领域，如自动化客服、智能助手和机器人控制等。通过提供一个灵活的训练框架，研究人员和开发者能够更高效地开发和优化智能代理，推动相关技术的进步。
📄 摘要（原文）
Modern AI agents rely on elaborate inference harnesses such as Claude Code, Codex, and OpenClaw to drive multi-turn reasoning, tool use, and access to external systems. While powerful, these complex harnesses also make agents hard to train end-to-end with open infrastructure, whose SFT/RL stacks cannot natively express stateful, multi-process harness inference. To address this, we present OpenForgeRL, an open-source framework for training harness-based agents end-to-end in diverse environments. OpenForgeRL achieves this with a lightweight proxy that serves the harness's model calls while recording them as training data for a standard RL codebase (e.g., veRL), and a Kubernetes orchestrator that runs each rollout in its own remote container, together enabling training on any harness in any environment at scale. By decoupling training and inference, OpenForgeRL allows researchers to easily train, study, and improve agents directly in the real harnesses and environments they are deployed with. We validate our framework across diverse, complex harnesses and environments, spanning tool/claw-based agents and multimodal GUI browser- and computer-use agents. Using only hundreds to a few thousand tasks, OpenForgeClaw reaches 31.7 pass^3 and 55.9 pass@3 on ClawEval and 33.7 on QwenClawBench. OpenForgeGUI reaches 37.7 on OSWorld-Verified, 63.0 on Online-Mind2Web, and 72.3 on WebVoyager. Both outperform open baselines of similar size on nearly all benchmarks, and in the GUI setting match or surpass models several times larger. Beyond benchmarks, we analyze how harness choice (e.g., ZeroClaw, OpenClaw, Codex) and RL shape agent behavior. We find that some harnesses are substantially harder to learn than others, and that RL improves agentic reliability, such as self-verification, tool coverage, and completing multi-step plans, though critical abilities such as error recovery remain weak.