Train An Imaginative Agent
作者: Jiaxin Ge, Junyi Zhang, Tianning Chai, Hanjun Yoo, Chenyang Wang, Xiuyu Li, Weijia Shi, XuDong Wang, Sewon Min, Luke Zettlemoyer, Trevor Darrell, Haiwen Feng
分类: cs.AI
发布日期: 2026-10-19
💡 一句话要点
提出Agentic Imagination框架以解决Sokoban等环境中的规划问题
🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture)
关键词: 代理想象 强化学习 规划问题 决策优化 复杂环境
📋 核心要点
- 现有方法在面对稀疏反馈或不可逆操作时，难以有效预测行为后果，导致规划错误。
- Agima框架通过训练代理在行动前进行想象，结合政策和世界建模，提升代理的前瞻性和推理能力。
- 在Maze2D、Sokoban、Pacman和PushT等四个环境中，Agima在复杂任务中表现优异，解决率显著提高。
📝 摘要（中文）
在Sokoban等环境中，合理的推动可能导致难以解决的局面。为避免此类错误，代理需要在执行前预测其行为的后果。本文提出了Agentic Imagination（Agima）训练框架，旨在培养代理在行动前进行想象。Agima通过一个统一的全能模型来建议行动、生成结果图像，并推理这些想象的未来，以决定在真实环境中执行的操作。与单一计划不同，代理能够模拟一系列动作，探索替代方案并比较可能的未来。实验表明，Agima在多个环境中表现出色，能够在训练未见的更复杂设置中进行推广。
🔬 方法详解
问题定义：本文旨在解决在稀疏反馈和不可逆操作情况下，代理如何有效预测行为后果的问题。现有方法往往无法处理复杂的规划任务，导致决策失误。
核心思路：Agima框架的核心思想是培养代理在行动前进行想象，通过模拟未来的结果来优化决策过程。这种设计使代理能够在执行前评估多种可能的行动后果。
技术框架：Agima的整体架构包括两个主要阶段：首先通过监督微调（SFT）统一政策和世界建模，接着使用代理强化学习（RL）来训练代理有效利用想象的未来进行规划。
关键创新：Agima的主要创新在于引入了想象能力，使代理能够在决策时考虑多种未来情境，而不是仅依赖单一计划。这种能力使得代理在复杂环境中表现出更高的灵活性和适应性。
关键设计：在训练过程中，Agima使用了特定的损失函数来平衡预测准确性和想象能力，同时设计了适应性强的网络结构，以支持多种想象和推理任务。
🖼️ 关键图片
📊 实验亮点
实验结果显示，Agima在仅使用SFT训练的情况下，能够在两箱Sokoban任务中解决40%的四箱难题，而仅依赖动作规划的基线仅解决2%。经过强化学习后，Agima展现出新的行为模式，能够在关键决策点更频繁地进行分支思考。
🎯 应用场景
该研究的潜在应用领域包括机器人导航、游戏AI和自动化决策系统等。通过提升代理的想象能力，能够在复杂环境中实现更高效的规划和决策，具有重要的实际价值和未来影响。
📄 摘要（原文）
A seemingly reasonable push in Sokoban can make the puzzle unsolvable. To avoid such mistakes, an agent has to anticipate the consequences of its actions before execution, especially when feedback from the real environment is sparse or actions are irreversible. We introduce Agentic Imagination (Agima), a training framework that trains an agent to imagine before acting, i.e. to be imaginative. Agima trains a single omni model to propose actions, generate images of their outcomes, and then reason over these imagined futures to decide what to execute in the real environment. Rather than committing to a single plan, the agent can simulate a sequence of actions, explore alternatives, and compare possible futures. We first use supervised fine-tuning (SFT) to unify policy and world modeling within a single omni model. This enables the agent to predict future outcomes and perform basic forms of imagination, such as forward thinking and counterfactual branching. We then use agentic reinforcement learning (RL) to teach the agent to reason with its imagined futures and use them effectively for planning. Across four environments spanning Maze2D, Sokoban, Pacman, and PushT, Agima generalizes to harder settings unseen during training. Trained with SFT alone on two-box Sokoban, Agima solves 40% of four-box puzzles while seeing the real board only once every five moves, compared with an action-only planner that only solves 2%. After RL, new behaviors emerge in how Agima uses its imagination: it learns a new unified format that was unseen during the SFT stage, learns to conduct forward thinking only when necessary, and branches more frequently at critical decision points. An agent that can picture where its actions lead can learn when and where to look ahead, that is, to plan. Project page: https://agima.net/.