Learning in Dreams, Winning in Reality: A Continuous Dyna Loop for a Ten-Hero MOBA

📄 arXiv: 2610.08033v1 📥 PDF

作者: Jordy Kieto

分类: cs.AI

发布日期: 2026-10-06

备注: 15 pages, 11 figures. Code, policies, evaluation and videos: https://github.com/JordyKieto/puffermoba-dyna

💡 一句话要点

提出连续Dyna循环以提升十英雄MOBA游戏中的策略表现

🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture)

关键词: 多智能体系统 世界模型 策略训练 电子竞技 Dyna循环

📋 核心要点

  1. 现有的世界模型评估方法主要依赖于内部指标，缺乏对真实游戏表现的直接反馈，导致模型在实际应用中效果不佳。
  2. 论文提出了一种基于连续Dyna循环的训练方法，通过在虚拟环境中训练策略并在真实游戏中评估，提升了策略的实际表现。
  3. 实验结果显示，策略在真实游戏中的获胜率从0%提升至70.2%，显著优于仅依赖虚拟训练的策略表现。

📝 摘要（中文）

本研究从外部评估世界模型，通过学习一个结构化的多智能体世界模型，训练策略并在真实游戏中进行评估。研究表明，使用1,400个虚拟回合进行自由运行的想象训练，策略在真实游戏中获胜率达到70.2%。此外，模型在训练数据上的准确性与策略自身游戏的表现存在显著差异，Dyna循环能够有效修正这一问题。最后，策略在宏观游戏表现良好，但在具体机制如控制、施法时机等方面存在不足。

🔬 方法详解

问题定义：本研究旨在解决现有世界模型在真实游戏中的应用效果不佳的问题。传统方法多依赖于内部评估指标，未能有效反映策略在实际对抗中的表现。

核心思路：论文提出通过构建一个结构化的多智能体世界模型，结合连续Dyna循环的训练方式，使策略在虚拟环境中进行训练，并在真实游戏中进行评估，从而提升策略的实际表现。

技术框架：整体架构包括世界模型的构建、策略的训练和真实游戏的评估三个主要模块。首先在虚拟环境中进行自由运行的想象训练，然后将训练得到的策略应用于真实游戏中进行在线评估。

关键创新：最重要的创新点在于将虚拟训练与真实游戏评估相结合，形成了一个持续的反馈循环。这种方法能够有效修正模型在真实游戏中的表现偏差，提升策略的实际获胜率。

关键设计：在模型训练中，使用了1,400个虚拟回合进行自由运行的想象训练，评估策略时采用了在线评估机制。此外，模型在训练数据上的准确性与策略自身游戏的表现存在显著差异，Dyna循环能够有效修正这一问题。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

实验结果显示，策略在真实游戏中的获胜率达到70.2%，相比于仅依赖虚拟训练的策略表现（0%）和之前的33.7%有显著提升。这一结果表明，连续Dyna循环的有效性和模型修正能力。

🎯 应用场景

该研究的潜在应用领域包括电子竞技、游戏AI开发和多智能体系统的优化。通过提升策略在复杂环境中的表现，能够为游戏开发者提供更强大的AI对手，同时也为其他领域的智能体训练提供借鉴。

📄 摘要（原文）

World models are usually judged from the inside: by prediction loss, by the return a policy earns in imagination, or by how convincing their frames look. We judge one from the outside. We learn a structured, multi-agent world model of a complete ten-hero MOBA (206 units, every hero acting every tick, games of up to 6,000 ticks), train a policy only inside it with 1,400-tick free-running imagined episodes, and measure that policy in the real game against the opponent the game ships with. The real game never provides a gradient; it provides the policy's own games as training data for the world model, and an online evaluation that selects and anchors the policy. Run as a continuous asynchronous Dyna loop, the policy wins 70.2% of real games as radiant (421 of 600; 95% CI 66.4-73.7) on seeds never used for any decision, up from 0% for dream training alone and 33.7% before the loop. It wins none as dire, and neither does the shipped opponent when it plays itself. Four findings explain the result. Model exploitation is invisible from inside the dream: every unanchored run collapsed within a few updates while no in-dream metric tracked the collapse. A world model that is accurate on its training corpus is badly wrong on the policy's own games, and Dyna repairs it there, which is worth +9.2 points of real win rate with the policy recipe held fixed. Finally, the policy inherits its world model's fidelity profile mechanic by mechanic: the model represents the macro game but not crowd control, cast timing or lethality, and the policy wins by map-wide pressure with almost no coordinated fighting. We release the world model, the dream-PPO harness, a world-model debugger, the evaluation protocol, and every policy and log.