PAWA: Panoramic Action-conditioned World Modeling for Multiple Agents

📄 arXiv: 2610.20240v1 📥 PDF

作者: Siqi Yang, Yimu Wang, Ziqi Cai, Chengxuan Zhu, Shuchen Weng, Erwin Wu, Zhaofei Yu, Kaipeng Zhang, Boxin Shi

分类: cs.CV, cs.AI

发布日期: 2026-10-14

备注: 24 pages. Project page: https://pawa.siki.moe

💡 一句话要点

提出PAWA以解决多玩家环境建模问题

🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture)

关键词: 多玩家建模 全景视频 动作条件建模 交叉注意力 Minecraft

📋 核心要点

  1. 现有方法无法有效捕捉多个玩家在共享环境中独立动作对环境的影响，尤其是视野外的变化。
  2. 论文提出PAWA数据集，结合POV视频和全景视频，通过双塔模型实现对两者的联合预测，并引入交叉注意力机制。
  3. 实验结果显示，PAWA在POV预测质量上超越了现有的多玩家世界模型，并且全景预测的保真度也很高。

📝 摘要（中文）

多玩家世界模型必须预测多个玩家的独立动作如何改变共享环境，包括玩家当前视野之外的变化。现有的POV视频只能在变化进入视野后捕捉这些变化，而全景视频则能记录所有方向的环境。本文提出PAWA，一个用于动作条件下的全景多玩家世界建模的数据集和框架。该数据集将每个玩家的POV视频与世界稳定的全景视频及对齐的动作配对。实验表明，PAWA在POV预测质量上优于现有的多玩家世界模型，同时也能生成高保真的全景预测。

🔬 方法详解

问题定义：本文旨在解决多玩家环境建模中，如何有效预测多个玩家独立动作对共享环境的影响，尤其是视野外的变化。现有方法缺乏同步的POV和全景视频数据，导致模型性能受限。

核心思路：提出PAWA数据集和框架，通过将POV视频与全景视频及玩家动作对齐，利用双塔模型实现对两者的联合预测。设计交叉注意力机制以对齐共享世界框架中观察相同方向的tokens。

技术框架：整体架构包括数据采集、视频对齐和模型训练三个主要模块。数据采集采用分布式管道，解耦CPU游戏捕获与GPU渲染，支持64个以上玩家的同时游戏。

关键创新：引入了二次相位玩家编码，提供比现有旋转编码更多的玩家代码，且在相同嵌入维度下实现更高的表达能力。

关键设计：模型采用双塔结构，损失函数设计为联合优化POV和全景视频的预测质量，网络结构中引入了交叉注意力层以增强信息流动。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

实验结果表明，PAWA在POV预测质量上超越了现有的多玩家世界模型，具体性能提升幅度达到XX%（具体数据未知），同时生成的全景预测也展现出高保真度，验证了模型的有效性和实用性。

🎯 应用场景

该研究的潜在应用领域包括在线多人游戏、虚拟现实和增强现实等场景，能够帮助开发更智能的游戏AI和增强玩家体验。未来，PAWA框架可能推动多玩家环境建模的研究进展，为复杂交互场景下的智能体行为预测提供新的思路。

📄 摘要（原文）

A multiplayer world model must predict how the independent actions of several players change a shared environment, including changes outside a player's current field of view. A point-of-view (POV) video captures such changes only after they enter its field of view, whereas panoramic video records the surroundings in every viewing direction and has served as an intermediate representation of the world state in recent world models. Learning this task therefore requires synchronized POV and panoramic videos of interacting players together with their actions, yet no existing dataset provides this combination in Minecraft, an established testbed in which players act independently in a shared, modifiable 3D world. We present PAWA, a dataset and framework for action-conditioned panoramic multiplayer world modeling. The dataset pairs each player's POV video with world-stabilized panoramic video and aligned actions. Its distributed pipeline decouples CPU gameplay capture from GPU rendering, scales beyond 64 simultaneous players, and re-renders each recorded session under new observation configurations without replaying the gameplay. On this dataset, we develop a dual-tower model that jointly predicts POV and panoramic video and couples the two streams through cross-attention that aligns tokens observing the same direction in a shared world frame. We further introduce a quadratic-phase player encoding that provides many more player codes than existing rotary codes at the same embedding dimension. Experiments show that PAWA outperforms state-of-the-art multiplayer world models in POV prediction quality while also producing high-fidelity panoramic predictions. The dataset and collection pipeline are open source to support further research on multiplayer world models.