Independent Multi-Agent Reinforcement Learning with Counterfactual Semantic-Social World Models

📄 arXiv: 2610.07704v1 📥 PDF

作者: Fernando Martinez, Tao Li, Yingdong Lu, Juntao Chen

分类: cs.MA, cs.LG

发布日期: 2026-10-06

💡 一句话要点

提出CASTLE框架以解决独立多智能体强化学习中的奖励信号模糊问题

🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture)

关键词: 多智能体强化学习 去中心化学习 反事实模拟 局部动态模型 语义社会模型

📋 核心要点

  1. 现有的独立多智能体强化学习方法在奖励信号的解释上存在模糊性，难以判断智能体行为的有效性。
  2. 本文提出CASTLE框架，通过对候选动作后果的前瞻性比较，提升智能体的学习效率，解决奖励信号模糊的问题。
  3. 在多个基准环境中，CASTLE的表现超越了最强基线，分别提高了10.67、6.46和0.33个标准化分数。

📝 摘要（中文）

完全去中心化的多智能体强化学习（MARL）要求每个智能体仅依赖其局部信息和经验进行学习和行动，缺乏集中式评估或智能体间通信。这种信息结构使得传统奖励信号变得模糊，导致难以判断奖励的来源。本文提出了CASTLE框架，通过前瞻性比较候选动作的后果，帮助智能体更有效地学习。CASTLE结合了局部动态世界模型和语义社会世界模型，前者总结了智能体的局部轨迹动态，后者则预测每个候选动作的短期任务和社会后果。实验结果表明，CASTLE在多个基准环境中表现优异，超越了现有最强基线。

🔬 方法详解

问题定义：本文旨在解决独立多智能体强化学习中奖励信号模糊的问题。现有方法无法有效区分智能体行为的有效性，导致学习效率低下。

核心思路：CASTLE框架通过前瞻性比较候选动作的后果，帮助智能体在缺乏集中式评估的情况下进行更有效的学习。此设计使得智能体能够在局部信息的基础上进行决策。

技术框架：CASTLE包含两个主要模块：局部动态世界模型和语义社会世界模型。局部动态模型在离线阶段预训练，捕捉智能体的局部轨迹动态；语义社会模型则通过反事实模拟器预测候选动作的短期后果。

关键创新：CASTLE的创新之处在于引入了反事实模拟器，能够展示不同动作下的潜在队友和对手反应，从而为智能体提供更丰富的决策依据。

关键设计：在CASTLE中，局部动态世界模型和语义社会世界模型在在线学习和执行阶段保持不变，智能体仅使用局部信息进行查询。模型的输出为独立PPO策略提供了上下文指导。

🖼️ 关键图片

fig_0
img_1
img_2

📊 实验亮点

在30个匹配种子下，CASTLE在Tag、Spread和Adversary等基准多粒子环境中表现出色，分别超越最强基线10.67、6.46和0.33个标准化分数，显示出显著的性能提升。

🎯 应用场景

该研究的潜在应用领域包括多智能体系统中的协作任务，如无人机编队、自动驾驶车辆的协同决策等。通过提升智能体的学习效率，CASTLE框架能够在复杂环境中实现更高效的决策，具有重要的实际价值和未来影响。

📄 摘要（原文）

Fully decentralized multi-agent reinforcement learning (MARL), also referred to as independent learning, requires each agent to learn and act using only its local information and experience, without a centralized critic or inter-agent communication. Such a stringent information structure renders the conventional reward signal ambiguous. A poor return may result from an ineffective ego action, an incompatible teammate response, or an effective opponent response, yet scalar rewards alone do not reveal which explanation is responsible. We argue that agents can learn more effectively by prospectively comparing the consequences of candidate actions rather than diagnosing failures only from realized returns. We introduce CASTLE (Counterfactual Action-conditioned Semantic Tokens for Local Execution in Decentralized MARL), an offline-training, online-in-context guidance framework with two complementary world models. A Local Dynamics World Model, offline pre-trained over agents' local trajectories, summarizes the agent's local trajectory dynamics and partial observability, while a Semantic-Social World Model predicts compact short-horizon task and social consequences for each candidate ego action. The latter is trained from counterfactual simulator rollouts that expose plausible teammate and opponent responses to alternative actions taken from the same logged rollout state. During online learning and execution, both world models remain frozen and are queried by agents using only locally available information. Their prediction logits provide in-context guidance to an independent PPO policy. Across 30 matched seeds on Tag, Spread, and Adversary in the benchmark multi-particle environments, our proposed CASTLE achieves the highest mean final score among the evaluated methods, exceeding the strongest baseline on each task by 10.67, 6.46, and 0.33 normalized points, respectively.