DRG-MAPPO: Hierarchical Dynamic Role-Graph Multi-Agent Reinforcement Learning for Cooperative Air Combat

📄 arXiv: 2609.11155v1 📥 PDF

作者: Junlin Liu, Chengwei Li, Yang Gao, Hui Chang, Xinchen Zhang, Zhijun Zhao, Hao Zhao

分类: cs.AI

发布日期: 2026-09-10

💡 一句话要点

提出DRG-MAPPO以解决空战中的复杂战术协调问题

🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture)

关键词: 多智能体强化学习 空战 图形建模 动态角色分配 战术协调 深度学习

📋 核心要点

  1. 现有多智能体强化学习方法在空战中面临复杂战术协调的挑战，主要由于缺乏有效的关系建模和角色分配机制。
  2. 本文提出的DRG-MAPPO框架通过图形建模和动态角色分配，提升了智能体间的协作能力和战术执行效率。
  3. 实验结果显示，DRG-MAPPO在空战场景中实现了87%的胜率，显著优于传统方法，验证了其有效性。

📝 摘要（中文）

多智能体强化学习（MARL）已成为自主系统和空战中复杂决策的重要范式。然而，实现复杂的战术协调仍然面临挑战，主要由于缺乏结构化的关系建模和传统扁平架构无法明确建模战术角色。为此，本文提出了层次动态角色图多智能体近端策略优化（DRG-MAPPO）框架，结合图形关系建模和动态角色分配。该框架通过图注意力机制提取战场实体间的关键关系特征，并通过高层策略动态分配战术责任，低层策略则执行离散机动动作。实验结果表明，DRG-MAPPO在空战中实现了87%的胜率，表明其在关系建模、可解释性和优化稳定性方面的有效性。

🔬 方法详解

问题定义：本文旨在解决空战中多智能体的复杂战术协调问题。现有方法的痛点在于缺乏结构化的关系建模，导致智能体无法有效捕捉战场实体间的复杂互动，以及传统扁平架构无法明确分配战术角色。

核心思路：DRG-MAPPO框架通过构建图形化的战场互动表示，结合动态角色分配机制，提升智能体的协作能力。高层策略负责角色分配，低层策略则执行具体的战术动作，从而实现战术策略的联合优化。

技术框架：DRG-MAPPO的整体架构包括图形关系建模模块、动态角色分配模块和低层策略执行模块。图形模块利用图注意力机制提取关键关系特征，角色分配模块根据这些特征确定战术责任，最后低层策略执行具体的机动动作。

关键创新：该框架的主要创新在于将图形关系建模与动态角色分配相结合，解决了传统方法在复杂环境中角色分配模糊的问题，提升了智能体的协作效率。

关键设计：在设计中，采用了图注意力机制来提取关系特征，并设计了目标优先级辅助任务，以促进智能体间的协作行为，如集中火力。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

实验结果表明，DRG-MAPPO在空战场景中实现了87%的胜率，显著高于传统方法，展示了其在关系建模和战术执行方面的优势。这一结果表明，DRG-MAPPO能够有效平衡关系建模、可解释性和优化稳定性。

🎯 应用场景

该研究的潜在应用领域包括军事空战、无人机编队、智能交通系统等复杂决策场景。通过提升多智能体系统的协作能力，DRG-MAPPO能够在动态环境中实现更高效的任务执行，具有重要的实际价值和未来影响。

📄 摘要（原文）

Multi-Agent Reinforcement Learning (MARL) has emerged as a pivotal paradigm for complex decision-making in autonomous systems and air combat. While MARL has demonstrated significant potential in air combat, achieving sophisticated tactical coordination remains a non-trivial challenge. This difficulty is largely attributed to two primary limitations: (1) the absence of structured relational modeling hinders agents from capturing complex, time-varying interactions among battlefield entities; and (2) conventional flat architectures often lack the capability to explicitly model tactical roles, leading to ambiguous task allocation in highly dynamic environments. To address these challenges, we propose Hierarchical Dynamic Role-Graph Multi-Agent Proximal Policy Optimization (DRG-MAPPO), a novel MARL framework that integrates graph-based relational modeling with dynamic role assignment. Specifically, DRG-MAPPO constructs a graph-based representation of battlefield interactions and leverages graph attention mechanisms to extract critical relational features among allies, enemies, and threats. Subsequently, a high-level policy employs a dynamic role assignment mechanism to determine tactical responsibilities (e.g., leader'' andsupporter''). Conditioned on these roles and encoded graph-relational features, a low-level policy executes discrete maneuver actions, facilitating the joint optimization of tactical strategy and collaborative execution. Furthermore, a target-priority auxiliary task is designed to foster the emergence of behaviors such as focus-fire. Experimental results demonstrate that DRG-MAPPO achieves a state-of-the-art win rate of 87%, suggesting that our framework effectively balances relational modeling, interpretability, and optimization stability for cooperative air combat.