MARVEL: Multi-Agent Reinforcement-Learning for Large-Scale Variable Speed Limits

📄 arXiv: 2310.12359v2 📥 PDF

作者: Yuhang Zhang, Marcos Quinones-Grueiro, Zhiyao Zhang, Yanbing Wang, William Barbour, Gautam Biswas, Daniel Work

分类: cs.MA, cs.LG

发布日期: 2023-10-18 (更新: 2024-03-18)

💡 一句话要点

提出MARVEL框架以解决高速公路可变限速控制问题

🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture)

关键词: 可变限速 多智能体强化学习 交通管理 智能交通 深度学习

📋 核心要点

  1. 现有的可变限速控制算法大多基于规则，缺乏在复杂交通场景下的适应性和普适性。
  2. 本文提出MARVEL框架，利用多智能体强化学习方法，基于实时交通信息进行动态限速控制。
  3. 实验结果显示，MARVEL方法在交通安全和流动性方面分别提升了63.4%和58.6%，显著优于现有算法。

📝 摘要（中文）

可变限速（VSL）控制作为一种有前景的高速公路交通管理策略，通过根据实时交通状况动态调整速度限制来提高交通安全。现有的VSL控制算法大多基于规则，缺乏在复杂交通场景下的普适性。本文提出了MARVEL（大规模可变限速的多智能体强化学习），该框架利用真实世界可观察的传感信息作为状态输入，通过包含适应性、安全性和流动性的奖励结构进行学习，从而实现多智能体协调。实验结果表明，MARVEL方法在交通安全和流动性方面分别提高了63.4%和58.6%。

🔬 方法详解

问题定义：本文旨在解决现有可变限速控制算法在复杂交通场景下的适应性不足的问题。现有方法通常基于固定规则，无法有效应对动态变化的交通状况。

核心思路：MARVEL框架通过多智能体强化学习，利用实时交通信息作为输入，设计了适应性强的奖励结构，从而实现智能体之间的协调与合作。

技术框架：MARVEL的整体架构包括状态输入模块、奖励结构设计、智能体训练和策略应用等主要阶段。状态输入模块负责收集实时交通数据，奖励结构则根据安全性和流动性进行设计。

关键创新：MARVEL的主要创新在于引入了多智能体协调机制和参数共享策略，使得框架能够扩展到大规模的交通网络中，显著提高了控制效果。

关键设计：在参数设置上，所有VSL智能体共享相同的策略参数，损失函数设计考虑了适应性、安全性和流动性等多个因素，网络结构采用深度强化学习模型以增强学习能力。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

实验结果显示，MARVEL方法在交通安全方面提高了63.4%，在流动性方面提升了58.6%，相较于未控制场景和现有的实践算法均表现出显著的优势。这些结果表明MARVEL在实际应用中的有效性和可行性。

🎯 应用场景

MARVEL框架具有广泛的应用潜力，尤其在高速公路交通管理、智能交通系统和自动驾驶领域。通过动态调整限速，能够有效提高交通安全和流动性，减少交通事故和拥堵，具有重要的社会价值和经济效益。未来，MARVEL的理念和方法可以推广到其他交通管理场景中，进一步提升智能交通系统的效率。

📄 摘要（原文）

Variable Speed Limit (VSL) control acts as a promising highway traffic management strategy with worldwide deployment, which can enhance traffic safety by dynamically adjusting speed limits according to real-time traffic conditions. Most of the deployed VSL control algorithms so far are rule-based, lacking generalizability under varying and complex traffic scenarios. In this work, we propose MARVEL (Multi-Agent Reinforcement-learning for large-scale Variable spEed Limits), a novel framework for large-scale VSL control on highway corridors with real-world deployment settings. MARVEL utilizes only sensing information observable in the real world as state input and learns through a reward structure that incorporates adaptability to traffic conditions, safety, and mobility, thereby enabling multi-agent coordination. With parameter sharing among all VSL agents, the proposed framework scales to cover corridors with many agents. The policies are trained in a microscopic traffic simulation environment, focusing on a short freeway stretch with 8 VSL agents spanning 7 miles. For testing, these policies are applied to a more extensive network with 34 VSL agents spanning 17 miles of I-24 near Nashville, TN, USA. MARVEL-based method improves traffic safety by 63.4% compared to the no control scenario and enhances traffic mobility by 58.6% compared to a state-of-the-practice algorithm that has been deployed on I-24. Besides, we conduct an explainability analysis to examine the decision-making process of the agents and explore the learned policy under different traffic conditions. Finally, we test the response of the policy learned from the simulation-based experiments with real-world data collected from I-24 and illustrate its deployment capability.