DrM: Mastering Visual Reinforcement Learning through Dormant Ratio Minimization

📄 arXiv: 2310.19668v2 📥 PDF

作者: Guowei Xu, Ruijie Zheng, Yongyuan Liang, Xiyao Wang, Zhecheng Yuan, Tianying Ji, Yu Luo, Xiaoyu Liu, Jiaxin Yuan, Pu Hua, Shuzhen Li, Yanjie Ze, Hal Daumé, Furong Huang, Huazhe Xu

分类: cs.LG, cs.CV

发布日期: 2023-10-30 (更新: 2024-02-14)

备注: Accepted at The Twelfth International Conference on Learning Representations (ICLR 2024)

💡 一句话要点

提出DrM以解决视觉强化学习中的无效探索问题

🎯 匹配领域: 支柱一：机器人控制 (Robot Control) 支柱二：RL算法与架构 (RL & Architecture)

关键词: 视觉强化学习 休眠比率 样本效率 探索与利用 深度学习 机器人控制 自动驾驶

📋 核心要点

  1. 现有视觉强化学习方法在样本效率和鲁棒性方面存在明显不足，尤其是在早期训练阶段代理的无活动状态影响了探索能力。
  2. 论文提出了一种新方法DrM，通过最小化代理的休眠比率来改善探索与利用的平衡，进而提升学习效率。
  3. 实验结果显示，DrM在DeepMind Control Suite等多个环境中显著提高了样本效率和渐近性能，且在多个任务中表现出色。

📝 摘要（中文）

视觉强化学习（RL）在连续控制任务中展现出潜力，但现有算法在样本效率、渐近性能和对随机种子的鲁棒性等方面仍显不足。本文识别出现有视觉RL方法的一个主要缺陷，即代理在早期训练中常表现出持续的无活动状态，限制了有效探索的能力。我们引入了“休眠比率”作为量化无活动的指标，并提出DrM方法，通过主动最小化休眠比率来指导代理的探索与利用权衡。实验表明，DrM在样本效率和渐近性能上显著提升，且在多个基准环境中表现优异。

🔬 方法详解

问题定义：本文旨在解决视觉强化学习中代理在早期训练阶段的无活动状态问题，现有方法未能有效促进代理的探索能力。

核心思路：论文提出通过量化代理的无活动状态（休眠比率）来指导其探索行为，主动最小化该比率以提升学习效率。

技术框架：DrM方法包含三个核心机制，分别用于优化探索与利用的权衡，具体流程包括监测休眠比率、调整策略网络和反馈学习信号。

关键创新：DrM是首个通过休眠比率作为独立指标来引导代理活动的模型无关算法，显著区别于传统方法。

关键设计：在实现中，休眠比率被用作损失函数的一部分，结合策略网络的神经活动进行动态调整，以确保代理在训练过程中的持续探索。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

实验结果表明，DrM在76个随机种子下的测试中，显著提高了样本效率和渐近性能，成功解决了DeepMind Control Suite中的多个任务，包括Dog和Manipulator领域的挑战，且在Adroit的手部操作任务中无需演示。

🎯 应用场景

该研究的潜在应用领域包括机器人控制、自动驾驶、游戏AI等，能够有效提升智能体在复杂环境中的学习能力和适应性。未来，DrM方法可能推动更高效的视觉强化学习算法的发展，促进智能体在实际应用中的表现。

📄 摘要（原文）

Visual reinforcement learning (RL) has shown promise in continuous control tasks. Despite its progress, current algorithms are still unsatisfactory in virtually every aspect of the performance such as sample efficiency, asymptotic performance, and their robustness to the choice of random seeds. In this paper, we identify a major shortcoming in existing visual RL methods that is the agents often exhibit sustained inactivity during early training, thereby limiting their ability to explore effectively. Expanding upon this crucial observation, we additionally unveil a significant correlation between the agents' inclination towards motorically inactive exploration and the absence of neuronal activity within their policy networks. To quantify this inactivity, we adopt dormant ratio as a metric to measure inactivity in the RL agent's network. Empirically, we also recognize that the dormant ratio can act as a standalone indicator of an agent's activity level, regardless of the received reward signals. Leveraging the aforementioned insights, we introduce DrM, a method that uses three core mechanisms to guide agents' exploration-exploitation trade-offs by actively minimizing the dormant ratio. Experiments demonstrate that DrM achieves significant improvements in sample efficiency and asymptotic performance with no broken seeds (76 seeds in total) across three continuous control benchmark environments, including DeepMind Control Suite, MetaWorld, and Adroit. Most importantly, DrM is the first model-free algorithm that consistently solves tasks in both the Dog and Manipulator domains from the DeepMind Control Suite as well as three dexterous hand manipulation tasks without demonstrations in Adroit, all based on pixel observations.