Deep Reinforcement Learning with Buffered Quantile Objectives

📄 arXiv: 2609.21327v1 📥 PDF

作者: Mohammad Alipour-vaezi, Sajad Khodadadian

分类: cs.LG, cs.AI, math.OC

发布日期: 2026-09-18

💡 一句话要点

提出深度强化学习框架以解决量化目标优化问题

🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture)

关键词: 深度强化学习 量化目标 风险敏感决策 无模型学习 神经网络 探索策略 缓冲量化

📋 核心要点

  1. 现有基于量化的强化学习方法在点量化目标下学习面临量化值变化剧烈和计算复杂度高的挑战。
  2. 论文提出Deep-BQRL框架，通过无模型学习直接从过渡中获取条件回报量化，构建缓冲动作评分以优化决策。
  3. 实验结果表明，Deep-BQRL在资产出售问题中相较于PPO和TRPO实现了更小的平均累积点量化政策差距，且决策可解释性强。

📝 摘要（中文）

基于量化的强化学习为风险敏感的决策提供了一种可解释的方法，通过优化累积回报分布的特定量化值来实现。然而，在点量化目标下学习面临挑战：量化值在回报分布的小扰动下可能会急剧变化，且精确的量化敏感规划需要计算密集型的分布优化。低缓冲量化通过平均目标水平下的相邻量化值来缓解这一困难，提供了更平滑的替代方案，同时保留了基础的点量化目标。现有基于此原则的方法仍然是基于模型的，并依赖于显式的回报规划，限制了其在小型表格问题之外的适用性。我们开发了Deep-BQRL，一个无模型的分布式强化学习框架，将缓冲量化学习扩展到神经函数逼近。该方法直接从采样的过渡中学习条件回报量化值，构建缓冲动作评分，并利用集成不一致性来指导探索。

🔬 方法详解

问题定义：本论文旨在解决基于量化的强化学习中点量化目标学习的挑战，尤其是量化值在小扰动下的剧烈变化和计算复杂度高的问题。

核心思路：提出Deep-BQRL框架，通过无模型学习直接从采样过渡中获取条件回报量化，构建缓冲动作评分以实现更平滑的决策过程。

技术框架：Deep-BQRL的整体架构包括条件回报量化的学习模块、缓冲动作评分的构建模块和探索引导模块，利用集成不一致性来增强探索效率。

关键创新：最重要的创新在于将缓冲量化学习扩展到神经网络函数逼近，突破了现有方法对模型的依赖，提升了在复杂问题上的适用性。

关键设计：在参数设置上，采用了集成学习的方法来处理不一致性，损失函数设计上关注于缓冲量化的优化，网络结构则通过增强输入表示来响应轨迹信息。

🖼️ 关键图片

img_0
img_1
img_2

📊 实验亮点

实验结果显示，Deep-BQRL在资产出售问题中实现了比PPO和TRPO更小的平均累积点量化政策差距，且UCB-BQRL保持了最小的差距，表明该方法在风险敏感决策中的有效性和可解释性。

🎯 应用场景

该研究的潜在应用领域包括金融决策、机器人控制和智能交通等场景，能够为风险敏感的决策提供更为有效和可解释的解决方案。未来，Deep-BQRL框架有望在更复杂的动态环境中发挥重要作用，推动强化学习的实际应用。

📄 摘要（原文）

Quantile-based reinforcement learning provides an interpretable approach to risk-sensitive decision-making by optimizing a prescribed quantile of the cumulative-return distribution. Despite this appeal, learning under a point quantile objective is challenging: quantiles can change abruptly under small perturbations of the return distribution, and exact quantile-sensitive planning requires computationally demanding distributional optimization. Lower-buffered quantiles alleviate the former difficulty by averaging neighboring quantiles immediately below the target level, providing a smoother surrogate while preserving the underlying point-quantile objective. Existing methods based on this principle, however, remain model-based and rely on explicit return-law planning, limiting their applicability beyond small tabular problems. We develop Deep-BQRL, a model-free distributional reinforcement-learning framework that extends buffered-quantile learning to neural function approximation. The method learns conditional return quantiles directly from sampled transitions, constructs buffered action scores from the relevant region of the learned quantile function, and uses ensemble disagreement to guide exploration. An augmented input representation allows the learned policy to respond to trajectory information without explicitly reproducing the quantile-state recursion required by exact planning. Experiments on an asset-selling optimal-stopping problem and slippery FrozenLake compare Deep-BQRL with model-based UCB-BQRL and tabular PPO and TRPO implementations. In asset selling, Deep-BQRL attains smaller mean cumulative point-quantile policy gaps than PPO and TRPO at the reported target levels, while UCB-BQRL retains the smallest gaps. The learned stopping decisions also vary with the target quantile, providing an interpretable illustration of the method's risk-sensitive behavior.