Homomorphic Advantage Operator: Stabilizing Reinforcement Learning Under Fully Homomorphic Encryption Constraints

📄 arXiv: 2610.02074v1 📥 PDF

作者: Abid Mohamed Nadhir, Ahmad Al Hanbali, Beggas Mounir

分类: cs.AI, cs.CR

发布日期: 2026-10-01

💡 一句话要点

提出同态优势算子以解决强化学习中的同态加密问题

🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture) 支柱五：交互与反应 (Interaction & Reaction)

关键词: 同态加密 强化学习 贝尔曼漂移 隐私保护 深度学习 多项式近似 智能系统

📋 核心要点

  1. 现有的强化学习方法在完全同态加密下应用时，非线性操作的多项式近似会导致贝尔曼漂移，影响学习稳定性。
  2. 本文提出同态优势算子（HAO），通过零均值中心投影来稳定多项式近似，避免贝尔曼漂移现象。
  3. HAO在多个实验中表现出色，所有随机种子下均未出现边界突破，且在表格域中策略准确性提升了18个百分点。

📝 摘要（中文）

隐私保护的机器学习在云计算中面临重大挑战，完全同态加密（FHE）为安全计算提供了有效解决方案。然而，将FHE应用于强化学习（RL）时，非线性操作需被多项式近似替代，这导致了贝尔曼漂移现象的出现。本文提出同态优势算子（HAO），旨在防止FHE基础深度RL中的多项式近似发散。HAO通过将零均值中心投影直接应用于时序差分（TD）目标，消除了驱动贝尔曼漂移的均匀状态值基线，同时避免了额外的非线性乘法深度和昂贵的密文重加密。实验结果表明，HAO有效限制了网络预激活在安全多项式近似域内，显著提升了策略准确性。

🔬 方法详解

问题定义：本文解决在完全同态加密环境下强化学习中的多项式近似发散问题，现有方法因贝尔曼漂移而导致学习不稳定。

核心思路：HAO通过将零均值中心投影应用于时序差分目标，消除驱动贝尔曼漂移的均匀状态值基线，从而保持每个状态的动作排名。

技术框架：HAO框架包括三个主要模块：首先是对优势值的估计，其次是零均值中心投影的应用，最后是对多项式近似的稳定性保障。

关键创新：HAO的主要创新在于其线性投影方法，避免了额外的非线性乘法深度，与传统方法相比显著提高了稳定性和效率。

关键设计：HAO设计中采用了零均值中心投影，确保了在多项式近似域内的安全性，同时在实验中未使用额外的重加密步骤，降低了计算复杂度。

🖼️ 关键图片

fig_0
img_1
img_2

📊 实验亮点

实验结果显示，HAO框架在所有随机种子下均未出现边界突破，而常规正则化方法在5个种子中有3个出现了边界突破，未稳定基线则在83.8%的回合中出现了突破。此外，HAO在表格域中的策略准确性提升了18个百分点，显示出其显著的性能优势。

🎯 应用场景

该研究的潜在应用领域包括云计算中的智能系统，尤其是涉及敏感数据的强化学习任务。通过实现安全的计算，HAO可在医疗、金融等领域中保护用户隐私，同时提升智能系统的决策能力。未来，HAO的框架可能会扩展到其他需要隐私保护的机器学习任务中。

📄 摘要（原文）

Privacy-preserving machine learning presents significant deployment challenges on the cloud for intelligent systems with confidential data. Fully Homomorphic Encryption (FHE) offers a compelling solution for secure computation, preserving data confidentiality of cloud computations. However, applying FHE to reinforcement learning (RL) requires replacing non-linear operations with polynomial approximations, which diverge catastrophically due to a unique recursive error phenomenon known as the Bellman drift. This article introduces the Homomorphic Advantage Operator (HAO), a stabilization framework designed to prevent polynomial approximation divergence in FHE-based deep RL. HAO adapts the zero-mean centering projection from advantage-based value estimation directly to temporal-difference (TD) targets. This linear projection annihilates the uniform state-value baseline that drives the Bellman drift, maintaining per-state action rankings while requiring zero additional non-linear multiplicative depth and avoiding expensive ciphertext bootstrapping. The proposed HAO framework was evaluated using a three-tier experimental methodology, including a tabular Markov Decision Process (MDP), an encrypted CartPole environment using real CKKS cryptographic operations, and a 20-node logistics routing benchmark with dense continuous features. The results demonstrate that the proposed HAO strictly bounds network pre-activations within the safe polynomial approximation domain. The proposed HAO RL agents achieved 0% boundary breaches across all random seeds used, whereas regularization alone (L2 weight decay and gradient clipping) breached the bound on 3 of 5 seeds and the unstabilized baseline did so in 83.8% of episodes. Finally, HAO agents improve optimal policy accuracy by 18.0 percentage points in tabular domains and remain stable when DP-SGD-style Gaussian noise is added to the clipped gradients.