Cooperative Multi-Agent Reinforcement Learning for Market Making on Prediction-Market Order Books

📄 arXiv: 2610.26789v1 📥 PDF

作者: Skander Lejmi

分类: stat.ML, cs.LG, cs.MA, q-fin.TR

发布日期: 2026-10-19

备注: 50 pages, 8 figures, 7 tables. Extended version of a paper accepted at ICAIF 2026 Workshop on AI-Driven Market Microstructure. MSc dissertation, London School of Economics and Political Science (2026)

💡 一句话要点

提出合作多智能体强化学习以解决预测市场做市问题

🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture)

关键词: 市场做市 多智能体强化学习 预测市场 流动性管理 风险控制 深度学习 金融科技

📋 核心要点

  1. 现有市场做市策略在真实数据上训练时可能因不利选择而停止报价，导致流动性不足。
  2. 通过重建订单数据并引入合作多智能体结构，论文提出了改进的市场做市商训练方法。
  3. 实验表明，重新设计的市场做市商在报价频率和持有损失方面优于基线模型，尽管各因素的独立贡献尚未明确。

📝 摘要（中文）

市场做市商通过报价买卖资产提供流动性，赚取价差并管理库存风险。本文研究了在预测市场（特别是Polymarket）上训练的市场做市商，面对稀薄的订单簿带来的挑战。研究中识别了三个主要挑战：不利选择可能导致基于真实数据训练的策略停止报价；报价的后果可能在多步之后才会显现；预测市场的L2数据无法识别单个订单。为解决这些问题，本文重建了Polymarket的订单级LOBSTER格式数据，并在417个真实市场上训练了智能体。通过丰富市场做市商的观察空间、重新设计其动作和库存控制，提出了基于递归独立近端策略优化（IPPO）和RNN策略的合作多智能体分解方法。实验结果显示，重新设计的单智能体市场做市商报价频率更高，持有损失更小。

🔬 方法详解

问题定义：本文旨在解决在稀薄订单簿的预测市场中，市场做市商因不利选择而停止报价的问题。现有方法在处理延迟反馈和缺乏订单识别方面存在不足。

核心思路：通过重建Polymarket的订单数据，论文提出了合作多智能体的市场做市商训练框架，分为报价者和风险管理者，以应对复杂的市场动态。

技术框架：整体架构包括数据重建、观察空间扩展、动作设计和风险控制。智能体使用递归独立近端策略优化（IPPO）进行训练，结合RNN策略处理时间序列数据。

关键创新：论文的主要创新在于引入合作多智能体结构和奖励设计，显著提升了市场做市商的表现，尤其是在报价频率和损失控制方面。

关键设计：在参数设置上，采用了奖励混合和后填充标记的设计，以解决智能体间的信用分配问题。网络结构上，使用了RNN以捕捉时间序列特征，增强了模型的适应性。

🖼️ 关键图片

img_0
img_1
img_2

📊 实验亮点

实验结果显示，重新设计的单智能体市场做市商在报价频率上提高了显著性，并且持有损失相较于基线模型减少。尽管未能在周期性评估中实现正的平均投资组合价值，但相对改进的表现为后续研究提供了基础。

🎯 应用场景

该研究为市场做市商在预测市场中的应用提供了新的思路，能够有效提升流动性管理和风险控制能力。未来，该方法可扩展至其他金融市场或多市场环境，具有广泛的实际应用价值。

📄 摘要（原文）

A market maker provides liquidity by quoting prices to buy and sell an asset, earning the spread while managing inventory risk. We study market makers trained with multi-agent reinforcement learning on prediction markets, specifically Polymarket, whose thin order books pose distinct challenges. Three challenges underpin this research: adverse selection may cause a policy trained on real data to stop quoting; the consequences of a quote can arrive many steps after the action; and prediction-market L2 data does not identify individual orders needed for simulation. We address these challenges by reconstructing order-level LOBSTER-format data from Polymarket and training agents in JaxMARL-HFT on 417 real markets. We enrich the baseline market maker's observation space with the book and recent flow, redesign its actions and inventory controls, and propose a cooperative multi-agent decomposition into a quoter and a risk manager trained using recurrent Independent Proximal Policy Optimisation (IPPO) with RNN policies. We investigate reward shaping, particularly reward blending and post-fill markout, to mitigate credit-assignment difficulties between the agents. Our redesigned single-agent market maker quotes more frequently and incurs smaller held-out losses than the retrained JaxMARL-HFT baseline, although the contributions of jointly changed observations, actions and risk constraints are not separately identified. Decomposition alone performs poorly; careful reward design substantially improves the desk relative to the unshaped split and yields our best result. Models are evaluated on realised end-of-episode portfolio value on held-out markets. None establishes positive average portfolio value in periodic held-out evaluation, and each configuration was trained with one seed. The environment, data and protocol lay a foundation for makers that quote many linked markets at once.