MIMESIS: Learning User Simulators as Training Environments for Interactive Agents

📄 arXiv: 2610.09484v1 📥 PDF

作者: Hoang Phan, Dat Huynh, Andrey Zhmoginov, Qi Zeng, Wancen Mu, Yue Cao, Shengjie Bi, Yun He, Changdae Oh, Deren Lei

分类: cs.AI

发布日期: 2026-10-07

备注: Code is available at https://github.com/VietHoang1512/mimesis

💡 一句话要点

提出MIMESIS以解决交互式代理训练中的用户模拟问题

🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture)

关键词: 用户模拟 交互式代理 显式推理 强化学习 行为模式 自我蒸馏 人机交互

📋 核心要点

  1. 现有的代理训练框架依赖于现成的助手大语言模型，导致模型行为过于合作且缺乏多样性。
  2. MIMESIS通过训练用户模拟器，结合人类对话和显式推理监督，提供更真实的用户行为模拟。
  3. 在八个环境中，MIMESIS训练的代理在所有九个未见用户模拟器上表现优于使用GPT-5.5训练的代理。

📝 摘要（中文）

训练和评估交互式语言代理通常需要丰富的用户交互，但收集人类反馈既昂贵又难以扩展。模拟用户提供了一种可扩展的替代方案，但必须真实地反映用户行为并为代理提供有用的学习体验。本文提出MIMESIS，一个专门训练的用户模拟器，基于人类对话和显式推理监督，具备13种真实用户交互的行为模式。实验表明，MIMESIS在多个指标上超越了现有模型，展示了其在新用户模拟器上的更强泛化能力。

🔬 方法详解

问题定义：本文旨在解决交互式语言代理训练中用户模拟的不足，现有方法往往依赖于过于合作的助手模型，导致行为同质化，难以真实反映用户需求。

核心思路：MIMESIS通过训练一个专门的用户模拟器，利用人类对话数据和显式推理监督，生成多样化且真实的用户行为模式，以提升代理的学习效果。

技术框架：MIMESIS的整体架构包括用户模拟器的训练和代理的强化学习训练。用户模拟器首先生成用户行为，然后代理通过与该模拟器的多轮交互进行训练。

关键创新：MIMESIS的主要创新在于其训练方法，结合了显式推理监督和多样化的行为模式，使得模拟用户的行为更接近真实用户，显著提高了代理的学习效果。

关键设计：在训练过程中，MIMESIS使用了13种真实用户行为模式，并引入了Coached On-Policy Self-Distillation (CSD)方法，通过生成的推理轨迹和后续发言作为反馈，提供更细致的监督信息。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

MIMESIS模型在SOUL-Index上达到了65.7，超越了最强的前沿模型。与Claude-Opus-5相比，MIMESIS在行为保真度上提高了13.4分，图灵距离减少了3.6分，展示了其在用户模拟领域的显著优势。

🎯 应用场景

MIMESIS的研究成果在多个领域具有潜在应用价值，包括智能客服、虚拟助手和教育领域的交互式学习系统。通过提供更真实的用户交互模拟，能够显著提升代理的适应能力和用户满意度，推动人机交互技术的发展。

📄 摘要（原文）

Training and evaluating interactive language agents typically requires rich user interactions, yet collecting human feedback is expensive and difficult to scale. Simulated users offer a scalable alternative, but they must both resemble real user behavior and provide useful learning experiences for agents. In contrast, most agent-training frameworks rely on off-the-shelf assistant LLMs, whose helpfulness can make them overly cooperative, explicit, and behaviorally homogeneous compared with real users. We introduce MIMESIS, a purpose-built user simulator trained on human conversations with explicit reasoning supervision and 13 realistic behavioral patterns derived from real user interactions. Empirically, our 9B model achieves a SOUL-Index of 65.7, surpassing the strongest frontier model. Compared with Claude-Opus-5, the strongest baseline on RealUserSim and SimulatorArena, MIMESIS improves behavioral fidelity by 13.4 points and reduces Turing distance by 3.6 points, respectively. We then freeze the simulator and train an agent by interacting with the frozen simulator using multi-turn reinforcement learning. Across eight environments, training with MIMESIS yields better agent performance than training with GPT-5.5 under all nine unseen user simulators, demonstrating stronger generalization to new user simulators. Moreover, we propose Coached On-Policy Self-Distillation (CSD), which leverages simulator-generated private reasoning traces and subsequent utterances as feedback on how well the agent addresses user needs. A coach converts this information into concise coaching notes that describe how the agent can better anticipate user needs and adapt its behavior over the course of an interaction. CSD turns this feedback into dense, token-level supervision beyond sparse task rewards, yielding further gains across all nine evaluation user models.