Events, Not Pixels: Where World Models Get the Return Wrong
作者: Haoyue Liu, Xiaoying Tang
分类: cs.AI
发布日期: 2026-10-21
💡 一句话要点
提出事件过渡模型以解决世界模型回报错误问题
🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture)
关键词: 世界模型 事件过渡 回报评估 规划算法 强化学习 Atari游戏 统计分析
📋 核心要点
- 现有的世界模型在规划任务中，虽然能生成视觉上真实的结果，但常常导致错误的回报评估。
- 本文提出了一种新的过渡级诊断方法，通过用真实过渡替换想象过渡来识别回报错误的来源。
- 实验结果显示，事件过渡的恢复能显著减少回报错误，且在44个Atari-100k模型对中，事件补丁在39个任务中表现优于随机补丁。
📝 摘要（中文）
世界模型在规划中越来越多地被使用，规划者通过模型想象的回报来对候选动作进行排名。然而，这些模型主要通过渲染帧的质量来评估，尽管视觉上真实的展开仍可能分配错误的回报。为识别这种不匹配的来源，本文引入了一种过渡级诊断方法，通过用真实过渡替换想象过渡，并测量与预算匹配的随机控制相比回报错误的减少。分析结果表明，事件过渡的恢复能显著减少回报错误，而其他过渡的恢复效果则不如随机选择。
🔬 方法详解
问题定义：本文旨在解决世界模型在规划中因想象过渡导致的回报错误问题。现有方法主要依赖视觉质量评估，忽视了回报的准确性。
核心思路：提出通过用真实过渡替换想象过渡的方式，来诊断和修复回报错误，重点关注事件过渡的影响。
技术框架：整体流程包括模型生成想象过渡、用真实过渡进行替换、测量回报错误的减少，并通过统计分析验证事件补丁的有效性。
关键创新：最重要的创新在于识别并利用事件过渡的特性，显著提高了回报评估的准确性，与传统方法相比，提供了更为有效的修复策略。
关键设计：在实验中，使用了中位数像素误差比率0.85的标准来评估过渡的质量，并通过统计检验确保事件补丁在多个任务中的优越性。
🖼️ 关键图片
📊 实验亮点
实验结果显示，在44个Atari-100k模型对中，事件补丁在39个任务中显著优于随机补丁，回报提升中位数达到44个百分点，且统计显著性达到p=1.4 x 10^{-7}，验证了事件过渡的重要性。
🎯 应用场景
该研究的潜在应用领域包括机器人规划、游戏AI和自动驾驶等需要高效决策的场景。通过提高回报评估的准确性，能够显著提升智能体在复杂环境中的表现，具有重要的实际价值和未来影响。
📄 摘要（原文）
World models are increasingly used for planning, where a planner ranks candidate actions by returns imagined by the model. However, these models are judged mainly by how well they render frames, even though a visually faithful rollout can still assign the wrong return. To identify where this mismatch arises, we introduce a transition-level diagnostic that replaces imagined transitions with real ones and measures the reduction in return error against a budget-matched random control. Our analysis yields two findings. (1) Event transitions, the few steps where a reward is earned or an episode ends, are rendered as well as other steps (median pixel-error ratio 0.85), yet restoring them removes a median of 70% of the return error. In contrast, restoring transitions with the largest MSE, SSIM or LPIPS error removes less than a random choice. Across the 44 Atari-100k model--collection pairs that pass our reliability checks, including three released families, event patches outperform random patches in 39 (sign test p=1.4 x 10^{-7}), with a median gain of +44 percentage points. (2) The remaining error is structured: a complementary patch separates errors at true events from those elsewhere, while an eight-step timing tolerance recovers few events missed by our deterministic models. This structure also guides repair: a rule recorded before four repair outcomes were known predicts when a diffusion model repairs these failures. Event patches improve plan ranking, and the same concentration appears on control tasks with rare reward events.