RoboJudge: Multimodal Language Models as Judges for Embodied Video Generation

📄 arXiv: 2610.27782v1 📥 PDF

作者: Siyuan Ma, Yu Qi, Cheng Ye, Ruotian Peng, Xinyi Xu, Zitiantao Lin, Zedong Cai, Yijian Huang, Weiyang Liu, Siyuan Wang, Yandong Wen

分类: cs.CV, cs.RO

发布日期: 2026-10-20

备注: Project page: https://spherelab.ai/RoboJudge

💡 一句话要点

提出RoboJudge以评估视频生成模型的物理有效性与任务一致性

🎯 匹配领域: 支柱一：机器人控制 (Robot Control) 支柱二：RL算法与架构 (RL & Architecture) 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 视频生成 多模态语言模型 物理遵循性 任务一致性 评估基准 强化学习 机器人技术

📋 核心要点

  1. 现有的多模态语言模型在评估视频生成的物理有效性和任务一致性方面存在不足，特别是在与人类判断的一致性上。
  2. 本文提出RoboJudge-Bench基准，系统评估MLLMs在具身视频生成中的表现，并构建RoboJudge-Train数据集以提升模型性能。
  3. RoboJudge-9B在RoboJudge-Bench上取得了最高的一致性，Pearson相关系数达到0.719，相较于最佳基线提升了0.089。

📝 摘要（中文）

视频生成模型在构建具身世界模型方面具有潜力，但其生成的结果必须在物理上有效且符合任务指令。多模态大型语言模型（MLLMs）越来越多地被用于判断这些属性，但它们与人类对具身视频的判断一致性尚未得到系统评估。为此，本文引入了RoboJudge-Bench，一个用于评估MLLMs作为具身视频生成评判者的基准，涵盖800个由人类标注的视频。实验结果显示，现有评判者与人类评分之间存在不均衡的对齐，特别是在物理遵循性方面。基于这些发现，构建了RoboJudge-Train，包含12000个视频和25000个经过人类验证的评分，最终训练出RoboJudge-9B模型，其在评估中表现优异。

🔬 方法详解

问题定义：本文旨在解决多模态语言模型在评估具身视频生成时与人类判断不一致的问题，尤其是在物理遵循性和任务一致性方面的不足。

核心思路：通过构建RoboJudge-Bench基准和RoboJudge-Train数据集，系统性地评估和训练多模态语言模型，使其能够更准确地判断视频生成的质量。

技术框架：整体流程包括数据集构建、模型训练和评估三个主要阶段。RoboJudge-Bench用于评估模型的表现，而RoboJudge-Train则为模型提供了丰富的训练数据。

关键创新：最重要的创新在于构建了专门针对具身视频生成的评估基准和训练数据集，使得模型能够在物理遵循性和任务一致性上进行有效学习。

关键设计：在训练过程中，使用了监督学习和强化学习相结合的方法，模型参数经过精细调优，以确保在评估中达到最佳性能。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

RoboJudge-9B在RoboJudge-Bench上的表现优异，Pearson相关系数达到0.719，相较于最佳基线Gemini-3.7-Flash提升了0.089。此外，该模型在外部基准测试VideoPhy2和Physion-Eval上也表现出显著的性能提升。

🎯 应用场景

该研究的潜在应用领域包括机器人技术、虚拟现实和自动化视频生成等。通过提高视频生成模型的评估准确性，可以在多个行业中实现更高效的自动化和智能化应用，推动具身智能的发展。

📄 摘要（原文）

Video generation models are promising foundations for embodied world models, but only if their generated rollouts are physically valid and faithful to task instructions. Multimodal large language models (MLLMs) are increasingly used to judge these properties, yet their agreement with human judgments on embodied videos has not been systematically evaluated. To address this gap, we introduce RoboJudge-Bench, a benchmark for evaluating MLLMs as judges of embodied video generation. The benchmark measures two fundamental dimensions, Physical Adherence and Instruction Alignment, across 800 human-annotated videos from 8 source corpora and 10 video generators, spanning grippers, human hands, and dexterous hands. Experiments reveal uneven alignment between existing judges and human ratings: general-purpose MLLMs show lower agreement on physical adherence, while specialized video judges designed for general-domain physical reasoning do not consistently transfer to embodied scenarios. Motivated by these findings, we construct RoboJudge-Train, comprising 12K videos annotated with 25K human-verified scores and model-assisted rationales for Physical Adherence and Instruction Alignment. We fine-tune a 9B MLLM on this dataset with supervised learning followed by reinforcement learning, yielding RoboJudge-9B. Among the model judges evaluated on RoboJudge-Bench, RoboJudge-9B achieves the highest overall agreement, reaching a Pearson correlation of 0.719 with a statistically significant 0.089 gain over Gemini-3.7-Flash, the best-performing proprietary baseline in our evaluation. Training on RoboJudge-Train also yields gains on external benchmarks, including VideoPhy2 and Physion-Eval. We further use RoboJudge-9B as a downstream evaluator to benchmark video generation models according to Physical Adherence and Instruction Alignment.