ReViV: Reconstructing the Viewer and the View in 4D from Monocular Egocentric Video
作者: Xiaozhong Lyu, Gen Li, Zhiyin Qian, Xucong Zhang, Marc Pollefeys, Siyu Tang
分类: cs.CV, cs.AI
发布日期: 2026-07-20
备注: Accepted to ECCV 2026. The first two authors contributed equally, and their author order is interchangeable
🔗 代码/项目: PROJECT_PAGE
💡 一句话要点
提出ReViV框架以解决单目自我中心视频的4D重建问题
🎯 匹配领域: 支柱三：空间感知与语义 (Perception & Semantics) 支柱六：视频提取与匹配 (Video Extraction) 支柱九：具身大模型 (Embodied Foundation Models)
关键词: 自我中心视频 4D重建 多模态信号 深度学习 计算机视觉
📋 核心要点
- 现有方法依赖于辅助输入，且将场景感知与自我运动建模视为独立问题，导致重建效果不佳。
- ReViV框架通过学习多模态信号的联合概率分布，能够同时提取观察者和视图的动态信息。
- 在多个基准测试中，ReViV在自我中心深度估计、相机跟踪等任务上表现出色，具有竞争力的准确性和效率。
📝 摘要（中文）
自我中心设备（如可穿戴前置摄像头）为捕捉人类观察者与周围环境的连续互动提供了独特视角。因此，开发一个能够高效重建这种4D表示的整体多模态模型显得尤为重要。然而，现有方法通常依赖于辅助输入，如预计算的相机轨迹，将场景感知与人类自我运动建模视为独立问题，且推理速度较慢。为了解决这些局限性，本文提出了ReViV，这是第一个统一框架，能够从单个单目RGB视频中提取观察者和视图动态。通过Masked Generative Egocentric Transformer，ReViV在单一前馈架构中同时重建时间一致的4D表示，具有快速推理速度。实验结果表明，ReViV在多个基准测试中实现了最先进的准确性和效率。
🔬 方法详解
问题定义：本文旨在解决从单目自我中心视频中重建4D表示的问题。现有方法通常依赖于额外的输入数据，如相机轨迹，且将场景感知与自我运动建模分开处理，导致重建效果不理想且推理速度较慢。
核心思路：ReViV框架的核心思想是通过学习多模态信号的联合概率分布，整合RGB视频、相机轨迹、视线方向、全身运动、手部运动和深度信息，从而实现观察者与视图动态的同时重建。
技术框架：ReViV采用Masked Generative Egocentric Transformer作为核心组件，整体架构为单一前馈网络，能够在快速推理的同时实现时间一致的4D重建。
关键创新：ReViV的主要创新在于其统一的框架设计，首次将观察者与视图的动态重建整合在一个模型中，克服了传统方法的局限性。
关键设计：在网络结构上，ReViV采用了Masked Generative Transformer，设计了适合自我中心视频的损失函数，确保了多模态信号的有效融合与重建。具体参数设置和网络细节在论文中有详细描述。
🖼️ 关键图片
📊 实验亮点
在HoloAssist、HOT3D、ARCTIC等多个基准测试中，ReViV在整体自我中心身体、手部和视线重建、相机跟踪等任务上达到了最先进的准确性和效率，且在自我中心深度估计方面表现出色，未依赖重型任务特定先验，展现了显著的性能提升。
🎯 应用场景
该研究的潜在应用领域包括虚拟现实、增强现实和人机交互等场景，能够为用户提供更自然的交互体验。通过精准的4D重建，ReViV可以在各种应用中提升用户的沉浸感和交互质量，具有广泛的实际价值和未来影响。
📄 摘要（原文）
Egocentric devices, such as wearable front-facing cameras, provide a unique perspective for capturing the continuous interaction between a human viewer and the surrounding environment. A holistic and efficient multimodal model capable of reconstructing this 4D representation is therefore highly desirable. However, existing approaches often rely on auxiliary inputs such as pre-computed camera trajectories, treat scene perception and human ego-motion modeling as separate problems despite their strong interdependency, and suffer from slow inference time. To address these limitations, we present ReViV, the first unified framework for holistic egocentric 4D reconstruction that extracts both viewer and view dynamics from a single monocular RGB video. We formulate the task as learning the full joint probability distribution over multimodal signals, including RGB video, camera trajectory, gaze direction, full-body motion, hand motion, and depth. Powered by a Masked Generative Egocentric Transformer, ReViV operates within a single feed-forward architecture to simultaneously reconstruct the temporally consistent 4D reconstruction across the viewer and the view with fast inference speed. Extensive experiments on diverse benchmarks, including HoloAssist, HOT3D, ARCTIC, Aria Digital Twin, and TACO, demonstrate that ReViV achieves state-of-the-art accuracy and efficiency across holistic ego-body, hand, and gaze reconstruction, camera tracking, while maintaining highly competitive egocentric depth estimation without relying on heavy task-specific priors. Code and models are fully open-sourced: https://reviv4d.github.io/.