EgoP$^{2}$C: Internalizing Plan-Perceive-Compose for Spatial Reasoning in Egocentric Video
作者: Chaoyu Li, Arka Sadhu, Yuliang Li, Hyo Jin Kim, Yuning Chai, Pooyan Fazli
分类: cs.CV
发布日期: 2026-10-15
💡 一句话要点
提出EgoP$^{2}$C以解决自我中心视频中的空间推理问题
🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture) 支柱六：视频提取与匹配 (Video Extraction)
关键词: 自我中心视频 空间推理 视觉-语言模型 计划-感知-组合 蒸馏训练 多教师模型 推理效率 机器人导航
📋 核心要点
- 现有视觉-语言模型在处理自我中心视频的空间推理时，往往无法可靠地执行所需的推理步骤，导致结果不准确。
- EgoP$^{2}$C框架通过内化计划-感知-组合过程，利用双教师模型来指导推理过程，从而提高了模型的推理能力。
- 在三个空间推理基准测试中，EgoP$^{2}$C的学生模型在性能上平均提升了9.2和7.5个百分点，同时推理计算量减少了65.8-66.3%。
📝 摘要（中文）
回答来自自我中心视频的复杂空间推理问题需要确定所需证据、获取目标观察并通过多个推理步骤进行组合。现有的视觉-语言模型（VLMs）在这一过程中表现不佳。为此，本文提出EgoP$^{2}$C框架，将计划-感知-组合的过程内化于单一VLM中。通过训练专门的计划和感知教师，并在其监督下微调相应的专家，最终在推理时实现自适应工作流。实验表明，EgoP$^{2}$C在三个空间推理基准上显著提升了性能，同时减少了推理计算量。
🔬 方法详解
问题定义：本文旨在解决自我中心视频中的复杂空间推理问题，现有方法在推理过程中依赖外部工具，导致不准确的观察可能导致错误答案。
核心思路：EgoP$^{2}$C框架通过内化计划-感知-组合的过程，利用双教师模型（计划教师和感知教师）来指导推理，避免了对外部工具的依赖。
技术框架：整体架构包括两个主要模块：计划教师和感知教师。计划教师负责指导证据收集的问题和何时停止，而感知教师则监督视频基础的答案。学生模型在其监督下进行微调，并在推理时执行自适应工作流。
关键创新：最重要的创新在于将计划-感知-组合过程内化于单一VLM中，避免了对外部工具的依赖，并通过双教师模型提高了推理的准确性和效率。
关键设计：采用双LoRA适配器训练计划和感知教师，并在混合专家模型中微调相应的专家。使用多教师在线蒸馏（MOPD）对生成的轨迹进行训练，分配不同的轨迹段给不同的教师进行监督。
🖼️ 关键图片
📊 实验亮点
实验结果显示，EgoP$^{2}$C在三个空间推理基准上，基于Qwen3-VL-8B和SenseNova-SI-1.3-8B的学生模型分别平均提升了9.2和7.5个百分点。同时，推理计算量减少了65.8-66.3%，实现了1.29-1.49倍的速度提升。
🎯 应用场景
该研究具有广泛的应用潜力，尤其是在机器人导航、自动驾驶、虚拟现实和增强现实等领域。通过提高自我中心视频的空间推理能力，EgoP$^{2}$C能够支持更复杂的交互和决策过程，推动智能系统的进一步发展。
📄 摘要（原文）
Answering complex spatial reasoning questions from egocentric video requires determining what evidence is needed, acquiring targeted observations, and combining them through multiple reasoning steps. Existing vision-language models (VLMs) struggle to perform this process reliably. Agentic systems address this challenge by querying external vision and geometry tools, but training their reasoning leaves these tools unchanged, so inaccurate observations can still lead to incorrect answers. We introduce EgoP$^{2}$C, a framework that internalizes a plan-perceive-compose process within a single VLM. To this end, we first train specialized Planner and Perception teachers using separate adapters (Dual-LoRA), then fine-tune the corresponding experts in a mixture-of-experts (MoE) student under their supervision. We subsequently train the student on trajectories generated by its current policy using a segment-level extension of multi-teacher on-policy distillation (MOPD), assigning different trajectory segments to different supervisors. The Planner teacher supervises which evidence-seeking questions to ask, when to stop gathering evidence, and how to compose the final answer. The Perception teacher supervises the video-grounded answers to those questions. At inference, the student executes this adaptive workflow end-to-end, without external spatial tools. Across three spatial reasoning benchmarks, EgoP$^{2}$C students built from Qwen3-VL-8B and SenseNova-SI-1.3-8B outperform their base models by an average of 9.2 and 7.5 percentage points, respectively. Across both backbones, EgoP$^{2}$C reduces inference FLOPs by 65.8-66.3% and achieves $1.29$-$1.49\times$ speedups over their shared Dual-LoRA teacher.