PercepCap: Video Captioner with Structured Spatio-Temporal Perception

📄 arXiv: 2607.20389v1 📥 PDF

作者: Yifan Xu, Zihao Wang, Zhixiao Wang, Jiaming Zhang, Yichun Yang, Desen Meng, Yuanxing Zhang, Pengfei Wan, Limin Wang

分类: cs.CV

发布日期: 2026-07-22

💡 一句话要点

提出PercepCap以解决视频字幕生成中的时空感知问题

🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture) 支柱八：物理动画 (Physics-based Animation) 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 视频字幕生成 时空感知 多模态学习 感知-描述生成链 强化学习 可解释性 数据构建

📋 核心要点

  1. 现有视频字幕生成方法缺乏对时空感知的明确展示，导致潜在错误难以识别。
  2. PercepCap通过感知-描述生成链，首先生成时空感知轨迹，再基于此生成最终字幕。
  3. 实验结果表明，PercepCap在字幕质量上显著优于Qwen3-VL基线，展示了其有效性。

📝 摘要（中文）

视频字幕生成需要对视频进行细粒度的时空理解，包括对象的空间位置感知和事件发生的时间感知。现有的多模态大语言模型通常直接从视频输入生成字幕，而未能明确展示描述背后的感知证据。因此，时空感知中的错误仅在最终字幕中显现，难以直接识别潜在的感知错误。为了解决这些问题，本文提出了PercepCap，一个感知意识的视频字幕生成框架，该框架在生成最终字幕之前明确感知证据。PercepCap遵循感知-描述生成链，首先生成包含对象轨迹和时间事件的时空感知轨迹，然后基于感知证据生成最终字幕。通过设计两阶段训练策略，PercepCap在直接字幕生成和字幕到问答评估中均显著提升了字幕质量。

🔬 方法详解

问题定义：本文旨在解决视频字幕生成中对时空感知的不足，现有方法无法明确展示感知证据，导致错误难以追踪。

核心思路：PercepCap通过感知-描述生成链，首先生成时空感知轨迹，确保生成的字幕与感知证据相一致，从而提高字幕的准确性和可解释性。

技术框架：该框架包括两个主要阶段：第一阶段为感知生成，提取对象轨迹和时间事件；第二阶段为基于感知证据生成最终字幕。训练过程中采用了感知-描述监督微调和感知基础强化学习策略。

关键创新：PercepCap的核心创新在于将感知证据显式化，采用感知-描述生成链的方式，使得生成过程更具可解释性，与现有方法相比，能够更好地识别和纠正潜在的感知错误。

关键设计：在训练过程中，采用了基于字幕的感知数据构建方法，确保生成的感知轨迹与字幕内容一致，使用联合奖励机制优化感知轨迹和字幕质量。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

实验结果显示，PercepCap在直接字幕生成和字幕到问答评估中均显著优于Qwen3-VL基线，具体提升幅度达到X%（具体数据未知），展示了其在字幕生成质量上的领先性。

🎯 应用场景

PercepCap的研究成果可广泛应用于视频内容分析、智能监控、自动化视频编辑等领域，提升视频理解的准确性和效率。未来，该框架有望推动多模态学习和自然语言处理的结合，促进更智能的内容生成和交互。

📄 摘要（原文）

Video captioning requires fine-grained spatio-temporal understanding of videos, including spatial perception of where objects are located and temporal perception of when events occur. Existing MLLMs usually generate captions directly from video inputs without exposing the perceptual evidence behind descriptions. As a result, mistakes in spatiotemporal perception are only observed in the final caption, making it difficult to identify the underlying perceptual errors directly. To address these issues, we present PercepCap, a perception-aware video captioning framework that makes perceptual evidence explicit before producing the final caption. Specifically, PercepCap follows a perceive-describe generation chain, where the model first produces a spatiotemporal perception trace comprising object trajectories and temporal events, and then generates the final caption conditioned on the perceived evidence. To support this, we design a two-stage training strategy. Perceive-then-Describe Supervised Fine-tuning adapts the model from caption-only generation to the proposed perceive-describe chain, while Perception-Grounded Reinforcement Learning optimizes perception trace and caption quality with joint rewards over perception chain and the final caption. To support our two-stage training, we introduce Caption-Anchored Perception Data Construction. This pipeline builds the SFT and RL training data by first generating a caption-only description, extracting the objects and events it mentions, and grounding them back in the video with boxes and timestamps. This yields caption-aligned perception data that provides solid training ground truth, ensuring that the explicit perception trace and final caption refer to the same objects and events. Across direct caption and caption-to-QA evaluation, PercepCap consistently improves upon the Qwen3-VL baseline and demonstrates leading caption quality.