Sprout: Building Dynamic Memory While Reasoning for Agentic Video Understanding
作者: Wei Chen, Xuanyu Zheng, Yancheng Long, Haoyang Xu, Kaiyu Jiang, Bin Wen, Tingting Gao, Han Li, Long Chen
分类: cs.CV
发布日期: 2026-09-28
备注: 20 pages, 9 figures
💡 一句话要点
提出Sprout框架以解决长视频理解中的动态记忆问题
🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)
关键词: 长视频理解 动态记忆 多模态学习 视频问答 时间树结构 在线学习 信息提炼
📋 核心要点
- 现有方法在处理长视频时，通常采用离线构建记忆的方式，导致在面对多个问题时效率低下。
- 本文提出Sprout框架，能够在推理过程中动态构建记忆，利用时间树结构逐步细化信息。
- 实验表明，Sprout在多个基准测试中表现出竞争力或更高的准确性，同时降低了每个问题的上下文成本。
📝 摘要（中文）
长视频理解依赖视频记忆来克服多模态大语言模型的上下文限制。现有方法采用离线构建记忆的流程，导致在面对多个问题时效率低下。为此，本文提出Sprout框架，在推理过程中动态构建记忆，通过时间树结构逐步细化节点。该框架在回答问题时逐段观看视频，并在关键时刻以更高帧率回访以提炼细节。实验结果表明，Sprout在多个基准测试中相较于传统离线记忆方法，取得了竞争力或更高的准确性，同时显著降低了上下文使用成本。
🔬 方法详解
问题定义：本文旨在解决长视频理解中，现有方法在面对多个问题时的记忆构建效率低下的问题。现有方法通常在离线阶段构建记忆，导致信息无法动态更新，影响推理效果。
核心思路：Sprout框架的核心思路是通过在推理过程中动态构建记忆，利用时间树结构逐步细化节点，确保信息在回答每个问题时都能得到更新和优化。
技术框架：Sprout的整体架构包括视频段的低帧率观看、问题回答时的记忆节点构建、关键时刻的高帧率回访等模块。每当回答一个问题时，都会记录相关视频段并更新记忆树。
关键创新：Sprout的主要创新在于其动态记忆构建机制，能够在推理过程中实时更新记忆，而非依赖于静态的离线构建。这一设计使得记忆能够随着问题的回答而不断优化。
关键设计：Sprout采用了时间树结构来组织记忆节点，关键参数设置包括节点的细化策略和视频段的选择机制。损失函数设计上，强调了记忆的准确性和上下文的有效利用。整体网络结构则支持动态更新和信息提炼。
🖼️ 关键图片
📊 实验亮点
在多个基准测试中，Sprout相较于传统的离线记忆方法，表现出竞争力或更高的准确性，且在上下文使用成本上显著降低。具体而言，Sprout在某些设置下实现了轻微的准确性提升，显示出其在动态记忆构建方面的优势。
🎯 应用场景
该研究具有广泛的应用潜力，尤其在视频问答、智能监控和视频内容分析等领域。通过动态记忆构建，Sprout能够提升系统在长视频理解中的效率和准确性，未来可能推动相关技术的进一步发展和应用。
📄 摘要（原文）
Long video understanding relies on video memory to overcome the context limits of multimodal large language models. Existing methods follow a build-then-reasoning pipeline: memory is built offline for the entire video, then reasoned over as a static source. In practice a long video is shared by several questions, and this pipeline is costly at both ends: with few questions, building memory for the whole video costs far more than answering them; with many questions, the memory is never updated, so what is learned while answering questions is lost to the next question. To alleviate these, we introduce Sprout, an agentic framework that builds memory while reasoning: a temporal tree that sprouts detailed nodes as questions are answered. The agent watches the video segment by segment at a low frame rate, stopping when the current question can be answered, remembers each segment as a coarse node of the tree, and revisits key intervals at a higher frame rate to refine the tree with the recovered details. Once a segment is recorded as text, its video input is removed from the context history, while the original video remains reachable through the video tools. The memory tree and prior question--answer records persist across questions, so the memory is online and dynamic: built from the first question onward and updated by every question thereafter. We find that replacing accumulated video inputs with textual memory substantially reduces context usage while maintaining accuracy, with slight improvements in some settings. Across benchmarks on three models, Sprout achieves competitive or improved accuracy relative to representative offline memory methods, with no upfront construction stage and lower context cost per question.