VideoHarness-SRSI: Structured Recursive Self-Improvement for Video Understanding Harness

📄 arXiv: 2610.19424v1 📥 PDF

作者: Zihao Lin, Haoyu Dong, Haobo Xu, Chi-Chih Chang, Xuying Ning, Yinglong Xia, Shuang Yang, Zhuokai Zhao, Xinying Song, Dongfang Liu, Lifu Huang, Yang Song, Qifan Wang

分类: cs.CV, cs.AI

发布日期: 2026-10-14

备注: 50 pages, 15 figures

💡 一句话要点

提出VideoHarness-SRSI以解决视频理解中的自我改进问题

🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 视频理解 自我改进 证据驱动 多模态学习 递归框架 性能优化 验证过拟合 搜索策略

📋 核心要点

  1. 现有方法在视频理解中面临验证过拟合、搜索预算稀释和早期经验陷阱等挑战，影响了自我改进的可靠性。
  2. 论文提出VideoHarness-SRSI框架，采用证据驱动的方式进行搜索决策，确保反馈内容无关且在搜索前收集。
  3. 在四个视频理解基准上，VideoHarness-SRSI相较于VideoHarness-BRSI提升了6.2%的均值得分，并减少了37.9%的验证-测试差距。

📝 摘要（中文）

大型语言模型（LLM）系统的能力不仅依赖于模型参数，还依赖于组织信息获取、工具使用、推理和记忆的可执行框架。递归框架的演化可以自动化设计，但如何在有限、噪声和重复反馈中可靠发现可泛化系统仍不清楚。本文研究了这一问题，提出了VideoHarness-SRSI框架，通过证据驱动的方式优化视频理解中的自我改进，显著提升了性能。

🔬 方法详解

问题定义：本文旨在解决视频理解中的自我改进框架在有限和噪声反馈下的可靠性问题。现有方法存在验证过拟合和搜索策略不当等痛点，导致性能不稳定。

核心思路：提出VideoHarness-SRSI框架，通过证据驱动的方式优化搜索决策，确保所用证据可转移且充足，从而提升自我改进的效果。

技术框架：框架包括几个主要模块：首先是冻结的视觉-语言模型，其次是基于证据的搜索决策机制，最后是独立的探测实验和联合优化阶段。

关键创新：最重要的创新在于引入了证据驱动的搜索策略，避免了以往方法中因不完善的验证证据导致的多重失败模式，确保了搜索过程的有效性。

关键设计：在设计中，限制了提议者使用内容无关的反馈，采用了仪器化的热身实验来指导探索，并在独立的探测分支中开发机制，确保了优化过程的系统性和可靠性。

🖼️ 关键图片

img_0
img_1
img_2

📊 实验亮点

实验结果显示，VideoHarness-SRSI在四个视频理解基准上相较于VideoHarness-BRSI提升了6.2%的均值得分，并显著减少了37.9%的验证-测试差距，证明了其在自我改进过程中的有效性和可靠性。

🎯 应用场景

该研究的潜在应用领域包括视频监控、自动视频摘要、视频内容分析等，能够为多模态信息处理提供更高效的解决方案。未来，该框架可能在智能视频理解和人机交互等领域产生深远影响。

📄 摘要（原文）

LLM system capabilities depend not only on model parameters but also on the executable harnesses that organize information acquisition, tool use, reasoning, and memory. Recursive harness evolution can automate their design, but how it can reliably discover generalizable systems from limited, noisy, and repeatedly reused feedback remains unclear. We study this question in agentic video understanding, where rich multimodal evidence and coupled mechanisms make harness optimization challenging. We first introduce VideoHarness-BRSI, a recursive self-improvement framework that evolves video harnesses around a frozen vision-language model. Analyzing over 4,000 evolved candidates, we find it competitive with hand-engineered systems but identify three interconnected failure modes: validation overfitting, search-budget dilution, and the early-experience trap. We trace all three to a common root: the same imperfect validation evidence drives both harness revisions and the proposer's search strategy. We therefore propose VideoHarness-SRSI, an evidence-governed framework in which every search decision rests on evidence that is transferable, sufficient, and gathered before the search commits. It restricts the proposer to content-free feedback, informs exploration with instrumented warm-up experiments, and develops mechanisms in isolated probe-race branches before factorial composition and joint refinement. Across four video understanding benchmarks, VideoHarness-SRSI improves the equally weighted mean held-out score by 6.2% relative to VideoHarness-BRSI and reduces the mean validation-test gap by 37.9%. Reliable harness evolution thus depends on how search is organized and informed, not only on how well it optimizes validation performance.