Learning via Self-Consistency for Diffusion-based Video Reasoning

📄 arXiv: 2609.36826v1 📥 PDF

作者: Zhenghao Ni, Weimin Qiu, Meng Tang

分类: cs.CV

发布日期: 2026-09-29

备注: 23 pages, including 10 pages for the main body

💡 一句话要点

提出自一致性方法以提升扩散模型的视频推理能力

🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture) 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 视频推理 自一致性 扩散模型 拒绝微调 视觉搜索 迷宫求解 蒸馏训练

📋 核心要点

  1. 现有的扩散视频生成模型在视觉推理任务中表现出随机性，导致其在确定性任务中效果不佳。
  2. 论文提出了一种基于自一致性的推理方法，通过聚合多个生成结果来提高视频推理的准确性和稳定性。
  3. 实验结果显示，该方法在多个任务中显著提升了性能，例如视觉搜索任务的准确率从48.4%提升至99.0%。

📝 摘要（中文）

视频生成模型在视觉推理、感知等任务中展现出新兴的零样本能力。然而，基于扩散的视频生成本质上是随机的，而许多下游视觉任务则是确定性的。本文探讨自一致性在扩散视频推理中的应用，提出了一种无训练的测试时缩放方法，通过自一致性聚合多个视频生成的预测。我们在去噪轨迹中提前读取预测，显著减少去噪步骤，同时保持共识质量。进一步提出拒绝微调（RFT）以将共识预测蒸馏到视频生成模型中，最终模型在推理时仅需单次生成，且性能显著优于原模型。实验表明，该方法在迷宫求解、视觉搜索和指代分割等任务中显著提升了视频感知和推理能力。对于视觉搜索，自一致性将任务准确率从48.4%提升至99.0%。

🔬 方法详解

问题定义：本文旨在解决扩散视频生成模型在确定性视觉推理任务中的随机性问题，现有方法在此类任务中表现不佳，导致准确率低下。

核心思路：通过引入自一致性的方法，聚合多个视频生成的结果，以提高推理的准确性和稳定性。该设计借鉴了大语言模型中的链式思维推理效果。

技术框架：整体流程包括生成多个视频样本、提取路径、位置或掩码，并通过自一致性进行聚合。为减少推理开销，提前在去噪轨迹中读取预测结果。

关键创新：提出了一种拒绝微调（RFT）方法，将共识预测蒸馏到生成模型中，使得模型在推理时仅需单次生成，且性能显著提升。与传统方法相比，减少了多次生成的需求。

关键设计：在去噪过程中，采用早期读取预测的策略，保持共识质量的同时减少去噪步骤，此外，RFT方法的设计确保了共识信息的有效传递。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

实验结果显示，自一致性推理方法在视觉搜索任务中将准确率从48.4%提升至99.0%，而蒸馏后的模型在单次生成中仍保留了大部分共识优势。在4x4迷宫求解任务中，基于共识的训练将单次生成的严格成功率从72.0%提升至84.0%。

🎯 应用场景

该研究的潜在应用领域包括视频理解、智能监控、自动驾驶等场景，能够在没有真实视频或任务特定验证的情况下，提升视频推理的准确性和可靠性。未来，该方法有望推动更复杂视觉任务的研究与应用。

📄 摘要（原文）

Video generation models have demonstrated emerging zero-shot capabilities for visual reasoning, perception, and other vision tasks. However, diffusion-based video generation is inherently stochastic, while many downstream vision tasks are deterministic. Motivated by the effectiveness of self-consistency in chain-of-thought reasoning for large language models, we investigate whether self-consistency can similarly improve diffusion-based video reasoning. We first introduce a training-free test-time scaling method that samples multiple video generations and aggregates their predictions through self-consistency. Specifically, we aggregate extracted paths, locations, or masks from multiple rollouts into a consensus prediction. To reduce the inference overhead of multi-rollout generation, we read out predictions early in the denoising trajectory, which preserves consensus quality while reducing denoising steps by more than half. We further propose Rejection Fine-Tuning (RFT) to distill consensus predictions into the video generation model. The resulting model internalizes the benefit of multi-sample consensus and requires only a single generation at inference time, while substantially outperforming the original model. Experiments on three tasks, including maze solving, visual search, and referring segmentation, show that both our self-consistency inference and consensus distillation dramatically improve video-based perception and reasoning, without requiring ground-truth videos or task-specific verification. For visual search, self-consistency raises task accuracy from 48.4% for a single generation to 99.0%. The distilled model retains much of the consensus benefit with a single rollout. For 4-by-4 maze solving, consensus-based training improves the single-generation strict success rate from 72.0% to 84.0% with the same inference latency.