Multimodal Flow: Unified Flow Modeling of Language and Vision in Embedding Spaces

📄 arXiv: 2609.40362v1 📥 PDF

作者: Hongyuan Tao, Xinggang Wang, Lianghui Zhu, Yongkang Li, Yunchao Wei, Bin Feng, Shaoyu Chen, Qian Zhang, Chang Huang, Kai Yu

分类: cs.CV

发布日期: 2026-09-30

备注: 18 pages, 5 figures, 10 tables. Code and model: https://github.com/hustvl/Multimodal-Flow

🔗 代码/项目: GITHUB

💡 一句话要点

提出Multimodal Flow以解决多模态建模的量化瓶颈问题

🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture) 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 多模态建模 生成模型 语言与视觉 连续建模 深度学习

📋 核心要点

  1. 现有统一多模态模型存在量化瓶颈，无法有效整合语言和视觉信息。
  2. 论文提出Multimodal Flow，通过完全连续建模，避免了离散建模的局限性，实现了共享生成过程。
  3. 在多模态数据上进行预训练后，MF-1在多个基准测试中表现优异，显示出其竞争力和有效性。

📝 摘要（中文）

我们提出了Multimodal Flow，这是一种语言和视觉的完全连续生成模型。现有的统一多模态模型通常将语言和量化图像建模为离散标记，或将离散语言预测与连续图像生成结合，前者引入了视觉量化瓶颈，后者需要依赖于特定模态的目标和采样过程。完全连续建模避免了这些权衡，并实现了共享生成过程，但在多模态预训练中仍未得到充分探索。Multimodal Flow引入了一种统一的连续架构，整合了多模态连续表示与共享的块因果流骨干，组织文本块和图像为有序的连续超块，保持文本标记顺序和视觉空间结构。该模型在训练期间并行预测多个目标块，在推理时顺序生成超块。

🔬 方法详解

问题定义：本论文旨在解决现有多模态模型在语言和视觉信息整合时的量化瓶颈问题。现有方法通常依赖于离散标记，导致信息损失和建模效率低下。

核心思路：论文提出的Multimodal Flow模型采用完全连续的生成方式，避免了离散建模的局限性，允许语言和视觉信息在同一生成过程中进行交互。通过共享的块因果流骨干，模型能够有效整合多模态信息。

技术框架：该模型的整体架构包括文本块和图像的有序连续超块，利用Flow Matching学习单一向量场。模型通过联合注意力机制实现跨模态交互，并使用特定的前馈网络处理各自模态。训练时并行预测多个目标块，推理时顺序生成超块。

关键创新：最重要的创新在于提出了基于连续块的嵌入流建模方法，这一方法在多模态建模中实现了完全连续的生成过程，与传统的离散模型形成鲜明对比。

关键设计：模型设计中采用了共享的块因果流骨干，文本和图像的组织方式保持了其原有的顺序和结构。此外，模型在训练过程中使用了特定的损失函数和优化策略，以确保多模态信息的有效融合。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

在多个基准测试中，MF-1在仅使用150B预训练标记的情况下，平均得分达到82.8，显示出其在多模态建模中的竞争力。此外，在匹配的数据和优化条件下，Multimodal Flow在性能上超越了代表性的混合和离散模型，证明了其有效性。

🎯 应用场景

该研究的潜在应用领域包括自然语言处理、计算机视觉和人机交互等。通过实现语言与视觉信息的有效整合，Multimodal Flow可以在智能助手、自动驾驶、内容生成等多个场景中发挥重要作用，推动多模态技术的发展。

📄 摘要（原文）

We present Multimodal Flow, a fully continuous generative model of language and vision. Most unified multimodal models either model both language and quantized images as discrete tokens or combine discrete language prediction with continuous image generation. The former introduces a visual quantization bottleneck. The latter requires modality-dependent objectives and sampling procedures. Fully continuous modeling avoids these trade-offs and enables a shared generative process, but remains underexplored for multimodal pretraining. Multimodal Flow introduces a unified continuous architecture that integrates multimodal continuous representations with a shared chunk-causal flow backbone. It organizes text blocks and images as ordered continuous hyperchunks, preserving textual token order and visual spatial structure. The backbone learns a single vector field over these hyperchunks through Flow Matching. Joint attention enables cross-modal interaction, while modality-specific feed-forward networks process each modality. The model predicts multiple target chunks in parallel during training and generates hyperchunks sequentially at inference. We instantiate MF-1 and pretrain it on multimodal data. Across 0.6B, 1.2B, and 1.6B scales, continued pretraining consistently improves multimodal modeling. With only 150B pretraining tokens, MF-1 achieves an average score of 82.8 across GenEval and DPG-Bench and 75.3 across VQAv2, MMBench, and POPE, remaining competitive with unified models trained on substantially more data. Under matched data, optimization, and parameter budgets, Multimodal Flow further outperforms representative hybrid and discrete models. These results establish continuous chunk-based embedding flow modeling as a new fully continuous paradigm for unified multimodal modeling. The related code and model are publicly released at https://github.com/hustvl/Multimodal-Flow.