MMOE: Modernizing Diffusion Transformers with Efficient Expert Design

📄 arXiv: 2607.24665v1 📥 PDF

作者: Yanhao Jia, Jiepeng Wang, Haibin Huang, Chi Zhang, Erik Cambria, Xuelong Li

分类: cs.CV, cs.GR, cs.LG

发布日期: 2026-07-27

备注: 13 pages, 5 figures

💡 一句话要点

提出MMOE以平衡生成质量与训练成本问题

🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 稀疏专家 扩散变换器 生成模型 AIGC 效率优化 模型收敛 质量平衡

📋 核心要点

  1. 现有的稀疏专家方法往往只关注参数数量的增加，未能有效控制生成质量与训练成本之间的平衡。
  2. 本文提出MMOE，通过引入现代化的专家设计，优化扩散变换器的生成效率与质量。
  3. 实验结果显示，MMOE在训练步骤中收敛速度更快，且在稀疏变体中实现了最佳的质量与成本平衡。

📝 摘要（中文）

现代大型语言模型通过容量增长与效率相结合，成功地控制了每个token的成本。尽管AIGC基础模型（AFMs）开始采用稀疏专家，但现有方法往往只增加参数数量和稀疏比，未能有效平衡生成质量与训练成本。为此，本文提出了现代化的MMOE，系统性地将路由专家、共享轻量专家、门控残差路由和注意力残差信息重用等机制应用于扩散变换器。实验结果表明，MMOE在训练和采样协议匹配的情况下，能够在每个记录的检查点上实现更低的FID，且在稀疏变体中达到最佳的质量-成本平衡。

🔬 方法详解

问题定义：本文旨在解决现有稀疏专家方法在生成质量与训练成本之间的失衡问题，尤其是在AIGC基础模型中的应用痛点。

核心思路：MMOE通过引入现代化的专家设计，系统性地适配路由专家和轻量专家，以提高生成效率和质量。

技术框架：整体架构包括扩散变换器的多个模块，主要涉及路由专家、共享专家、门控残差路由和注意力残差信息重用等。

关键创新：MMOE的创新在于将多种现代专家组件组合应用于扩散变换器，研究其对收敛性、效率和生成质量的影响，而非简单的插件替换。

关键设计：在实验中，使用单个八GPU H100节点，批量大小为256，训练400k步，确保了实验的可重复性和可访问性。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

实验结果显示，MMOE在每个记录的检查点上均实现了更低的FID，相较于密集和中间稀疏专家基线，收敛速度更快。此外，在稀疏变体中，MMOE达到了最佳的质量与成本平衡，展示了其在生成任务中的优势。

🎯 应用场景

该研究的潜在应用领域包括自然语言生成、对话系统和内容创作等。通过优化生成模型的效率与质量，MMOE有望在实际应用中降低成本，提高用户体验，推动AIGC技术的广泛应用。

📄 摘要（原文）

Modern large language models scale successfully by pairing capacity growth with efficiency, keeping per-token and deployment costs under control as capacity grows. AIGC Foundation Models (AFMs), especially diffusion-transformer backbones, have begun to adopt sparse experts, but recent efforts mostly enlarge total parameter counts and sparsity ratios without importing the efficiency mechanisms that made LLM scaling practical, so generation quality is seldom balanced against training and deployment cost. This raises a natural question: can the architectural principles behind efficient LLM scaling be adapted to AFMs in a more balanced way? We introduce ModernMOE (MMOE), a modernization of SiT-style diffusion transformers that systematically adapts routed experts, shared and lightweight experts, gate-residual routing, and attention-residual information reuse to AIGC generation. Rather than treating MoE as a single plug-in replacement, MMOE studies how different modern expert components affect convergence, efficiency, and generation quality when composed inside a diffusion transformer. Every experiment in this paper is trained on a single eight-GPU H100 node with batch size 256 for 400k steps, an accessible single-machine budget. Under matched training and sampling protocols and at this budget, MMOE reaches lower FID at every recorded checkpoint, that is, it converges faster per training step, than dense and intermediate sparse-expert baselines, and among the sparse variants it attains the best quality-cost balance. Routing analysis further shows stable expert specialization across depth, substantial use of lightweight routes, and modest step-to-step routing changes during denoising. These results suggest that AFMs can follow the balanced scaling path of LLMs by importing proven efficiency designs, rather than by simply increasing total parameters and sparsity ratios.