Idea23D: Collaborative LMM Agents Enable 3D Model Generation from Interleaved Multimodal Inputs

📄 arXiv: 2404.04363v2 📥 PDF

作者: Junhao Chen, Xiang Li, Xiaojun Ye, Chao Li, Zhaoxin Fan, Hao Zhao

分类: cs.CV

发布日期: 2024-04-05 (更新: 2024-12-18)

备注: Accepted by COLING 2025 (The 31st International Conference on Computational Linguistics) Project Page: https://idea23d.github.io/ Code: https://github.com/yisuanwang/Idea23D

💡 一句话要点

提出Idea23D以解决多模态输入下3D模型生成问题

🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 3D生成 多模态输入 人工智能生成内容 大型多模态模型 自动化生成 创意设计 虚拟现实

📋 核心要点

  1. 现有3D AIGC方法未能充分利用多模态输入，限制了人类创造力的发挥。
  2. 提出Idea23D框架，通过三个LMM代理协作生成3D模型，自动化程度高，无需人工干预。
  3. 实验结果显示Idea23D在3D生成任务中显著提高了成功率和准确性，超越了以往方法。

📝 摘要（中文）

随着2D扩散模型的成功，2D AIGC内容已经改变了我们的生活。最近，这一成功扩展到了3D AIGC，现有方法能够从单幅图像或文本生成纹理3D模型。然而，当前的3D AIGC方法尚未充分释放人类创造力。本文提出了一种新颖的3D AIGC方法：从IDEA生成3D内容。IDEA是由文本、图像和3D模型组成的多模态输入。我们提出的Idea23D框架结合了三个基于大型多模态模型的代理，分别负责提示生成、模型选择和反馈反思。实验结果表明，Idea23D在3D生成的成功率和准确性上显著提升，并且在各种LMM、文本到图像和图像到3D模型之间具有良好的兼容性。

🔬 方法详解

问题定义：本文旨在解决当前3D AIGC方法在多模态输入下生成3D模型的不足，尤其是无法充分利用人类的创造性想象。现有方法主要依赖单一输入，限制了生成内容的丰富性和多样性。

核心思路：Idea23D框架通过引入IDEA这一多模态输入形式，结合文本、图像和3D模型，利用三个大型多模态模型代理进行协作，生成更符合人类想象的3D内容。这样的设计使得生成过程更加灵活和多样化。

技术框架：Idea23D框架包含三个主要模块：提示生成代理、模型选择代理和反馈反思代理。这些代理在一个完全自动化的循环中协作，生成与输入IDEA紧密对齐的3D模型。

关键创新：最重要的创新在于引入了IDEA这一概念，结合多模态输入进行3D内容生成，突破了传统方法的局限，能够更好地反映人类的创造性想象。

关键设计：框架中采用了先进的LMM技术，设计了适应多模态输入的损失函数和网络结构，以确保生成的3D模型与输入IDEA的高一致性和准确性。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

实验结果表明，Idea23D在3D生成任务中的成功率和准确性显著提高，具体数据表明其性能超越了以往的基线方法，成功率提升幅度达到XX%，准确性提升幅度达到YY%。

🎯 应用场景

该研究的潜在应用领域包括游戏开发、虚拟现实、动画制作等，能够为创作者提供更丰富的工具，提升3D内容生成的效率和质量。未来，该技术可能在教育、设计等多个领域产生深远影响，促进人机协作的创新。

📄 摘要（原文）

With the success of 2D diffusion models, 2D AIGC content has already transformed our lives. Recently, this success has been extended to 3D AIGC, with state-of-the-art methods generating textured 3D models from single images or text. However, we argue that current 3D AIGC methods still do not fully unleash human creativity. We often imagine 3D content made from multimodal inputs, such as what it would look like if my pet bunny were eating a doughnut on the table. In this paper, we explore a novel 3D AIGC approach: generating 3D content from IDEAs. An IDEA is a multimodal input composed of text, image, and 3D models. To our knowledge, this challenging and exciting 3D AIGC setting has not been studied before. We propose the new framework Idea23D, which combines three agents based on large multimodal models (LMMs) and existing algorithmic tools. These three LMM-based agents are tasked with prompt generation, model selection, and feedback reflection. They collaborate and critique each other in a fully automated loop, without human intervention. The framework then generates a text prompt to create 3D models that align closely with the input IDEAs. We demonstrate impressive 3D AIGC results that surpass previous methods. To comprehensively assess the 3D AIGC capabilities of Idea23D, we introduce the Eval3DAIGC-198 dataset, containing 198 multimodal inputs for 3D generation tasks. This dataset evaluates the alignment between generated 3D content and input IDEAs. Our user study and quantitative results show that Idea23D significantly improves the success rate and accuracy of 3D generation, with excellent compatibility across various LMM, Text-to-Image, and Image-to-3D models. Code and dataset are available at \url{https://idea23d.github.io/}.