SweetDreamer: Aligning Geometric Priors in 2D Diffusion for Consistent Text-to-3D
作者: Weiyu Li, Rui Chen, Xuelin Chen, Ping Tan
分类: cs.CV
发布日期: 2023-10-04 (更新: 2023-10-20)
备注: Project page: https://sweetdreamer3d.github.io/
💡 一句话要点
提出SweetDreamer以解决2D到3D生成中的几何不一致问题
🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture)
关键词: 文本到3D生成 几何一致性 扩散模型 视角感知 高质量生成
📋 核心要点
- 现有的2D扩散模型在将2D结果提升到3D时缺乏几何知识，导致多视角不一致问题。
- 本文通过对齐2D几何先验与3D形状，微调扩散模型，使其具备视角感知能力，从而解决几何不一致问题。
- 实验结果显示，本文方法在一致性评估中达到85%以上，显著优于传统方法的30%一致性率。
📝 摘要（中文）
在文本到3D生成中，从预训练的扩散模型提升2D结果到3D世界存在固有的模糊性。现有的2D扩散模型仅学习视角无关的先验知识，缺乏3D知识，导致多视角不一致问题。本文提出通过将2D几何先验与明确的3D形状对齐来改善一致性，显著缓解了最终输出中的几何不一致问题。通过微调2D扩散模型，使其具备视角感知能力，生成特定视角的3D对象坐标图，最终实现85%以上的人类评估一致性，远超以往方法的30%。
🔬 方法详解
问题定义：本文旨在解决文本到3D生成中的几何不一致问题，现有方法在提升2D结果时缺乏3D知识，导致多视角输出不一致。
核心思路：通过对齐2D扩散模型中的几何先验与明确的3D形状，提升模型的视角感知能力，从而减少几何结构的错误定位。
技术框架：整体流程包括对2D扩散模型的微调，使其能够生成视角特定的3D对象坐标图，使用粗略的3D信息进行对齐，确保几何一致性。
关键创新：本文的主要创新在于引入了对齐几何先验（AGP），使得2D扩散模型不仅能生成高质量的对象，还能在未见的形状和视觉外观上保持高通用性。
关键设计：在模型微调过程中，采用了特定的损失函数来优化视角感知能力，并确保生成的坐标图与3D形状的几何结构相一致。
🖼️ 关键图片
📊 实验亮点
实验结果表明，SweetDreamer在一致性评估中达到了85%以上的成功率，而许多传统方法的成功率仅为30%。这一显著提升展示了其在解决多视角不一致问题上的有效性和创新性。
🎯 应用场景
该研究的潜在应用领域包括游戏开发、虚拟现实和增强现实等，能够为这些领域提供更为一致和高质量的3D模型生成方案。未来，该方法可能推动更广泛的3D内容创作和设计工具的发展，提升用户体验。
📄 摘要（原文）
It is inherently ambiguous to lift 2D results from pre-trained diffusion models to a 3D world for text-to-3D generation. 2D diffusion models solely learn view-agnostic priors and thus lack 3D knowledge during the lifting, leading to the multi-view inconsistency problem. We find that this problem primarily stems from geometric inconsistency, and avoiding misplaced geometric structures substantially mitigates the problem in the final outputs. Therefore, we improve the consistency by aligning the 2D geometric priors in diffusion models with well-defined 3D shapes during the lifting, addressing the vast majority of the problem. This is achieved by fine-tuning the 2D diffusion model to be viewpoint-aware and to produce view-specific coordinate maps of canonically oriented 3D objects. In our process, only coarse 3D information is used for aligning. This "coarse" alignment not only resolves the multi-view inconsistency in geometries but also retains the ability in 2D diffusion models to generate detailed and diversified high-quality objects unseen in the 3D datasets. Furthermore, our aligned geometric priors (AGP) are generic and can be seamlessly integrated into various state-of-the-art pipelines, obtaining high generalizability in terms of unseen shapes and visual appearance while greatly alleviating the multi-view inconsistency problem. Our method represents a new state-of-the-art performance with an 85+% consistency rate by human evaluation, while many previous methods are around 30%. Our project page is https://sweetdreamer3d.github.io/