SketchBodyNet: A Sketch-Driven Multi-faceted Decoder Network for 3D Human Reconstruction

📄 arXiv: 2310.06577v1 📥 PDF

作者: Fei Wang, Kongzhang Tang, Hefeng Wu, Baoquan Zhao, Hao Cai, Teng Zhou

分类: cs.CV, cs.GR

发布日期: 2023-10-10

备注: 9 pages, to appear in Pacific Graphics 2023

💡 一句话要点

提出SketchBodyNet以解决从自由手绘草图重建3D人形的问题

🎯 匹配领域: 支柱三：空间感知与语义 (Perception & Semantics) 支柱六：视频提取与匹配 (Video Extraction)

关键词: 3D重建 自由手绘草图 多面解码器 自注意力机制 SMPL模型 计算机视觉 深度学习

📋 核心要点

  1. 现有方法在从自由手绘草图重建3D人形方面存在显著不足，主要体现在草图的稀疏性和缺乏细节上。
  2. 本文提出的SketchBodyNet通过多面解码器结构，结合多头自注意力机制，增强特征提取能力，从而实现更准确的3D重建。
  3. 实验结果表明，SketchBodyNet在重建自由手绘草图的3D人形网格方面表现优越，显著提升了重建精度。

📝 摘要（中文）

从2D图像重建3D人形的任务近年来受到越来越多的关注，尤其是自由手绘草图因其灵活性为3D重建提供了新的可能性。然而，草图的稀疏特性使得这一任务面临诸多挑战，如任意性、不准确性和缺乏细节等。现有方法在从单视角图像重建3D人形方面取得了成功，但在处理自由手绘草图时效果不佳。为此，本文提出了一种新颖的草图驱动多面解码器网络SketchBodyNet，旨在解决这一问题。该网络由主干和三个独立的注意力解码器分支组成，利用多头自注意力模块增强特征，最终与SMPL模型结合重建3D人形网格。通过收集约26k个自由手绘草图及其对应的3D网格数据集，验证了该方法的有效性。

🔬 方法详解

问题定义：本文旨在解决从自由手绘草图重建3D人形的挑战。现有方法在处理草图时效果不佳，主要由于草图的稀疏性和缺乏细节，导致重建过程中的不准确性和任意性。

核心思路：SketchBodyNet通过设计一个多面解码器网络，利用多头自注意力模块来增强特征提取能力，分别预测相机、形状和姿态参数，从而提高3D重建的准确性。

技术框架：该网络包括一个主干和三个独立的解码器分支。每个解码器通过多头自注意力模块提取特征，并通过多层感知机进行处理，最终与SMPL模型结合以重建3D人形网格。

关键创新：SketchBodyNet的主要创新在于其多面解码器结构和自注意力机制的结合，这使得网络能够更好地处理草图的稀疏特性，与传统方法相比，显著提升了重建效果。

关键设计：网络的设计包括多个关键参数设置，如解码器的层数、注意力头的数量等。此外，损失函数结合了草图与合成2D草图的对比，优化了模型的学习过程。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

实验结果显示，SketchBodyNet在重建自由手绘草图的3D人形网格方面表现优越，相较于基线方法，重建精度提升了显著的比例，具体性能数据未明确给出，但实验表明其在多种姿态下均能有效重建。

🎯 应用场景

该研究在3D人形重建领域具有广泛的应用潜力，尤其是在虚拟现实、动画制作和游戏开发等行业。通过提高从草图到3D模型的转换精度，能够为设计师和艺术家提供更高效的工具，推动创作过程的创新与发展。

📄 摘要（原文）

Reconstructing 3D human shapes from 2D images has received increasing attention recently due to its fundamental support for many high-level 3D applications. Compared with natural images, freehand sketches are much more flexible to depict various shapes, providing a high potential and valuable way for 3D human reconstruction. However, such a task is highly challenging. The sparse abstract characteristics of sketches add severe difficulties, such as arbitrariness, inaccuracy, and lacking image details, to the already badly ill-posed problem of 2D-to-3D reconstruction. Although current methods have achieved great success in reconstructing 3D human bodies from a single-view image, they do not work well on freehand sketches. In this paper, we propose a novel sketch-driven multi-faceted decoder network termed SketchBodyNet to address this task. Specifically, the network consists of a backbone and three separate attention decoder branches, where a multi-head self-attention module is exploited in each decoder to obtain enhanced features, followed by a multi-layer perceptron. The multi-faceted decoders aim to predict the camera, shape, and pose parameters, respectively, which are then associated with the SMPL model to reconstruct the corresponding 3D human mesh. In learning, existing 3D meshes are projected via the camera parameters into 2D synthetic sketches with joints, which are combined with the freehand sketches to optimize the model. To verify our method, we collect a large-scale dataset of about 26k freehand sketches and their corresponding 3D meshes containing various poses of human bodies from 14 different angles. Extensive experimental results demonstrate our SketchBodyNet achieves superior performance in reconstructing 3D human meshes from freehand sketches.