Unsupervised learning of 3D model reconstruction from hand-drawn sketches

Lingjing Wang, Jifei Wang, Cheng Qian, Yi Fang

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

3D objects modeling has gained considerable attention in the visual computing community. We propose a low-cost unsupervised learning model for 3D objects reconstruction from hand-drawn sketches. Recent advancements in deep learning opened new opportunities to learn high-quality 3D objects from 2D sketches via supervised networks. However, the limited availability of labeled 2D hand-drawn sketches data (i.e. sketches and its corresponding 3D ground truth models) hinders the training process of supervised methods. In this paper, driven by a novel design of combination of retrieval and reconstruction process, we developed a learning paradigm to reconstruct 3D objects from hand-drawn sketches, without the use of well-labeled hand-drawn sketch data during the entire training process. Specifically, the paradigm begins with the training of an adaption network via autoencoder with adversarial loss, embedding the unpaired 2D rendered image domain with the hand-drawn sketch domain to a shared latent vector space. Then from the embedding latent space, for each testing sketch image, we retrieve a few (e.g. five) nearest neighbors from the training 3D data set as prior knowledge for a 3D Generative Adversarial Network. Our experiments verify our network's robust and superior performance in handling 3D volumetric object generation from single hand-drawn sketch without requiring any 3D ground truth labels.

Original languageEnglish (US)
Title of host publicationMM 2018 - Proceedings of the 2018 ACM Multimedia Conference
PublisherAssociation for Computing Machinery, Inc
Pages1820-1828
Number of pages9
ISBN (Electronic)9781450356657
DOIs
Publication statusPublished - Oct 15 2018
Event26th ACM Multimedia conference, MM 2018 - Seoul, Korea, Republic of
Duration: Oct 22 2018Oct 26 2018

Other

Other26th ACM Multimedia conference, MM 2018
CountryKorea, Republic of
CitySeoul
Period10/22/1810/26/18

    Fingerprint

Keywords

  • Generative Model
  • Sketch Modeling
  • Unsupervised Learning

ASJC Scopus subject areas

  • Computer Graphics and Computer-Aided Design
  • Human-Computer Interaction

Cite this

Wang, L., Wang, J., Qian, C., & Fang, Y. (2018). Unsupervised learning of 3D model reconstruction from hand-drawn sketches. In MM 2018 - Proceedings of the 2018 ACM Multimedia Conference (pp. 1820-1828). Association for Computing Machinery, Inc. https://doi.org/10.1145/3240508.3240699