3DensiNet: A robust neural network architecture towards 3D volumetric object prediction from 2D image

Meng Wang, Lingjing Wang, Yi Fang

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

3D volumetric object generation/prediction from single 2D image is a quite challenging but meaningful task in 3D visual computing. In this paper, we propose a novel neural network architecture, named "3DensiNet", which uses density heat-map as an intermediate supervision tool for 2D-to-3D transformation. Specifically, we firstly present a 2D density heat-map to 3D volumetric object encoding-decoding network, which outperforms classical 3D autoencoder. Then we show that using 2D image to predict its density heat-map via a 2D to 2D encoding-decoding network is feasible. In addition, we leverage adversarial loss to fine tune our network, which improves the generated/predicted 3D voxel objects to be more similar to the ground truth voxel object. Experimental results on 3D volumetric prediction from 2D images demonstrates superior performance of 3DensiNet over other state-of-the-art techniques in handling 3D volumetric object generation/prediction from single 2D image.

Original languageEnglish (US)
Title of host publicationMM 2017 - Proceedings of the 2017 ACM Multimedia Conference
PublisherAssociation for Computing Machinery, Inc
Pages961-969
Number of pages9
ISBN (Electronic)9781450349062
DOIs
StatePublished - Oct 23 2017
Event25th ACM International Conference on Multimedia, MM 2017 - Mountain View, United States
Duration: Oct 23 2017Oct 27 2017

Other

Other25th ACM International Conference on Multimedia, MM 2017
CountryUnited States
CityMountain View
Period10/23/1710/27/17

Fingerprint

Network architecture
Neural networks
Decoding
Hot Temperature

Keywords

  • 3D reconstruction
  • 3D volumetric prediction
  • Deep learning

ASJC Scopus subject areas

  • Computer Graphics and Computer-Aided Design
  • Media Technology
  • Computer Vision and Pattern Recognition
  • Software

Cite this

Wang, M., Wang, L., & Fang, Y. (2017). 3DensiNet: A robust neural network architecture towards 3D volumetric object prediction from 2D image. In MM 2017 - Proceedings of the 2017 ACM Multimedia Conference (pp. 961-969). Association for Computing Machinery, Inc. https://doi.org/10.1145/3123266.3123340

3DensiNet : A robust neural network architecture towards 3D volumetric object prediction from 2D image. / Wang, Meng; Wang, Lingjing; Fang, Yi.

MM 2017 - Proceedings of the 2017 ACM Multimedia Conference. Association for Computing Machinery, Inc, 2017. p. 961-969.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Wang, M, Wang, L & Fang, Y 2017, 3DensiNet: A robust neural network architecture towards 3D volumetric object prediction from 2D image. in MM 2017 - Proceedings of the 2017 ACM Multimedia Conference. Association for Computing Machinery, Inc, pp. 961-969, 25th ACM International Conference on Multimedia, MM 2017, Mountain View, United States, 10/23/17. https://doi.org/10.1145/3123266.3123340
Wang M, Wang L, Fang Y. 3DensiNet: A robust neural network architecture towards 3D volumetric object prediction from 2D image. In MM 2017 - Proceedings of the 2017 ACM Multimedia Conference. Association for Computing Machinery, Inc. 2017. p. 961-969 https://doi.org/10.1145/3123266.3123340
Wang, Meng ; Wang, Lingjing ; Fang, Yi. / 3DensiNet : A robust neural network architecture towards 3D volumetric object prediction from 2D image. MM 2017 - Proceedings of the 2017 ACM Multimedia Conference. Association for Computing Machinery, Inc, 2017. pp. 961-969
@inproceedings{ef2aa13b05a548379a4443c9147c3b74,
title = "3DensiNet: A robust neural network architecture towards 3D volumetric object prediction from 2D image",
abstract = "3D volumetric object generation/prediction from single 2D image is a quite challenging but meaningful task in 3D visual computing. In this paper, we propose a novel neural network architecture, named {"}3DensiNet{"}, which uses density heat-map as an intermediate supervision tool for 2D-to-3D transformation. Specifically, we firstly present a 2D density heat-map to 3D volumetric object encoding-decoding network, which outperforms classical 3D autoencoder. Then we show that using 2D image to predict its density heat-map via a 2D to 2D encoding-decoding network is feasible. In addition, we leverage adversarial loss to fine tune our network, which improves the generated/predicted 3D voxel objects to be more similar to the ground truth voxel object. Experimental results on 3D volumetric prediction from 2D images demonstrates superior performance of 3DensiNet over other state-of-the-art techniques in handling 3D volumetric object generation/prediction from single 2D image.",
keywords = "3D reconstruction, 3D volumetric prediction, Deep learning",
author = "Meng Wang and Lingjing Wang and Yi Fang",
year = "2017",
month = "10",
day = "23",
doi = "10.1145/3123266.3123340",
language = "English (US)",
pages = "961--969",
booktitle = "MM 2017 - Proceedings of the 2017 ACM Multimedia Conference",
publisher = "Association for Computing Machinery, Inc",

}

TY - GEN

T1 - 3DensiNet

T2 - A robust neural network architecture towards 3D volumetric object prediction from 2D image

AU - Wang, Meng

AU - Wang, Lingjing

AU - Fang, Yi

PY - 2017/10/23

Y1 - 2017/10/23

N2 - 3D volumetric object generation/prediction from single 2D image is a quite challenging but meaningful task in 3D visual computing. In this paper, we propose a novel neural network architecture, named "3DensiNet", which uses density heat-map as an intermediate supervision tool for 2D-to-3D transformation. Specifically, we firstly present a 2D density heat-map to 3D volumetric object encoding-decoding network, which outperforms classical 3D autoencoder. Then we show that using 2D image to predict its density heat-map via a 2D to 2D encoding-decoding network is feasible. In addition, we leverage adversarial loss to fine tune our network, which improves the generated/predicted 3D voxel objects to be more similar to the ground truth voxel object. Experimental results on 3D volumetric prediction from 2D images demonstrates superior performance of 3DensiNet over other state-of-the-art techniques in handling 3D volumetric object generation/prediction from single 2D image.

AB - 3D volumetric object generation/prediction from single 2D image is a quite challenging but meaningful task in 3D visual computing. In this paper, we propose a novel neural network architecture, named "3DensiNet", which uses density heat-map as an intermediate supervision tool for 2D-to-3D transformation. Specifically, we firstly present a 2D density heat-map to 3D volumetric object encoding-decoding network, which outperforms classical 3D autoencoder. Then we show that using 2D image to predict its density heat-map via a 2D to 2D encoding-decoding network is feasible. In addition, we leverage adversarial loss to fine tune our network, which improves the generated/predicted 3D voxel objects to be more similar to the ground truth voxel object. Experimental results on 3D volumetric prediction from 2D images demonstrates superior performance of 3DensiNet over other state-of-the-art techniques in handling 3D volumetric object generation/prediction from single 2D image.

KW - 3D reconstruction

KW - 3D volumetric prediction

KW - Deep learning

UR - http://www.scopus.com/inward/record.url?scp=85035201598&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85035201598&partnerID=8YFLogxK

U2 - 10.1145/3123266.3123340

DO - 10.1145/3123266.3123340

M3 - Conference contribution

AN - SCOPUS:85035201598

SP - 961

EP - 969

BT - MM 2017 - Proceedings of the 2017 ACM Multimedia Conference

PB - Association for Computing Machinery, Inc

ER -