Learning barycentric representations of 3D shapes for sketch-based 3D shape retrieval

Jin Xie, Guoxian Dai, Fan Zhu, Yi Fang

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Retrieving 3D shapes with sketches is a challenging problem since 2D sketches and 3D shapes are from two heterogeneous domains, which results in large discrepancy between them. In this paper, we propose to learn barycenters of 2D projections of 3D shapes for sketch-based 3D shape retrieval. Specifically, we first use two deep convolutional neural networks (CNNs) to extract deep features of sketches and 2D projections of 3D shapes. For 3D shapes, we then compute the Wasserstein barycenters of deep features of multiple projections to form a barycentric representation. Finally, by constructing a metric network, a discriminative loss is formulated on the Wasserstein barycenters of 3D shapes and sketches in the deep feature space to learn discriminative and compact 3D shape and sketch features for retrieval. The proposed method is evaluated on the SHREC'13 and SHREC'14 sketch track benchmark datasets. Compared to the state-of-the-art methods, our proposed method can significantly improve the retrieval performance.

Original languageEnglish (US)
Title of host publicationProceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages3615-3623
Number of pages9
Volume2017-January
ISBN (Electronic)9781538604571
DOIs
StatePublished - Nov 6 2017
Event30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017 - Honolulu, United States
Duration: Jul 21 2017Jul 26 2017

Other

Other30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017
CountryUnited States
CityHonolulu
Period7/21/177/26/17

Fingerprint

Neural networks

ASJC Scopus subject areas

  • Signal Processing
  • Computer Vision and Pattern Recognition

Cite this

Xie, J., Dai, G., Zhu, F., & Fang, Y. (2017). Learning barycentric representations of 3D shapes for sketch-based 3D shape retrieval. In Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017 (Vol. 2017-January, pp. 3615-3623). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/CVPR.2017.385

Learning barycentric representations of 3D shapes for sketch-based 3D shape retrieval. / Xie, Jin; Dai, Guoxian; Zhu, Fan; Fang, Yi.

Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017. Vol. 2017-January Institute of Electrical and Electronics Engineers Inc., 2017. p. 3615-3623.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Xie, J, Dai, G, Zhu, F & Fang, Y 2017, Learning barycentric representations of 3D shapes for sketch-based 3D shape retrieval. in Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017. vol. 2017-January, Institute of Electrical and Electronics Engineers Inc., pp. 3615-3623, 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, United States, 7/21/17. https://doi.org/10.1109/CVPR.2017.385
Xie J, Dai G, Zhu F, Fang Y. Learning barycentric representations of 3D shapes for sketch-based 3D shape retrieval. In Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017. Vol. 2017-January. Institute of Electrical and Electronics Engineers Inc. 2017. p. 3615-3623 https://doi.org/10.1109/CVPR.2017.385
Xie, Jin ; Dai, Guoxian ; Zhu, Fan ; Fang, Yi. / Learning barycentric representations of 3D shapes for sketch-based 3D shape retrieval. Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017. Vol. 2017-January Institute of Electrical and Electronics Engineers Inc., 2017. pp. 3615-3623
@inproceedings{d1b225fc26e048de89d1243e03f65dd8,
title = "Learning barycentric representations of 3D shapes for sketch-based 3D shape retrieval",
abstract = "Retrieving 3D shapes with sketches is a challenging problem since 2D sketches and 3D shapes are from two heterogeneous domains, which results in large discrepancy between them. In this paper, we propose to learn barycenters of 2D projections of 3D shapes for sketch-based 3D shape retrieval. Specifically, we first use two deep convolutional neural networks (CNNs) to extract deep features of sketches and 2D projections of 3D shapes. For 3D shapes, we then compute the Wasserstein barycenters of deep features of multiple projections to form a barycentric representation. Finally, by constructing a metric network, a discriminative loss is formulated on the Wasserstein barycenters of 3D shapes and sketches in the deep feature space to learn discriminative and compact 3D shape and sketch features for retrieval. The proposed method is evaluated on the SHREC'13 and SHREC'14 sketch track benchmark datasets. Compared to the state-of-the-art methods, our proposed method can significantly improve the retrieval performance.",
author = "Jin Xie and Guoxian Dai and Fan Zhu and Yi Fang",
year = "2017",
month = "11",
day = "6",
doi = "10.1109/CVPR.2017.385",
language = "English (US)",
volume = "2017-January",
pages = "3615--3623",
booktitle = "Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017",
publisher = "Institute of Electrical and Electronics Engineers Inc.",

}

TY - GEN

T1 - Learning barycentric representations of 3D shapes for sketch-based 3D shape retrieval

AU - Xie, Jin

AU - Dai, Guoxian

AU - Zhu, Fan

AU - Fang, Yi

PY - 2017/11/6

Y1 - 2017/11/6

N2 - Retrieving 3D shapes with sketches is a challenging problem since 2D sketches and 3D shapes are from two heterogeneous domains, which results in large discrepancy between them. In this paper, we propose to learn barycenters of 2D projections of 3D shapes for sketch-based 3D shape retrieval. Specifically, we first use two deep convolutional neural networks (CNNs) to extract deep features of sketches and 2D projections of 3D shapes. For 3D shapes, we then compute the Wasserstein barycenters of deep features of multiple projections to form a barycentric representation. Finally, by constructing a metric network, a discriminative loss is formulated on the Wasserstein barycenters of 3D shapes and sketches in the deep feature space to learn discriminative and compact 3D shape and sketch features for retrieval. The proposed method is evaluated on the SHREC'13 and SHREC'14 sketch track benchmark datasets. Compared to the state-of-the-art methods, our proposed method can significantly improve the retrieval performance.

AB - Retrieving 3D shapes with sketches is a challenging problem since 2D sketches and 3D shapes are from two heterogeneous domains, which results in large discrepancy between them. In this paper, we propose to learn barycenters of 2D projections of 3D shapes for sketch-based 3D shape retrieval. Specifically, we first use two deep convolutional neural networks (CNNs) to extract deep features of sketches and 2D projections of 3D shapes. For 3D shapes, we then compute the Wasserstein barycenters of deep features of multiple projections to form a barycentric representation. Finally, by constructing a metric network, a discriminative loss is formulated on the Wasserstein barycenters of 3D shapes and sketches in the deep feature space to learn discriminative and compact 3D shape and sketch features for retrieval. The proposed method is evaluated on the SHREC'13 and SHREC'14 sketch track benchmark datasets. Compared to the state-of-the-art methods, our proposed method can significantly improve the retrieval performance.

UR - http://www.scopus.com/inward/record.url?scp=85044287997&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85044287997&partnerID=8YFLogxK

U2 - 10.1109/CVPR.2017.385

DO - 10.1109/CVPR.2017.385

M3 - Conference contribution

VL - 2017-January

SP - 3615

EP - 3623

BT - Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017

PB - Institute of Electrical and Electronics Engineers Inc.

ER -