Deep cross-modality adaptation via semantics preserving adversarial learning for sketch-based 3D shape retrieval

Jiaxin Chen, Yi Fang

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Due to the large cross-modality discrepancy between 2D sketches and 3D shapes, retrieving 3D shapes by sketches is a significantly challenging task. To address this problem, we propose a novel framework to learn a discriminative deep cross-modality adaptation model in this paper. Specifically, we first separately adopt two metric networks, following two deep convolutional neural networks (CNNs), to learn modality-specific discriminative features based on an importance-aware metric learning method. Subsequently, we explicitly introduce a cross-modality transformation network to compensate for the divergence between two modalities, which can transfer features of 2D sketches to the feature space of 3D shapes. We develop an adversarial learning based method to train the transformation model, by simultaneously enhancing the holistic correlations between data distributions of two modalities, and mitigating the local semantic divergences through minimizing a cross-modality mean discrepancy term. Experimental results on the SHREC 2013 and SHREC 2014 datasets clearly show the superior retrieval performance of our proposed model, compared to the state-of-the-art approaches.

Original languageEnglish (US)
Title of host publicationComputer Vision – ECCV 2018 - 15th European Conference, 2018, Proceedings
EditorsVittorio Ferrari, Cristian Sminchisescu, Yair Weiss, Martial Hebert
PublisherSpringer-Verlag
Pages624-640
Number of pages17
ISBN (Print)9783030012601
DOIs
StatePublished - Jan 1 2018
Event15th European Conference on Computer Vision, ECCV 2018 - Munich, Germany
Duration: Sep 8 2018Sep 14 2018

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume11217 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Other

Other15th European Conference on Computer Vision, ECCV 2018
CountryGermany
CityMunich
Period9/8/189/14/18

Fingerprint

3D shape
Modality
Retrieval
Semantics
Discrepancy
Divergence
Neural networks
Metric
Transformation Model
Data Distribution
Learning
Feature Space
Neural Networks
Experimental Results
Term
Model

Keywords

  • Adversarial learning
  • Cross-modality transformation
  • Importance-aware metric learning
  • Sketch-based 3D shape retrieval

ASJC Scopus subject areas

  • Theoretical Computer Science
  • Computer Science(all)

Cite this

Chen, J., & Fang, Y. (2018). Deep cross-modality adaptation via semantics preserving adversarial learning for sketch-based 3D shape retrieval. In V. Ferrari, C. Sminchisescu, Y. Weiss, & M. Hebert (Eds.), Computer Vision – ECCV 2018 - 15th European Conference, 2018, Proceedings (pp. 624-640). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11217 LNCS). Springer-Verlag. https://doi.org/10.1007/978-3-030-01261-8_37

Deep cross-modality adaptation via semantics preserving adversarial learning for sketch-based 3D shape retrieval. / Chen, Jiaxin; Fang, Yi.

Computer Vision – ECCV 2018 - 15th European Conference, 2018, Proceedings. ed. / Vittorio Ferrari; Cristian Sminchisescu; Yair Weiss; Martial Hebert. Springer-Verlag, 2018. p. 624-640 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11217 LNCS).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Chen, J & Fang, Y 2018, Deep cross-modality adaptation via semantics preserving adversarial learning for sketch-based 3D shape retrieval. in V Ferrari, C Sminchisescu, Y Weiss & M Hebert (eds), Computer Vision – ECCV 2018 - 15th European Conference, 2018, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 11217 LNCS, Springer-Verlag, pp. 624-640, 15th European Conference on Computer Vision, ECCV 2018, Munich, Germany, 9/8/18. https://doi.org/10.1007/978-3-030-01261-8_37
Chen J, Fang Y. Deep cross-modality adaptation via semantics preserving adversarial learning for sketch-based 3D shape retrieval. In Ferrari V, Sminchisescu C, Weiss Y, Hebert M, editors, Computer Vision – ECCV 2018 - 15th European Conference, 2018, Proceedings. Springer-Verlag. 2018. p. 624-640. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). https://doi.org/10.1007/978-3-030-01261-8_37
Chen, Jiaxin ; Fang, Yi. / Deep cross-modality adaptation via semantics preserving adversarial learning for sketch-based 3D shape retrieval. Computer Vision – ECCV 2018 - 15th European Conference, 2018, Proceedings. editor / Vittorio Ferrari ; Cristian Sminchisescu ; Yair Weiss ; Martial Hebert. Springer-Verlag, 2018. pp. 624-640 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).
@inproceedings{c87fa788b615470f903d0959e1188b1c,
title = "Deep cross-modality adaptation via semantics preserving adversarial learning for sketch-based 3D shape retrieval",
abstract = "Due to the large cross-modality discrepancy between 2D sketches and 3D shapes, retrieving 3D shapes by sketches is a significantly challenging task. To address this problem, we propose a novel framework to learn a discriminative deep cross-modality adaptation model in this paper. Specifically, we first separately adopt two metric networks, following two deep convolutional neural networks (CNNs), to learn modality-specific discriminative features based on an importance-aware metric learning method. Subsequently, we explicitly introduce a cross-modality transformation network to compensate for the divergence between two modalities, which can transfer features of 2D sketches to the feature space of 3D shapes. We develop an adversarial learning based method to train the transformation model, by simultaneously enhancing the holistic correlations between data distributions of two modalities, and mitigating the local semantic divergences through minimizing a cross-modality mean discrepancy term. Experimental results on the SHREC 2013 and SHREC 2014 datasets clearly show the superior retrieval performance of our proposed model, compared to the state-of-the-art approaches.",
keywords = "Adversarial learning, Cross-modality transformation, Importance-aware metric learning, Sketch-based 3D shape retrieval",
author = "Jiaxin Chen and Yi Fang",
year = "2018",
month = "1",
day = "1",
doi = "10.1007/978-3-030-01261-8_37",
language = "English (US)",
isbn = "9783030012601",
series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",
publisher = "Springer-Verlag",
pages = "624--640",
editor = "Vittorio Ferrari and Cristian Sminchisescu and Yair Weiss and Martial Hebert",
booktitle = "Computer Vision – ECCV 2018 - 15th European Conference, 2018, Proceedings",

}

TY - GEN

T1 - Deep cross-modality adaptation via semantics preserving adversarial learning for sketch-based 3D shape retrieval

AU - Chen, Jiaxin

AU - Fang, Yi

PY - 2018/1/1

Y1 - 2018/1/1

N2 - Due to the large cross-modality discrepancy between 2D sketches and 3D shapes, retrieving 3D shapes by sketches is a significantly challenging task. To address this problem, we propose a novel framework to learn a discriminative deep cross-modality adaptation model in this paper. Specifically, we first separately adopt two metric networks, following two deep convolutional neural networks (CNNs), to learn modality-specific discriminative features based on an importance-aware metric learning method. Subsequently, we explicitly introduce a cross-modality transformation network to compensate for the divergence between two modalities, which can transfer features of 2D sketches to the feature space of 3D shapes. We develop an adversarial learning based method to train the transformation model, by simultaneously enhancing the holistic correlations between data distributions of two modalities, and mitigating the local semantic divergences through minimizing a cross-modality mean discrepancy term. Experimental results on the SHREC 2013 and SHREC 2014 datasets clearly show the superior retrieval performance of our proposed model, compared to the state-of-the-art approaches.

AB - Due to the large cross-modality discrepancy between 2D sketches and 3D shapes, retrieving 3D shapes by sketches is a significantly challenging task. To address this problem, we propose a novel framework to learn a discriminative deep cross-modality adaptation model in this paper. Specifically, we first separately adopt two metric networks, following two deep convolutional neural networks (CNNs), to learn modality-specific discriminative features based on an importance-aware metric learning method. Subsequently, we explicitly introduce a cross-modality transformation network to compensate for the divergence between two modalities, which can transfer features of 2D sketches to the feature space of 3D shapes. We develop an adversarial learning based method to train the transformation model, by simultaneously enhancing the holistic correlations between data distributions of two modalities, and mitigating the local semantic divergences through minimizing a cross-modality mean discrepancy term. Experimental results on the SHREC 2013 and SHREC 2014 datasets clearly show the superior retrieval performance of our proposed model, compared to the state-of-the-art approaches.

KW - Adversarial learning

KW - Cross-modality transformation

KW - Importance-aware metric learning

KW - Sketch-based 3D shape retrieval

UR - http://www.scopus.com/inward/record.url?scp=85055589088&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85055589088&partnerID=8YFLogxK

U2 - 10.1007/978-3-030-01261-8_37

DO - 10.1007/978-3-030-01261-8_37

M3 - Conference contribution

AN - SCOPUS:85055589088

SN - 9783030012601

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 624

EP - 640

BT - Computer Vision – ECCV 2018 - 15th European Conference, 2018, Proceedings

A2 - Ferrari, Vittorio

A2 - Sminchisescu, Cristian

A2 - Weiss, Yair

A2 - Hebert, Martial

PB - Springer-Verlag

ER -