Deep Correlated Holistic Metric Learning for Sketch-Based 3D Shape Retrieval

Guoxian Dai, Jin Xie, Yi Fang

    Research output: Contribution to journalArticle

    Abstract

    How to effectively retrieve desired 3D models with simple queries is a long-standing problem in computer vision community. The model-based approach is quite straightforward but nontrivial, since people could not always have the desired 3D query model available by side. Recently, large amounts of wide-screen electronic devices are prevail in our daily lives, which makes the sketch-based 3D shape retrieval a promising candidate due to its simpleness and efficiency. The main challenge of sketch-based approach is the huge modality gap between sketch and 3D shape. In this paper, we proposed a novel deep correlated holistic metric learning (DCHML) method to mitigate the discrepancy between sketch and 3D shape domains. The proposed DCHML trains two distinct deep neural networks (one for each domain) jointly, which learns two deep nonlinear transformations to map features from both domains into a new feature space. The proposed loss, including discriminative loss and correlation loss, aims to increase the discrimination of features within each domain as well as the correlation between different domains. In the new feature space, the discriminative loss minimizes the intra-class distance of the deep transformed features and maximizes the inter-class distance of the deep transformed features to a large margin within each domain, while the correlation loss focused on mitigating the distribution discrepancy across different domains. Different from existing deep metric learning methods only with loss at the output layer, our proposed DCHML is trained with loss at both hidden layer and output layer to further improve the performance by encouraging features in the hidden layer also with desired properties. Our proposed method is evaluated on three benchmarks, including 3D Shape Retrieval Contest 2013, 2014, and 2016 benchmarks, and the experimental results demonstrate the superiority of our proposed method over the state-of-the-art methods.

    Original languageEnglish (US)
    Pages (from-to)3374-3386
    Number of pages13
    JournalIEEE Transactions on Image Processing
    Volume27
    Issue number7
    DOIs
    StatePublished - Jul 1 2018

    Fingerprint

    Computer vision
    Deep neural networks

    Keywords

    • deep correlated holistic metric learning
    • discrepancy across different domains
    • mitigate
    • Sketch-based 3D shape retrieval

    ASJC Scopus subject areas

    • Software
    • Computer Graphics and Computer-Aided Design

    Cite this

    Deep Correlated Holistic Metric Learning for Sketch-Based 3D Shape Retrieval. / Dai, Guoxian; Xie, Jin; Fang, Yi.

    In: IEEE Transactions on Image Processing, Vol. 27, No. 7, 01.07.2018, p. 3374-3386.

    Research output: Contribution to journalArticle

    Dai, Guoxian ; Xie, Jin ; Fang, Yi. / Deep Correlated Holistic Metric Learning for Sketch-Based 3D Shape Retrieval. In: IEEE Transactions on Image Processing. 2018 ; Vol. 27, No. 7. pp. 3374-3386.
    @article{1fc0a5ec368349bf9dcc25e9efecba58,
    title = "Deep Correlated Holistic Metric Learning for Sketch-Based 3D Shape Retrieval",
    abstract = "How to effectively retrieve desired 3D models with simple queries is a long-standing problem in computer vision community. The model-based approach is quite straightforward but nontrivial, since people could not always have the desired 3D query model available by side. Recently, large amounts of wide-screen electronic devices are prevail in our daily lives, which makes the sketch-based 3D shape retrieval a promising candidate due to its simpleness and efficiency. The main challenge of sketch-based approach is the huge modality gap between sketch and 3D shape. In this paper, we proposed a novel deep correlated holistic metric learning (DCHML) method to mitigate the discrepancy between sketch and 3D shape domains. The proposed DCHML trains two distinct deep neural networks (one for each domain) jointly, which learns two deep nonlinear transformations to map features from both domains into a new feature space. The proposed loss, including discriminative loss and correlation loss, aims to increase the discrimination of features within each domain as well as the correlation between different domains. In the new feature space, the discriminative loss minimizes the intra-class distance of the deep transformed features and maximizes the inter-class distance of the deep transformed features to a large margin within each domain, while the correlation loss focused on mitigating the distribution discrepancy across different domains. Different from existing deep metric learning methods only with loss at the output layer, our proposed DCHML is trained with loss at both hidden layer and output layer to further improve the performance by encouraging features in the hidden layer also with desired properties. Our proposed method is evaluated on three benchmarks, including 3D Shape Retrieval Contest 2013, 2014, and 2016 benchmarks, and the experimental results demonstrate the superiority of our proposed method over the state-of-the-art methods.",
    keywords = "deep correlated holistic metric learning, discrepancy across different domains, mitigate, Sketch-based 3D shape retrieval",
    author = "Guoxian Dai and Jin Xie and Yi Fang",
    year = "2018",
    month = "7",
    day = "1",
    doi = "10.1109/TIP.2018.2817042",
    language = "English (US)",
    volume = "27",
    pages = "3374--3386",
    journal = "IEEE Transactions on Image Processing",
    issn = "1057-7149",
    publisher = "Institute of Electrical and Electronics Engineers Inc.",
    number = "7",

    }

    TY - JOUR

    T1 - Deep Correlated Holistic Metric Learning for Sketch-Based 3D Shape Retrieval

    AU - Dai, Guoxian

    AU - Xie, Jin

    AU - Fang, Yi

    PY - 2018/7/1

    Y1 - 2018/7/1

    N2 - How to effectively retrieve desired 3D models with simple queries is a long-standing problem in computer vision community. The model-based approach is quite straightforward but nontrivial, since people could not always have the desired 3D query model available by side. Recently, large amounts of wide-screen electronic devices are prevail in our daily lives, which makes the sketch-based 3D shape retrieval a promising candidate due to its simpleness and efficiency. The main challenge of sketch-based approach is the huge modality gap between sketch and 3D shape. In this paper, we proposed a novel deep correlated holistic metric learning (DCHML) method to mitigate the discrepancy between sketch and 3D shape domains. The proposed DCHML trains two distinct deep neural networks (one for each domain) jointly, which learns two deep nonlinear transformations to map features from both domains into a new feature space. The proposed loss, including discriminative loss and correlation loss, aims to increase the discrimination of features within each domain as well as the correlation between different domains. In the new feature space, the discriminative loss minimizes the intra-class distance of the deep transformed features and maximizes the inter-class distance of the deep transformed features to a large margin within each domain, while the correlation loss focused on mitigating the distribution discrepancy across different domains. Different from existing deep metric learning methods only with loss at the output layer, our proposed DCHML is trained with loss at both hidden layer and output layer to further improve the performance by encouraging features in the hidden layer also with desired properties. Our proposed method is evaluated on three benchmarks, including 3D Shape Retrieval Contest 2013, 2014, and 2016 benchmarks, and the experimental results demonstrate the superiority of our proposed method over the state-of-the-art methods.

    AB - How to effectively retrieve desired 3D models with simple queries is a long-standing problem in computer vision community. The model-based approach is quite straightforward but nontrivial, since people could not always have the desired 3D query model available by side. Recently, large amounts of wide-screen electronic devices are prevail in our daily lives, which makes the sketch-based 3D shape retrieval a promising candidate due to its simpleness and efficiency. The main challenge of sketch-based approach is the huge modality gap between sketch and 3D shape. In this paper, we proposed a novel deep correlated holistic metric learning (DCHML) method to mitigate the discrepancy between sketch and 3D shape domains. The proposed DCHML trains two distinct deep neural networks (one for each domain) jointly, which learns two deep nonlinear transformations to map features from both domains into a new feature space. The proposed loss, including discriminative loss and correlation loss, aims to increase the discrimination of features within each domain as well as the correlation between different domains. In the new feature space, the discriminative loss minimizes the intra-class distance of the deep transformed features and maximizes the inter-class distance of the deep transformed features to a large margin within each domain, while the correlation loss focused on mitigating the distribution discrepancy across different domains. Different from existing deep metric learning methods only with loss at the output layer, our proposed DCHML is trained with loss at both hidden layer and output layer to further improve the performance by encouraging features in the hidden layer also with desired properties. Our proposed method is evaluated on three benchmarks, including 3D Shape Retrieval Contest 2013, 2014, and 2016 benchmarks, and the experimental results demonstrate the superiority of our proposed method over the state-of-the-art methods.

    KW - deep correlated holistic metric learning

    KW - discrepancy across different domains

    KW - mitigate

    KW - Sketch-based 3D shape retrieval

    UR - http://www.scopus.com/inward/record.url?scp=85044095838&partnerID=8YFLogxK

    UR - http://www.scopus.com/inward/citedby.url?scp=85044095838&partnerID=8YFLogxK

    U2 - 10.1109/TIP.2018.2817042

    DO - 10.1109/TIP.2018.2817042

    M3 - Article

    VL - 27

    SP - 3374

    EP - 3386

    JO - IEEE Transactions on Image Processing

    JF - IEEE Transactions on Image Processing

    SN - 1057-7149

    IS - 7

    ER -