Learning local descriptors with adversarial enhancer from volumetric geometry patches

Jing Zhu, Yi Fang

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Local matching problems (e.g. key point matching, geometry registration) are significant but challenging tasks in computer vision field. In this paper, we propose to learn a robust local 3D descriptor from volumetric point patches to tackle the local matching tasks. Intuitively, given two inputs, it would be easy for a network to map the inputs to a space with similar characteristics (e.g. similar outputs for similar inputs, far different outputs for far different inputs), but the difficult case for a network would be to map the inputs into a space with opposite characteristics (e.g. far different outputs for very similar inputs but very similar outputs for far different inputs). Inspired by this intuition, in our proposed method, we design a siamese-network-based local descriptor generator to learn a local descriptor with small distances between match pairs and large distances between non-match pairs. Specifically, an adversarial enhancer is introduced to map the outputs of the local descriptor generator into an opposite space that match pairs have the maximum differences and non-match pairs have the minimum differences. The local descriptor generator and the adversarial enhancer are trained in an adversarial manner. By competing with the adversarial enhancer, the local descriptor generator learns to generate a much stronger descriptor for given volumetric point patches. The experiments conducted on real-world scan datasets, including 7-scenes and SUN3D, and the synthetic scan augmented ICL-NUIM dataset show that our method can achieve superior performance over other state-of-the-art approaches on both keypoint matching and geometry registration, such as fragment alignment and scene reconstruction.

Original languageEnglish (US)
Title of host publicationMM 2018 - Proceedings of the 2018 ACM Multimedia Conference
PublisherAssociation for Computing Machinery, Inc
Pages1466-1474
Number of pages9
ISBN (Electronic)9781450356657
DOIs
StatePublished - Oct 15 2018
Event26th ACM Multimedia conference, MM 2018 - Seoul, Korea, Republic of
Duration: Oct 22 2018Oct 26 2018

Other

Other26th ACM Multimedia conference, MM 2018
CountryKorea, Republic of
CitySeoul
Period10/22/1810/26/18

Fingerprint

Geometry
Computer vision
Experiments

Keywords

  • Adversarial learning
  • Geometry registration
  • Keypoint matching

ASJC Scopus subject areas

  • Computer Graphics and Computer-Aided Design
  • Human-Computer Interaction

Cite this

Zhu, J., & Fang, Y. (2018). Learning local descriptors with adversarial enhancer from volumetric geometry patches. In MM 2018 - Proceedings of the 2018 ACM Multimedia Conference (pp. 1466-1474). Association for Computing Machinery, Inc. https://doi.org/10.1145/3240508.3240666

Learning local descriptors with adversarial enhancer from volumetric geometry patches. / Zhu, Jing; Fang, Yi.

MM 2018 - Proceedings of the 2018 ACM Multimedia Conference. Association for Computing Machinery, Inc, 2018. p. 1466-1474.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Zhu, J & Fang, Y 2018, Learning local descriptors with adversarial enhancer from volumetric geometry patches. in MM 2018 - Proceedings of the 2018 ACM Multimedia Conference. Association for Computing Machinery, Inc, pp. 1466-1474, 26th ACM Multimedia conference, MM 2018, Seoul, Korea, Republic of, 10/22/18. https://doi.org/10.1145/3240508.3240666
Zhu J, Fang Y. Learning local descriptors with adversarial enhancer from volumetric geometry patches. In MM 2018 - Proceedings of the 2018 ACM Multimedia Conference. Association for Computing Machinery, Inc. 2018. p. 1466-1474 https://doi.org/10.1145/3240508.3240666
Zhu, Jing ; Fang, Yi. / Learning local descriptors with adversarial enhancer from volumetric geometry patches. MM 2018 - Proceedings of the 2018 ACM Multimedia Conference. Association for Computing Machinery, Inc, 2018. pp. 1466-1474
@inproceedings{56b57177c1ca46c3b1a86393db8a7dd9,
title = "Learning local descriptors with adversarial enhancer from volumetric geometry patches",
abstract = "Local matching problems (e.g. key point matching, geometry registration) are significant but challenging tasks in computer vision field. In this paper, we propose to learn a robust local 3D descriptor from volumetric point patches to tackle the local matching tasks. Intuitively, given two inputs, it would be easy for a network to map the inputs to a space with similar characteristics (e.g. similar outputs for similar inputs, far different outputs for far different inputs), but the difficult case for a network would be to map the inputs into a space with opposite characteristics (e.g. far different outputs for very similar inputs but very similar outputs for far different inputs). Inspired by this intuition, in our proposed method, we design a siamese-network-based local descriptor generator to learn a local descriptor with small distances between match pairs and large distances between non-match pairs. Specifically, an adversarial enhancer is introduced to map the outputs of the local descriptor generator into an opposite space that match pairs have the maximum differences and non-match pairs have the minimum differences. The local descriptor generator and the adversarial enhancer are trained in an adversarial manner. By competing with the adversarial enhancer, the local descriptor generator learns to generate a much stronger descriptor for given volumetric point patches. The experiments conducted on real-world scan datasets, including 7-scenes and SUN3D, and the synthetic scan augmented ICL-NUIM dataset show that our method can achieve superior performance over other state-of-the-art approaches on both keypoint matching and geometry registration, such as fragment alignment and scene reconstruction.",
keywords = "Adversarial learning, Geometry registration, Keypoint matching",
author = "Jing Zhu and Yi Fang",
year = "2018",
month = "10",
day = "15",
doi = "10.1145/3240508.3240666",
language = "English (US)",
pages = "1466--1474",
booktitle = "MM 2018 - Proceedings of the 2018 ACM Multimedia Conference",
publisher = "Association for Computing Machinery, Inc",

}

TY - GEN

T1 - Learning local descriptors with adversarial enhancer from volumetric geometry patches

AU - Zhu, Jing

AU - Fang, Yi

PY - 2018/10/15

Y1 - 2018/10/15

N2 - Local matching problems (e.g. key point matching, geometry registration) are significant but challenging tasks in computer vision field. In this paper, we propose to learn a robust local 3D descriptor from volumetric point patches to tackle the local matching tasks. Intuitively, given two inputs, it would be easy for a network to map the inputs to a space with similar characteristics (e.g. similar outputs for similar inputs, far different outputs for far different inputs), but the difficult case for a network would be to map the inputs into a space with opposite characteristics (e.g. far different outputs for very similar inputs but very similar outputs for far different inputs). Inspired by this intuition, in our proposed method, we design a siamese-network-based local descriptor generator to learn a local descriptor with small distances between match pairs and large distances between non-match pairs. Specifically, an adversarial enhancer is introduced to map the outputs of the local descriptor generator into an opposite space that match pairs have the maximum differences and non-match pairs have the minimum differences. The local descriptor generator and the adversarial enhancer are trained in an adversarial manner. By competing with the adversarial enhancer, the local descriptor generator learns to generate a much stronger descriptor for given volumetric point patches. The experiments conducted on real-world scan datasets, including 7-scenes and SUN3D, and the synthetic scan augmented ICL-NUIM dataset show that our method can achieve superior performance over other state-of-the-art approaches on both keypoint matching and geometry registration, such as fragment alignment and scene reconstruction.

AB - Local matching problems (e.g. key point matching, geometry registration) are significant but challenging tasks in computer vision field. In this paper, we propose to learn a robust local 3D descriptor from volumetric point patches to tackle the local matching tasks. Intuitively, given two inputs, it would be easy for a network to map the inputs to a space with similar characteristics (e.g. similar outputs for similar inputs, far different outputs for far different inputs), but the difficult case for a network would be to map the inputs into a space with opposite characteristics (e.g. far different outputs for very similar inputs but very similar outputs for far different inputs). Inspired by this intuition, in our proposed method, we design a siamese-network-based local descriptor generator to learn a local descriptor with small distances between match pairs and large distances between non-match pairs. Specifically, an adversarial enhancer is introduced to map the outputs of the local descriptor generator into an opposite space that match pairs have the maximum differences and non-match pairs have the minimum differences. The local descriptor generator and the adversarial enhancer are trained in an adversarial manner. By competing with the adversarial enhancer, the local descriptor generator learns to generate a much stronger descriptor for given volumetric point patches. The experiments conducted on real-world scan datasets, including 7-scenes and SUN3D, and the synthetic scan augmented ICL-NUIM dataset show that our method can achieve superior performance over other state-of-the-art approaches on both keypoint matching and geometry registration, such as fragment alignment and scene reconstruction.

KW - Adversarial learning

KW - Geometry registration

KW - Keypoint matching

UR - http://www.scopus.com/inward/record.url?scp=85058242554&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85058242554&partnerID=8YFLogxK

U2 - 10.1145/3240508.3240666

DO - 10.1145/3240508.3240666

M3 - Conference contribution

SP - 1466

EP - 1474

BT - MM 2018 - Proceedings of the 2018 ACM Multimedia Conference

PB - Association for Computing Machinery, Inc

ER -