A 6-dimensional hilbert approach to index full waveform lidar data in a distributed computing environment

A. V. Vo, N. Chauhan, Debra Laefer, M. Bertolotto

Research output: Contribution to journalConference article

Abstract

Laser scanning data are increasingly available across the globe. To maximize the data's usability requires proper storage and indexing. While significant research has been invested in developing storage and indexing solutions for laser scanning point clouds (i.e. using the discrete form of the data), little attention has been paid to developing equivalent solutions for full waveform (FWF) laser scanning data, especially in a distributed computing environment. Given the growing availability of FWF sensors and datasets, FWF data management solutions are increasingly needed. This paper presents an attempt towards establishing a scalable solution for handling large FWF datasets by introducing the distributed computing solution for FWF data. The work involves a FWF database built atop HBase – the distributed database system running on Hadoop commodity clusters. By combining a 6-dimensional (6D) Hilbert spatial code and a temporal index into a compound indexing key, the database system is capable of supporting multiple spatial, temporal, and spatio-temporal queries. Such queries are important for FWF data exploration and dissemination. The proposed spatial decomposition at a fine resolution of 0.05m allows the storage of each LiDAR FWF measurement (i.e. pulse, waves, and points) on a single row of the database, thereby providing the full capabilities to add, modify, and remove each measurement record anatomically. While the feasibility and capabilities of the 6D Hilbert solution are evident, the Hilbert decomposition is not due to the complications from the combination of the data’s high dimensionality, fine resolution, and large spatial extent. These factors lead to a complex set of both attractive attributes and limitation in the proposed solution, which are described in this paper based on experimental tests using a 1.1 billion pulse LiDAR scan of a portion of Dublin, Ireland.

Original languageEnglish (US)
Pages (from-to)737-743
Number of pages7
JournalInternational Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences - ISPRS Archives
Volume42
Issue number4
DOIs
StatePublished - Sep 19 2018
EventISPRS TC IV Mid-Term Symposium on 3D Spatial Information Science - The Engine of Change - Delft, Netherlands
Duration: Oct 1 2018Oct 5 2018

Fingerprint

Distributed computer systems
Optical radar
lidar
laser
indexing
Scanning
decomposition
Lasers
data management
Distributed database systems
Decomposition
commodity
index
sensor
Information management
Ireland
Availability
Sensors
management

Keywords

  • Aerial laser scanning
  • Distributed database
  • Full waveform
  • High dimensional
  • Hilbert
  • LiDAR
  • Spatial database
  • Spatial indexing

ASJC Scopus subject areas

  • Information Systems
  • Geography, Planning and Development

Cite this

@article{d1187cffb01b499fa20c3c9f5971b9ac,
title = "A 6-dimensional hilbert approach to index full waveform lidar data in a distributed computing environment",
abstract = "Laser scanning data are increasingly available across the globe. To maximize the data's usability requires proper storage and indexing. While significant research has been invested in developing storage and indexing solutions for laser scanning point clouds (i.e. using the discrete form of the data), little attention has been paid to developing equivalent solutions for full waveform (FWF) laser scanning data, especially in a distributed computing environment. Given the growing availability of FWF sensors and datasets, FWF data management solutions are increasingly needed. This paper presents an attempt towards establishing a scalable solution for handling large FWF datasets by introducing the distributed computing solution for FWF data. The work involves a FWF database built atop HBase – the distributed database system running on Hadoop commodity clusters. By combining a 6-dimensional (6D) Hilbert spatial code and a temporal index into a compound indexing key, the database system is capable of supporting multiple spatial, temporal, and spatio-temporal queries. Such queries are important for FWF data exploration and dissemination. The proposed spatial decomposition at a fine resolution of 0.05m allows the storage of each LiDAR FWF measurement (i.e. pulse, waves, and points) on a single row of the database, thereby providing the full capabilities to add, modify, and remove each measurement record anatomically. While the feasibility and capabilities of the 6D Hilbert solution are evident, the Hilbert decomposition is not due to the complications from the combination of the data’s high dimensionality, fine resolution, and large spatial extent. These factors lead to a complex set of both attractive attributes and limitation in the proposed solution, which are described in this paper based on experimental tests using a 1.1 billion pulse LiDAR scan of a portion of Dublin, Ireland.",
keywords = "Aerial laser scanning, Distributed database, Full waveform, High dimensional, Hilbert, LiDAR, Spatial database, Spatial indexing",
author = "Vo, {A. V.} and N. Chauhan and Debra Laefer and M. Bertolotto",
year = "2018",
month = "9",
day = "19",
doi = "10.5194/isprs-archives-XLII-4-671-2018",
language = "English (US)",
volume = "42",
pages = "737--743",
journal = "International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences - ISPRS Archives",
issn = "1682-1750",
number = "4",

}

TY - JOUR

T1 - A 6-dimensional hilbert approach to index full waveform lidar data in a distributed computing environment

AU - Vo, A. V.

AU - Chauhan, N.

AU - Laefer, Debra

AU - Bertolotto, M.

PY - 2018/9/19

Y1 - 2018/9/19

N2 - Laser scanning data are increasingly available across the globe. To maximize the data's usability requires proper storage and indexing. While significant research has been invested in developing storage and indexing solutions for laser scanning point clouds (i.e. using the discrete form of the data), little attention has been paid to developing equivalent solutions for full waveform (FWF) laser scanning data, especially in a distributed computing environment. Given the growing availability of FWF sensors and datasets, FWF data management solutions are increasingly needed. This paper presents an attempt towards establishing a scalable solution for handling large FWF datasets by introducing the distributed computing solution for FWF data. The work involves a FWF database built atop HBase – the distributed database system running on Hadoop commodity clusters. By combining a 6-dimensional (6D) Hilbert spatial code and a temporal index into a compound indexing key, the database system is capable of supporting multiple spatial, temporal, and spatio-temporal queries. Such queries are important for FWF data exploration and dissemination. The proposed spatial decomposition at a fine resolution of 0.05m allows the storage of each LiDAR FWF measurement (i.e. pulse, waves, and points) on a single row of the database, thereby providing the full capabilities to add, modify, and remove each measurement record anatomically. While the feasibility and capabilities of the 6D Hilbert solution are evident, the Hilbert decomposition is not due to the complications from the combination of the data’s high dimensionality, fine resolution, and large spatial extent. These factors lead to a complex set of both attractive attributes and limitation in the proposed solution, which are described in this paper based on experimental tests using a 1.1 billion pulse LiDAR scan of a portion of Dublin, Ireland.

AB - Laser scanning data are increasingly available across the globe. To maximize the data's usability requires proper storage and indexing. While significant research has been invested in developing storage and indexing solutions for laser scanning point clouds (i.e. using the discrete form of the data), little attention has been paid to developing equivalent solutions for full waveform (FWF) laser scanning data, especially in a distributed computing environment. Given the growing availability of FWF sensors and datasets, FWF data management solutions are increasingly needed. This paper presents an attempt towards establishing a scalable solution for handling large FWF datasets by introducing the distributed computing solution for FWF data. The work involves a FWF database built atop HBase – the distributed database system running on Hadoop commodity clusters. By combining a 6-dimensional (6D) Hilbert spatial code and a temporal index into a compound indexing key, the database system is capable of supporting multiple spatial, temporal, and spatio-temporal queries. Such queries are important for FWF data exploration and dissemination. The proposed spatial decomposition at a fine resolution of 0.05m allows the storage of each LiDAR FWF measurement (i.e. pulse, waves, and points) on a single row of the database, thereby providing the full capabilities to add, modify, and remove each measurement record anatomically. While the feasibility and capabilities of the 6D Hilbert solution are evident, the Hilbert decomposition is not due to the complications from the combination of the data’s high dimensionality, fine resolution, and large spatial extent. These factors lead to a complex set of both attractive attributes and limitation in the proposed solution, which are described in this paper based on experimental tests using a 1.1 billion pulse LiDAR scan of a portion of Dublin, Ireland.

KW - Aerial laser scanning

KW - Distributed database

KW - Full waveform

KW - High dimensional

KW - Hilbert

KW - LiDAR

KW - Spatial database

KW - Spatial indexing

UR - http://www.scopus.com/inward/record.url?scp=85056181866&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85056181866&partnerID=8YFLogxK

U2 - 10.5194/isprs-archives-XLII-4-671-2018

DO - 10.5194/isprs-archives-XLII-4-671-2018

M3 - Conference article

AN - SCOPUS:85056181866

VL - 42

SP - 737

EP - 743

JO - International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences - ISPRS Archives

JF - International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences - ISPRS Archives

SN - 1682-1750

IS - 4

ER -