Sliding-Window Temporal Attention Based Deep Learning System for Robust Sensor Modality Fusion for UGV Navigation

Halil Utku Unlu, Naman Patel, Prashanth Krishnamurthy, Farshad Khorrami

Research output: Contribution to journalArticle

Abstract

We propose a novel temporal attention based neural network architecture for robotics tasks that involve fusion of time series of sensor data, and evaluate the performance improvements in the context of autonomous navigation of unmanned ground vehicles (UGVs) in uncertain environments. The architecture generates feature vectors by fusing raw pixel and depth values collected by camera(s) and LiDAR(s), stores a history of the generated feature vectors, and incorporates the temporally attended history with current features to predict a steering command. The experimental studies show the robust performance in unknown and cluttered environments. Furthermore, the temporal attention is resilient to noise, bias, blur, and occlusions in the sensor signals. We trained the network on indoor corridor datasets (that will be publicly released) from our UGV. The datasets have LiDAR depth measurements, camera images, and human tele-operation commands.

Original languageEnglish (US)
Article number8770056
Pages (from-to)4216-4223
Number of pages8
JournalIEEE Robotics and Automation Letters
Volume4
Issue number4
DOIs
StatePublished - Oct 1 2019

Fingerprint

Unmanned Ground Vehicle
Unmanned vehicles
Ground vehicles
Sliding Window
Lidar
Learning Systems
Feature Vector
Modality
Learning systems
Navigation
Fusion
Fusion reactions
Camera
Cameras
Autonomous Navigation
Sensor
Teleoperation
Robust Performance
Sensors
Network Architecture

Keywords

  • Autonomous vehicle navigation
  • deep learning in robotics and automation
  • sensor fusion

ASJC Scopus subject areas

  • Control and Systems Engineering
  • Biomedical Engineering
  • Human-Computer Interaction
  • Mechanical Engineering
  • Computer Vision and Pattern Recognition
  • Computer Science Applications
  • Control and Optimization
  • Artificial Intelligence

Cite this

Sliding-Window Temporal Attention Based Deep Learning System for Robust Sensor Modality Fusion for UGV Navigation. / Unlu, Halil Utku; Patel, Naman; Krishnamurthy, Prashanth; Khorrami, Farshad.

In: IEEE Robotics and Automation Letters, Vol. 4, No. 4, 8770056, 01.10.2019, p. 4216-4223.

Research output: Contribution to journalArticle

@article{ca3269ef43794edd9a87456d0ab5ade2,
title = "Sliding-Window Temporal Attention Based Deep Learning System for Robust Sensor Modality Fusion for UGV Navigation",
abstract = "We propose a novel temporal attention based neural network architecture for robotics tasks that involve fusion of time series of sensor data, and evaluate the performance improvements in the context of autonomous navigation of unmanned ground vehicles (UGVs) in uncertain environments. The architecture generates feature vectors by fusing raw pixel and depth values collected by camera(s) and LiDAR(s), stores a history of the generated feature vectors, and incorporates the temporally attended history with current features to predict a steering command. The experimental studies show the robust performance in unknown and cluttered environments. Furthermore, the temporal attention is resilient to noise, bias, blur, and occlusions in the sensor signals. We trained the network on indoor corridor datasets (that will be publicly released) from our UGV. The datasets have LiDAR depth measurements, camera images, and human tele-operation commands.",
keywords = "Autonomous vehicle navigation, deep learning in robotics and automation, sensor fusion",
author = "Unlu, {Halil Utku} and Naman Patel and Prashanth Krishnamurthy and Farshad Khorrami",
year = "2019",
month = "10",
day = "1",
doi = "10.1109/LRA.2019.2930475",
language = "English (US)",
volume = "4",
pages = "4216--4223",
journal = "IEEE Robotics and Automation Letters",
issn = "2377-3766",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
number = "4",

}

TY - JOUR

T1 - Sliding-Window Temporal Attention Based Deep Learning System for Robust Sensor Modality Fusion for UGV Navigation

AU - Unlu, Halil Utku

AU - Patel, Naman

AU - Krishnamurthy, Prashanth

AU - Khorrami, Farshad

PY - 2019/10/1

Y1 - 2019/10/1

N2 - We propose a novel temporal attention based neural network architecture for robotics tasks that involve fusion of time series of sensor data, and evaluate the performance improvements in the context of autonomous navigation of unmanned ground vehicles (UGVs) in uncertain environments. The architecture generates feature vectors by fusing raw pixel and depth values collected by camera(s) and LiDAR(s), stores a history of the generated feature vectors, and incorporates the temporally attended history with current features to predict a steering command. The experimental studies show the robust performance in unknown and cluttered environments. Furthermore, the temporal attention is resilient to noise, bias, blur, and occlusions in the sensor signals. We trained the network on indoor corridor datasets (that will be publicly released) from our UGV. The datasets have LiDAR depth measurements, camera images, and human tele-operation commands.

AB - We propose a novel temporal attention based neural network architecture for robotics tasks that involve fusion of time series of sensor data, and evaluate the performance improvements in the context of autonomous navigation of unmanned ground vehicles (UGVs) in uncertain environments. The architecture generates feature vectors by fusing raw pixel and depth values collected by camera(s) and LiDAR(s), stores a history of the generated feature vectors, and incorporates the temporally attended history with current features to predict a steering command. The experimental studies show the robust performance in unknown and cluttered environments. Furthermore, the temporal attention is resilient to noise, bias, blur, and occlusions in the sensor signals. We trained the network on indoor corridor datasets (that will be publicly released) from our UGV. The datasets have LiDAR depth measurements, camera images, and human tele-operation commands.

KW - Autonomous vehicle navigation

KW - deep learning in robotics and automation

KW - sensor fusion

UR - http://www.scopus.com/inward/record.url?scp=85071443555&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85071443555&partnerID=8YFLogxK

U2 - 10.1109/LRA.2019.2930475

DO - 10.1109/LRA.2019.2930475

M3 - Article

AN - SCOPUS:85071443555

VL - 4

SP - 4216

EP - 4223

JO - IEEE Robotics and Automation Letters

JF - IEEE Robotics and Automation Letters

SN - 2377-3766

IS - 4

M1 - 8770056

ER -