Multimodal fusion of brain structural and functional imaging with a deep neural machine translation approach

Md Faijul Amin, Sergey M. Plis, Eswar Damaraju, Devon Hjelm, Kyunghyun Cho, Vince D. Calhoun

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

In this work, we study a novel approach of deep neural machine translation to find linkage between multimodal brain imaging data, such as structural MRI (sMRI) and functional MRI (fMRI). The idea is to consider two different imaging views of the same brain like two different languages conveying some common concepts or facts. An important aspect of the translation model is an attention network module that learns alignment between features from fMRI and sMRI. We use independent component analysis (ICA) based features for the translation model. Our study shows significant group differences between healthy controls and patients with schizophrenia in the learned alignments. Furthermore, this novel approach reveals a group differential relation between a cognitive score (attention and vigilance) and alignments that could not be found when individual modality of data were considered.

Original languageEnglish (US)
Title of host publication2016 IEEE Southwest Symposium on Image Analysis and Interpretation, SSIAI 2016 - Proceedings
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages1-4
Number of pages4
Volume2016-April
ISBN (Print)9781467399197
DOIs
StatePublished - Apr 25 2016
EventIEEE Southwest Symposium on Image Analysis and Interpretation, SSIAI 2016 - Santa Fe, United States
Duration: Mar 6 2016Mar 8 2016

Other

OtherIEEE Southwest Symposium on Image Analysis and Interpretation, SSIAI 2016
CountryUnited States
CitySanta Fe
Period3/6/163/8/16

Fingerprint

Brain
Fusion reactions
Imaging techniques
Magnetic resonance imaging
Independent component analysis
Conveying
Magnetic Resonance Imaging

ASJC Scopus subject areas

  • Software
  • Computer Vision and Pattern Recognition
  • Computer Science Applications

Cite this

Amin, M. F., Plis, S. M., Damaraju, E., Hjelm, D., Cho, K., & Calhoun, V. D. (2016). Multimodal fusion of brain structural and functional imaging with a deep neural machine translation approach. In 2016 IEEE Southwest Symposium on Image Analysis and Interpretation, SSIAI 2016 - Proceedings (Vol. 2016-April, pp. 1-4). [7459160] Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/SSIAI.2016.7459160

Multimodal fusion of brain structural and functional imaging with a deep neural machine translation approach. / Amin, Md Faijul; Plis, Sergey M.; Damaraju, Eswar; Hjelm, Devon; Cho, Kyunghyun; Calhoun, Vince D.

2016 IEEE Southwest Symposium on Image Analysis and Interpretation, SSIAI 2016 - Proceedings. Vol. 2016-April Institute of Electrical and Electronics Engineers Inc., 2016. p. 1-4 7459160.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Amin, MF, Plis, SM, Damaraju, E, Hjelm, D, Cho, K & Calhoun, VD 2016, Multimodal fusion of brain structural and functional imaging with a deep neural machine translation approach. in 2016 IEEE Southwest Symposium on Image Analysis and Interpretation, SSIAI 2016 - Proceedings. vol. 2016-April, 7459160, Institute of Electrical and Electronics Engineers Inc., pp. 1-4, IEEE Southwest Symposium on Image Analysis and Interpretation, SSIAI 2016, Santa Fe, United States, 3/6/16. https://doi.org/10.1109/SSIAI.2016.7459160
Amin MF, Plis SM, Damaraju E, Hjelm D, Cho K, Calhoun VD. Multimodal fusion of brain structural and functional imaging with a deep neural machine translation approach. In 2016 IEEE Southwest Symposium on Image Analysis and Interpretation, SSIAI 2016 - Proceedings. Vol. 2016-April. Institute of Electrical and Electronics Engineers Inc. 2016. p. 1-4. 7459160 https://doi.org/10.1109/SSIAI.2016.7459160
Amin, Md Faijul ; Plis, Sergey M. ; Damaraju, Eswar ; Hjelm, Devon ; Cho, Kyunghyun ; Calhoun, Vince D. / Multimodal fusion of brain structural and functional imaging with a deep neural machine translation approach. 2016 IEEE Southwest Symposium on Image Analysis and Interpretation, SSIAI 2016 - Proceedings. Vol. 2016-April Institute of Electrical and Electronics Engineers Inc., 2016. pp. 1-4
@inproceedings{8fca69ab1bdd4bb491a7e9620a106922,
title = "Multimodal fusion of brain structural and functional imaging with a deep neural machine translation approach",
abstract = "In this work, we study a novel approach of deep neural machine translation to find linkage between multimodal brain imaging data, such as structural MRI (sMRI) and functional MRI (fMRI). The idea is to consider two different imaging views of the same brain like two different languages conveying some common concepts or facts. An important aspect of the translation model is an attention network module that learns alignment between features from fMRI and sMRI. We use independent component analysis (ICA) based features for the translation model. Our study shows significant group differences between healthy controls and patients with schizophrenia in the learned alignments. Furthermore, this novel approach reveals a group differential relation between a cognitive score (attention and vigilance) and alignments that could not be found when individual modality of data were considered.",
author = "Amin, {Md Faijul} and Plis, {Sergey M.} and Eswar Damaraju and Devon Hjelm and Kyunghyun Cho and Calhoun, {Vince D.}",
year = "2016",
month = "4",
day = "25",
doi = "10.1109/SSIAI.2016.7459160",
language = "English (US)",
isbn = "9781467399197",
volume = "2016-April",
pages = "1--4",
booktitle = "2016 IEEE Southwest Symposium on Image Analysis and Interpretation, SSIAI 2016 - Proceedings",
publisher = "Institute of Electrical and Electronics Engineers Inc.",

}

TY - GEN

T1 - Multimodal fusion of brain structural and functional imaging with a deep neural machine translation approach

AU - Amin, Md Faijul

AU - Plis, Sergey M.

AU - Damaraju, Eswar

AU - Hjelm, Devon

AU - Cho, Kyunghyun

AU - Calhoun, Vince D.

PY - 2016/4/25

Y1 - 2016/4/25

N2 - In this work, we study a novel approach of deep neural machine translation to find linkage between multimodal brain imaging data, such as structural MRI (sMRI) and functional MRI (fMRI). The idea is to consider two different imaging views of the same brain like two different languages conveying some common concepts or facts. An important aspect of the translation model is an attention network module that learns alignment between features from fMRI and sMRI. We use independent component analysis (ICA) based features for the translation model. Our study shows significant group differences between healthy controls and patients with schizophrenia in the learned alignments. Furthermore, this novel approach reveals a group differential relation between a cognitive score (attention and vigilance) and alignments that could not be found when individual modality of data were considered.

AB - In this work, we study a novel approach of deep neural machine translation to find linkage between multimodal brain imaging data, such as structural MRI (sMRI) and functional MRI (fMRI). The idea is to consider two different imaging views of the same brain like two different languages conveying some common concepts or facts. An important aspect of the translation model is an attention network module that learns alignment between features from fMRI and sMRI. We use independent component analysis (ICA) based features for the translation model. Our study shows significant group differences between healthy controls and patients with schizophrenia in the learned alignments. Furthermore, this novel approach reveals a group differential relation between a cognitive score (attention and vigilance) and alignments that could not be found when individual modality of data were considered.

UR - http://www.scopus.com/inward/record.url?scp=84971280454&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84971280454&partnerID=8YFLogxK

U2 - 10.1109/SSIAI.2016.7459160

DO - 10.1109/SSIAI.2016.7459160

M3 - Conference contribution

SN - 9781467399197

VL - 2016-April

SP - 1

EP - 4

BT - 2016 IEEE Southwest Symposium on Image Analysis and Interpretation, SSIAI 2016 - Proceedings

PB - Institute of Electrical and Electronics Engineers Inc.

ER -