Reading the (functional) writing on the (structural) wall

Multimodal fusion of brain structure and function via a deep neural network based translation approach reveals novel impairments in schizophrenia

Sergey M. Plis, Md Faijul Amin, Adam Chekroud, Devon Hjelm, Eswar Damaraju, Hyo Jong Lee, Juan R. Bustillo, Kyunghyun Cho, Godfrey D. Pearlson, Vince D. Calhoun

Research output: Contribution to journalArticle

Abstract

This work presents a novel approach to finding linkage/association between multimodal brain imaging data, such as structural MRI (sMRI) and functional MRI (fMRI). Motivated by the machine translation domain, we employ a deep learning model, and consider two different imaging views of the same brain like two different languages conveying some common facts. That analogy enables finding linkages between two modalities. The proposed translation-based fusion model contains a computing layer that learns “alignments” (or links) between dynamic connectivity features from fMRI data and static gray matter patterns from sMRI data. The approach is evaluated on a multi-site dataset consisting of eyes-closed resting state imaging data collected from 298 subjects (age- and gender matched 154 healthy controls and 144 patients with schizophrenia). Results are further confirmed on an independent dataset consisting of eyes-open resting state imaging data from 189 subjects (age- and gender matched 91 healthy controls and 98 patients with schizophrenia). We used dynamic functional connectivity (dFNC) states as the functional features and ICA-based sources from gray matter densities as the structural features. The dFNC states characterized by weakly correlated intrinsic connectivity networks (ICNs) were found to have stronger association with putamen and insular gray matter pattern, while the dFNC states of profuse strongly correlated ICNs exhibited stronger links with the gray matter pattern in precuneus, posterior cingulate cortex (PCC), and temporal cortex. Further investigation with the estimated link strength (or alignment score) showed significant group differences between healthy controls and patients with schizophrenia in several key regions including temporal lobe, and linked these to connectivity states showing less occupancy in healthy controls. Moreover, this novel approach revealed significant correlation between a cognitive score (attention/vigilance) and the function/structure alignment score that was not detected when data modalities were considered separately.

Original languageEnglish (US)
Pages (from-to)734-747
Number of pages14
JournalNeuroImage
Volume181
DOIs
StatePublished - Nov 1 2018

Fingerprint

Reading
Schizophrenia
Brain
Temporal Lobe
Multimodal Imaging
Magnetic Resonance Imaging
Parietal Lobe
Putamen
Gyrus Cinguli
Neuroimaging
Language
Learning
Gray Matter
Datasets

Keywords

  • Deep learning
  • Multimodal fusion
  • Psychosis
  • Schizophrenia

ASJC Scopus subject areas

  • Neurology
  • Cognitive Neuroscience

Cite this

Reading the (functional) writing on the (structural) wall : Multimodal fusion of brain structure and function via a deep neural network based translation approach reveals novel impairments in schizophrenia. / Plis, Sergey M.; Amin, Md Faijul; Chekroud, Adam; Hjelm, Devon; Damaraju, Eswar; Lee, Hyo Jong; Bustillo, Juan R.; Cho, Kyunghyun; Pearlson, Godfrey D.; Calhoun, Vince D.

In: NeuroImage, Vol. 181, 01.11.2018, p. 734-747.

Research output: Contribution to journalArticle

Plis, Sergey M. ; Amin, Md Faijul ; Chekroud, Adam ; Hjelm, Devon ; Damaraju, Eswar ; Lee, Hyo Jong ; Bustillo, Juan R. ; Cho, Kyunghyun ; Pearlson, Godfrey D. ; Calhoun, Vince D. / Reading the (functional) writing on the (structural) wall : Multimodal fusion of brain structure and function via a deep neural network based translation approach reveals novel impairments in schizophrenia. In: NeuroImage. 2018 ; Vol. 181. pp. 734-747.
@article{e7b403b0df6142b8811fa86a14327bf1,
title = "Reading the (functional) writing on the (structural) wall: Multimodal fusion of brain structure and function via a deep neural network based translation approach reveals novel impairments in schizophrenia",
abstract = "This work presents a novel approach to finding linkage/association between multimodal brain imaging data, such as structural MRI (sMRI) and functional MRI (fMRI). Motivated by the machine translation domain, we employ a deep learning model, and consider two different imaging views of the same brain like two different languages conveying some common facts. That analogy enables finding linkages between two modalities. The proposed translation-based fusion model contains a computing layer that learns “alignments” (or links) between dynamic connectivity features from fMRI data and static gray matter patterns from sMRI data. The approach is evaluated on a multi-site dataset consisting of eyes-closed resting state imaging data collected from 298 subjects (age- and gender matched 154 healthy controls and 144 patients with schizophrenia). Results are further confirmed on an independent dataset consisting of eyes-open resting state imaging data from 189 subjects (age- and gender matched 91 healthy controls and 98 patients with schizophrenia). We used dynamic functional connectivity (dFNC) states as the functional features and ICA-based sources from gray matter densities as the structural features. The dFNC states characterized by weakly correlated intrinsic connectivity networks (ICNs) were found to have stronger association with putamen and insular gray matter pattern, while the dFNC states of profuse strongly correlated ICNs exhibited stronger links with the gray matter pattern in precuneus, posterior cingulate cortex (PCC), and temporal cortex. Further investigation with the estimated link strength (or alignment score) showed significant group differences between healthy controls and patients with schizophrenia in several key regions including temporal lobe, and linked these to connectivity states showing less occupancy in healthy controls. Moreover, this novel approach revealed significant correlation between a cognitive score (attention/vigilance) and the function/structure alignment score that was not detected when data modalities were considered separately.",
keywords = "Deep learning, Multimodal fusion, Psychosis, Schizophrenia",
author = "Plis, {Sergey M.} and Amin, {Md Faijul} and Adam Chekroud and Devon Hjelm and Eswar Damaraju and Lee, {Hyo Jong} and Bustillo, {Juan R.} and Kyunghyun Cho and Pearlson, {Godfrey D.} and Calhoun, {Vince D.}",
year = "2018",
month = "11",
day = "1",
doi = "10.1016/j.neuroimage.2018.07.047",
language = "English (US)",
volume = "181",
pages = "734--747",
journal = "NeuroImage",
issn = "1053-8119",
publisher = "Academic Press Inc.",

}

TY - JOUR

T1 - Reading the (functional) writing on the (structural) wall

T2 - Multimodal fusion of brain structure and function via a deep neural network based translation approach reveals novel impairments in schizophrenia

AU - Plis, Sergey M.

AU - Amin, Md Faijul

AU - Chekroud, Adam

AU - Hjelm, Devon

AU - Damaraju, Eswar

AU - Lee, Hyo Jong

AU - Bustillo, Juan R.

AU - Cho, Kyunghyun

AU - Pearlson, Godfrey D.

AU - Calhoun, Vince D.

PY - 2018/11/1

Y1 - 2018/11/1

N2 - This work presents a novel approach to finding linkage/association between multimodal brain imaging data, such as structural MRI (sMRI) and functional MRI (fMRI). Motivated by the machine translation domain, we employ a deep learning model, and consider two different imaging views of the same brain like two different languages conveying some common facts. That analogy enables finding linkages between two modalities. The proposed translation-based fusion model contains a computing layer that learns “alignments” (or links) between dynamic connectivity features from fMRI data and static gray matter patterns from sMRI data. The approach is evaluated on a multi-site dataset consisting of eyes-closed resting state imaging data collected from 298 subjects (age- and gender matched 154 healthy controls and 144 patients with schizophrenia). Results are further confirmed on an independent dataset consisting of eyes-open resting state imaging data from 189 subjects (age- and gender matched 91 healthy controls and 98 patients with schizophrenia). We used dynamic functional connectivity (dFNC) states as the functional features and ICA-based sources from gray matter densities as the structural features. The dFNC states characterized by weakly correlated intrinsic connectivity networks (ICNs) were found to have stronger association with putamen and insular gray matter pattern, while the dFNC states of profuse strongly correlated ICNs exhibited stronger links with the gray matter pattern in precuneus, posterior cingulate cortex (PCC), and temporal cortex. Further investigation with the estimated link strength (or alignment score) showed significant group differences between healthy controls and patients with schizophrenia in several key regions including temporal lobe, and linked these to connectivity states showing less occupancy in healthy controls. Moreover, this novel approach revealed significant correlation between a cognitive score (attention/vigilance) and the function/structure alignment score that was not detected when data modalities were considered separately.

AB - This work presents a novel approach to finding linkage/association between multimodal brain imaging data, such as structural MRI (sMRI) and functional MRI (fMRI). Motivated by the machine translation domain, we employ a deep learning model, and consider two different imaging views of the same brain like two different languages conveying some common facts. That analogy enables finding linkages between two modalities. The proposed translation-based fusion model contains a computing layer that learns “alignments” (or links) between dynamic connectivity features from fMRI data and static gray matter patterns from sMRI data. The approach is evaluated on a multi-site dataset consisting of eyes-closed resting state imaging data collected from 298 subjects (age- and gender matched 154 healthy controls and 144 patients with schizophrenia). Results are further confirmed on an independent dataset consisting of eyes-open resting state imaging data from 189 subjects (age- and gender matched 91 healthy controls and 98 patients with schizophrenia). We used dynamic functional connectivity (dFNC) states as the functional features and ICA-based sources from gray matter densities as the structural features. The dFNC states characterized by weakly correlated intrinsic connectivity networks (ICNs) were found to have stronger association with putamen and insular gray matter pattern, while the dFNC states of profuse strongly correlated ICNs exhibited stronger links with the gray matter pattern in precuneus, posterior cingulate cortex (PCC), and temporal cortex. Further investigation with the estimated link strength (or alignment score) showed significant group differences between healthy controls and patients with schizophrenia in several key regions including temporal lobe, and linked these to connectivity states showing less occupancy in healthy controls. Moreover, this novel approach revealed significant correlation between a cognitive score (attention/vigilance) and the function/structure alignment score that was not detected when data modalities were considered separately.

KW - Deep learning

KW - Multimodal fusion

KW - Psychosis

KW - Schizophrenia

UR - http://www.scopus.com/inward/record.url?scp=85050870950&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85050870950&partnerID=8YFLogxK

U2 - 10.1016/j.neuroimage.2018.07.047

DO - 10.1016/j.neuroimage.2018.07.047

M3 - Article

VL - 181

SP - 734

EP - 747

JO - NeuroImage

JF - NeuroImage

SN - 1053-8119

ER -