Neural portraits of perception: Reconstructing face images from evoked brain activity

Alan S. Cowen, Marvin M. Chun, Brice A. Kuhl

Research output: Contribution to journalArticle

Abstract

Recent neuroimaging advances have allowed visual experience to be reconstructed from patterns of brain activity. While neural reconstructions have ranged in complexity, they have relied almost exclusively on retinotopic mappings between visual input and activity in early visual cortex. However, subjective perceptual information is tied more closely to higher-level cortical regions that have not yet been used as the primary basis for neural reconstructions. Furthermore, no reconstruction studies to date have reported reconstructions of face images, which activate a highly distributed cortical network. Thus, we investigated (a) whether individual face images could be accurately reconstructed from distributed patterns of neural activity, and (b) whether this could be achieved even when excluding activity within occipital cortex. Our approach involved four steps. (1) Principal component analysis (PCA) was used to identify components that efficiently represented a set of training faces. (2) The identified components were then mapped, using a machine learning algorithm, to fMRI activity collected during viewing of the training faces. (3) Based on activity elicited by a new set of test faces, the algorithm predicted associated component scores. (4) Finally, these scores were transformed into reconstructed images. Using both objective and subjective validation measures, we show that our methods yield strikingly accurate neural reconstructions of faces even when excluding occipital cortex. This methodology not only represents a novel and promising approach for investigating face perception, but also suggests avenues for reconstructing 'offline' visual experiences-including dreams, memories, and imagination-which are chiefly represented in higher-level cortical areas.

Original languageEnglish (US)
Pages (from-to)12-22
Number of pages11
JournalNeuroImage
Volume94
DOIs
StatePublished - Jul 1 2014

Fingerprint

Brain
Occipital Lobe
Imagination
Computer-Assisted Image Processing
Visual Cortex
Principal Component Analysis
Neuroimaging
Magnetic Resonance Imaging

ASJC Scopus subject areas

  • Cognitive Neuroscience
  • Neurology
  • Medicine(all)

Cite this

Neural portraits of perception : Reconstructing face images from evoked brain activity. / Cowen, Alan S.; Chun, Marvin M.; Kuhl, Brice A.

In: NeuroImage, Vol. 94, 01.07.2014, p. 12-22.

Research output: Contribution to journalArticle

Cowen, Alan S. ; Chun, Marvin M. ; Kuhl, Brice A. / Neural portraits of perception : Reconstructing face images from evoked brain activity. In: NeuroImage. 2014 ; Vol. 94. pp. 12-22.
@article{ff5ef5b4053e491abf605aecd877dfb7,
title = "Neural portraits of perception: Reconstructing face images from evoked brain activity",
abstract = "Recent neuroimaging advances have allowed visual experience to be reconstructed from patterns of brain activity. While neural reconstructions have ranged in complexity, they have relied almost exclusively on retinotopic mappings between visual input and activity in early visual cortex. However, subjective perceptual information is tied more closely to higher-level cortical regions that have not yet been used as the primary basis for neural reconstructions. Furthermore, no reconstruction studies to date have reported reconstructions of face images, which activate a highly distributed cortical network. Thus, we investigated (a) whether individual face images could be accurately reconstructed from distributed patterns of neural activity, and (b) whether this could be achieved even when excluding activity within occipital cortex. Our approach involved four steps. (1) Principal component analysis (PCA) was used to identify components that efficiently represented a set of training faces. (2) The identified components were then mapped, using a machine learning algorithm, to fMRI activity collected during viewing of the training faces. (3) Based on activity elicited by a new set of test faces, the algorithm predicted associated component scores. (4) Finally, these scores were transformed into reconstructed images. Using both objective and subjective validation measures, we show that our methods yield strikingly accurate neural reconstructions of faces even when excluding occipital cortex. This methodology not only represents a novel and promising approach for investigating face perception, but also suggests avenues for reconstructing 'offline' visual experiences-including dreams, memories, and imagination-which are chiefly represented in higher-level cortical areas.",
author = "Cowen, {Alan S.} and Chun, {Marvin M.} and Kuhl, {Brice A.}",
year = "2014",
month = "7",
day = "1",
doi = "10.1016/j.neuroimage.2014.03.018",
language = "English (US)",
volume = "94",
pages = "12--22",
journal = "NeuroImage",
issn = "1053-8119",
publisher = "Academic Press Inc.",

}

TY - JOUR

T1 - Neural portraits of perception

T2 - Reconstructing face images from evoked brain activity

AU - Cowen, Alan S.

AU - Chun, Marvin M.

AU - Kuhl, Brice A.

PY - 2014/7/1

Y1 - 2014/7/1

N2 - Recent neuroimaging advances have allowed visual experience to be reconstructed from patterns of brain activity. While neural reconstructions have ranged in complexity, they have relied almost exclusively on retinotopic mappings between visual input and activity in early visual cortex. However, subjective perceptual information is tied more closely to higher-level cortical regions that have not yet been used as the primary basis for neural reconstructions. Furthermore, no reconstruction studies to date have reported reconstructions of face images, which activate a highly distributed cortical network. Thus, we investigated (a) whether individual face images could be accurately reconstructed from distributed patterns of neural activity, and (b) whether this could be achieved even when excluding activity within occipital cortex. Our approach involved four steps. (1) Principal component analysis (PCA) was used to identify components that efficiently represented a set of training faces. (2) The identified components were then mapped, using a machine learning algorithm, to fMRI activity collected during viewing of the training faces. (3) Based on activity elicited by a new set of test faces, the algorithm predicted associated component scores. (4) Finally, these scores were transformed into reconstructed images. Using both objective and subjective validation measures, we show that our methods yield strikingly accurate neural reconstructions of faces even when excluding occipital cortex. This methodology not only represents a novel and promising approach for investigating face perception, but also suggests avenues for reconstructing 'offline' visual experiences-including dreams, memories, and imagination-which are chiefly represented in higher-level cortical areas.

AB - Recent neuroimaging advances have allowed visual experience to be reconstructed from patterns of brain activity. While neural reconstructions have ranged in complexity, they have relied almost exclusively on retinotopic mappings between visual input and activity in early visual cortex. However, subjective perceptual information is tied more closely to higher-level cortical regions that have not yet been used as the primary basis for neural reconstructions. Furthermore, no reconstruction studies to date have reported reconstructions of face images, which activate a highly distributed cortical network. Thus, we investigated (a) whether individual face images could be accurately reconstructed from distributed patterns of neural activity, and (b) whether this could be achieved even when excluding activity within occipital cortex. Our approach involved four steps. (1) Principal component analysis (PCA) was used to identify components that efficiently represented a set of training faces. (2) The identified components were then mapped, using a machine learning algorithm, to fMRI activity collected during viewing of the training faces. (3) Based on activity elicited by a new set of test faces, the algorithm predicted associated component scores. (4) Finally, these scores were transformed into reconstructed images. Using both objective and subjective validation measures, we show that our methods yield strikingly accurate neural reconstructions of faces even when excluding occipital cortex. This methodology not only represents a novel and promising approach for investigating face perception, but also suggests avenues for reconstructing 'offline' visual experiences-including dreams, memories, and imagination-which are chiefly represented in higher-level cortical areas.

UR - http://www.scopus.com/inward/record.url?scp=84898669850&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84898669850&partnerID=8YFLogxK

U2 - 10.1016/j.neuroimage.2014.03.018

DO - 10.1016/j.neuroimage.2014.03.018

M3 - Article

C2 - 24650597

AN - SCOPUS:84898669850

VL - 94

SP - 12

EP - 22

JO - NeuroImage

JF - NeuroImage

SN - 1053-8119

ER -