Reconstructing Speech Stimuli From Human Auditory Cortex Activity Using a WaveNet Approach

Ran Wang, Yao Wang, Adeen Flinker

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

The superior temporal gyrus (STG) region of cortex critically contributes to speech recognition. In this work, we show that a proposed WaveNet, with limited available data, is able to reconstruct speech stimuli from STG intracranial recordings. We further investigate the impulse response of the fitted model for each recording electrode and observe phoneme level temporospectral tuning properties for the recorded area of cortex. This discovery is consistent with previous studies implicating the posterior STG (pSTG) in a phonetic representation of speech and provides detailed acoustic features that certain electrode sites possibly extract during speech recognition.

Original languageEnglish (US)
Title of host publication2018 IEEE Signal Processing in Medicine and Biology Symposium, SPMB 2018 - Proceedings
PublisherInstitute of Electrical and Electronics Engineers Inc.
ISBN (Electronic)9781538659168
DOIs
StatePublished - Jan 16 2019
Event2018 IEEE Signal Processing in Medicine and Biology Symposium, SPMB 2018 - Philadelphia, United States
Duration: Dec 1 2018 → …

Publication series

Name2018 IEEE Signal Processing in Medicine and Biology Symposium, SPMB 2018 - Proceedings

Conference

Conference2018 IEEE Signal Processing in Medicine and Biology Symposium, SPMB 2018
CountryUnited States
CityPhiladelphia
Period12/1/18 → …

Fingerprint

Auditory Cortex
Speech recognition
Temporal Lobe
Electrodes
Speech analysis
Impulse response
Tuning
Acoustics
Phonetics

ASJC Scopus subject areas

  • Signal Processing
  • Health Informatics

Cite this

Wang, R., Wang, Y., & Flinker, A. (2019). Reconstructing Speech Stimuli From Human Auditory Cortex Activity Using a WaveNet Approach. In 2018 IEEE Signal Processing in Medicine and Biology Symposium, SPMB 2018 - Proceedings [8615605] (2018 IEEE Signal Processing in Medicine and Biology Symposium, SPMB 2018 - Proceedings). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/SPMB.2018.8615605

Reconstructing Speech Stimuli From Human Auditory Cortex Activity Using a WaveNet Approach. / Wang, Ran; Wang, Yao; Flinker, Adeen.

2018 IEEE Signal Processing in Medicine and Biology Symposium, SPMB 2018 - Proceedings. Institute of Electrical and Electronics Engineers Inc., 2019. 8615605 (2018 IEEE Signal Processing in Medicine and Biology Symposium, SPMB 2018 - Proceedings).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Wang, R, Wang, Y & Flinker, A 2019, Reconstructing Speech Stimuli From Human Auditory Cortex Activity Using a WaveNet Approach. in 2018 IEEE Signal Processing in Medicine and Biology Symposium, SPMB 2018 - Proceedings., 8615605, 2018 IEEE Signal Processing in Medicine and Biology Symposium, SPMB 2018 - Proceedings, Institute of Electrical and Electronics Engineers Inc., 2018 IEEE Signal Processing in Medicine and Biology Symposium, SPMB 2018, Philadelphia, United States, 12/1/18. https://doi.org/10.1109/SPMB.2018.8615605
Wang R, Wang Y, Flinker A. Reconstructing Speech Stimuli From Human Auditory Cortex Activity Using a WaveNet Approach. In 2018 IEEE Signal Processing in Medicine and Biology Symposium, SPMB 2018 - Proceedings. Institute of Electrical and Electronics Engineers Inc. 2019. 8615605. (2018 IEEE Signal Processing in Medicine and Biology Symposium, SPMB 2018 - Proceedings). https://doi.org/10.1109/SPMB.2018.8615605
Wang, Ran ; Wang, Yao ; Flinker, Adeen. / Reconstructing Speech Stimuli From Human Auditory Cortex Activity Using a WaveNet Approach. 2018 IEEE Signal Processing in Medicine and Biology Symposium, SPMB 2018 - Proceedings. Institute of Electrical and Electronics Engineers Inc., 2019. (2018 IEEE Signal Processing in Medicine and Biology Symposium, SPMB 2018 - Proceedings).
@inproceedings{8cc554d0f1a140deb123ce9074b9fe17,
title = "Reconstructing Speech Stimuli From Human Auditory Cortex Activity Using a WaveNet Approach",
abstract = "The superior temporal gyrus (STG) region of cortex critically contributes to speech recognition. In this work, we show that a proposed WaveNet, with limited available data, is able to reconstruct speech stimuli from STG intracranial recordings. We further investigate the impulse response of the fitted model for each recording electrode and observe phoneme level temporospectral tuning properties for the recorded area of cortex. This discovery is consistent with previous studies implicating the posterior STG (pSTG) in a phonetic representation of speech and provides detailed acoustic features that certain electrode sites possibly extract during speech recognition.",
author = "Ran Wang and Yao Wang and Adeen Flinker",
year = "2019",
month = "1",
day = "16",
doi = "10.1109/SPMB.2018.8615605",
language = "English (US)",
series = "2018 IEEE Signal Processing in Medicine and Biology Symposium, SPMB 2018 - Proceedings",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
booktitle = "2018 IEEE Signal Processing in Medicine and Biology Symposium, SPMB 2018 - Proceedings",

}

TY - GEN

T1 - Reconstructing Speech Stimuli From Human Auditory Cortex Activity Using a WaveNet Approach

AU - Wang, Ran

AU - Wang, Yao

AU - Flinker, Adeen

PY - 2019/1/16

Y1 - 2019/1/16

N2 - The superior temporal gyrus (STG) region of cortex critically contributes to speech recognition. In this work, we show that a proposed WaveNet, with limited available data, is able to reconstruct speech stimuli from STG intracranial recordings. We further investigate the impulse response of the fitted model for each recording electrode and observe phoneme level temporospectral tuning properties for the recorded area of cortex. This discovery is consistent with previous studies implicating the posterior STG (pSTG) in a phonetic representation of speech and provides detailed acoustic features that certain electrode sites possibly extract during speech recognition.

AB - The superior temporal gyrus (STG) region of cortex critically contributes to speech recognition. In this work, we show that a proposed WaveNet, with limited available data, is able to reconstruct speech stimuli from STG intracranial recordings. We further investigate the impulse response of the fitted model for each recording electrode and observe phoneme level temporospectral tuning properties for the recorded area of cortex. This discovery is consistent with previous studies implicating the posterior STG (pSTG) in a phonetic representation of speech and provides detailed acoustic features that certain electrode sites possibly extract during speech recognition.

UR - http://www.scopus.com/inward/record.url?scp=85062060563&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85062060563&partnerID=8YFLogxK

U2 - 10.1109/SPMB.2018.8615605

DO - 10.1109/SPMB.2018.8615605

M3 - Conference contribution

T3 - 2018 IEEE Signal Processing in Medicine and Biology Symposium, SPMB 2018 - Proceedings

BT - 2018 IEEE Signal Processing in Medicine and Biology Symposium, SPMB 2018 - Proceedings

PB - Institute of Electrical and Electronics Engineers Inc.

ER -