Long-term prediction of μeCOG signals with a spatio-temporal pyramid of adversarial convolutional networks

Ran Wang, Yilin Song, Yao Wang, Jonathan Viventi

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Video prediction into sufficiently long future has many potential applications. Modeling long-term dynamics for times series is challenging with convolution neural network structure, which is usually good for capturing short-term dependencies. In this work, we propose to embed the convolutional neural network within a spatial-temporal pyramid structure, to exploit both long-term and short-term temporal dependency and capture both macro-scale and micro-scale spatial structures. The prediction at a given scale is conditioned on the features extracted from a lower scale and past observations from the current scale. In order to overcome the blurry issue caused by the mean square error loss, we add a critic model with Wasserstein distance based adversarial loss to complement MSE. We compare our spatio-temporal pyramid model against a single scale convolution network as well as a model with multiple spatial scales only, and demonstrate that our pyramid structure performs better for predicting up to 24 future frames.

Original languageEnglish (US)
Title of host publication2018 IEEE 15th International Symposium on Biomedical Imaging, ISBI 2018
PublisherIEEE Computer Society
Pages1313-1317
Number of pages5
Volume2018-April
ISBN (Electronic)9781538636367
DOIs
StatePublished - May 23 2018
Event15th IEEE International Symposium on Biomedical Imaging, ISBI 2018 - Washington, United States
Duration: Apr 4 2018Apr 7 2018

Other

Other15th IEEE International Symposium on Biomedical Imaging, ISBI 2018
CountryUnited States
CityWashington
Period4/4/184/7/18

Fingerprint

Convolution
Neural networks
Mean square error
Macros
Time series

Keywords

  • ECoG
  • Machine learning
  • Video prediction

ASJC Scopus subject areas

  • Biomedical Engineering
  • Radiology Nuclear Medicine and imaging

Cite this

Wang, R., Song, Y., Wang, Y., & Viventi, J. (2018). Long-term prediction of μeCOG signals with a spatio-temporal pyramid of adversarial convolutional networks. In 2018 IEEE 15th International Symposium on Biomedical Imaging, ISBI 2018 (Vol. 2018-April, pp. 1313-1317). IEEE Computer Society. https://doi.org/10.1109/ISBI.2018.8363813

Long-term prediction of μeCOG signals with a spatio-temporal pyramid of adversarial convolutional networks. / Wang, Ran; Song, Yilin; Wang, Yao; Viventi, Jonathan.

2018 IEEE 15th International Symposium on Biomedical Imaging, ISBI 2018. Vol. 2018-April IEEE Computer Society, 2018. p. 1313-1317.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Wang, R, Song, Y, Wang, Y & Viventi, J 2018, Long-term prediction of μeCOG signals with a spatio-temporal pyramid of adversarial convolutional networks. in 2018 IEEE 15th International Symposium on Biomedical Imaging, ISBI 2018. vol. 2018-April, IEEE Computer Society, pp. 1313-1317, 15th IEEE International Symposium on Biomedical Imaging, ISBI 2018, Washington, United States, 4/4/18. https://doi.org/10.1109/ISBI.2018.8363813
Wang R, Song Y, Wang Y, Viventi J. Long-term prediction of μeCOG signals with a spatio-temporal pyramid of adversarial convolutional networks. In 2018 IEEE 15th International Symposium on Biomedical Imaging, ISBI 2018. Vol. 2018-April. IEEE Computer Society. 2018. p. 1313-1317 https://doi.org/10.1109/ISBI.2018.8363813
Wang, Ran ; Song, Yilin ; Wang, Yao ; Viventi, Jonathan. / Long-term prediction of μeCOG signals with a spatio-temporal pyramid of adversarial convolutional networks. 2018 IEEE 15th International Symposium on Biomedical Imaging, ISBI 2018. Vol. 2018-April IEEE Computer Society, 2018. pp. 1313-1317
@inproceedings{4ec18844aa7a49f4b0e65bdcb833d2a6,
title = "Long-term prediction of μeCOG signals with a spatio-temporal pyramid of adversarial convolutional networks",
abstract = "Video prediction into sufficiently long future has many potential applications. Modeling long-term dynamics for times series is challenging with convolution neural network structure, which is usually good for capturing short-term dependencies. In this work, we propose to embed the convolutional neural network within a spatial-temporal pyramid structure, to exploit both long-term and short-term temporal dependency and capture both macro-scale and micro-scale spatial structures. The prediction at a given scale is conditioned on the features extracted from a lower scale and past observations from the current scale. In order to overcome the blurry issue caused by the mean square error loss, we add a critic model with Wasserstein distance based adversarial loss to complement MSE. We compare our spatio-temporal pyramid model against a single scale convolution network as well as a model with multiple spatial scales only, and demonstrate that our pyramid structure performs better for predicting up to 24 future frames.",
keywords = "ECoG, Machine learning, Video prediction",
author = "Ran Wang and Yilin Song and Yao Wang and Jonathan Viventi",
year = "2018",
month = "5",
day = "23",
doi = "10.1109/ISBI.2018.8363813",
language = "English (US)",
volume = "2018-April",
pages = "1313--1317",
booktitle = "2018 IEEE 15th International Symposium on Biomedical Imaging, ISBI 2018",
publisher = "IEEE Computer Society",

}

TY - GEN

T1 - Long-term prediction of μeCOG signals with a spatio-temporal pyramid of adversarial convolutional networks

AU - Wang, Ran

AU - Song, Yilin

AU - Wang, Yao

AU - Viventi, Jonathan

PY - 2018/5/23

Y1 - 2018/5/23

N2 - Video prediction into sufficiently long future has many potential applications. Modeling long-term dynamics for times series is challenging with convolution neural network structure, which is usually good for capturing short-term dependencies. In this work, we propose to embed the convolutional neural network within a spatial-temporal pyramid structure, to exploit both long-term and short-term temporal dependency and capture both macro-scale and micro-scale spatial structures. The prediction at a given scale is conditioned on the features extracted from a lower scale and past observations from the current scale. In order to overcome the blurry issue caused by the mean square error loss, we add a critic model with Wasserstein distance based adversarial loss to complement MSE. We compare our spatio-temporal pyramid model against a single scale convolution network as well as a model with multiple spatial scales only, and demonstrate that our pyramid structure performs better for predicting up to 24 future frames.

AB - Video prediction into sufficiently long future has many potential applications. Modeling long-term dynamics for times series is challenging with convolution neural network structure, which is usually good for capturing short-term dependencies. In this work, we propose to embed the convolutional neural network within a spatial-temporal pyramid structure, to exploit both long-term and short-term temporal dependency and capture both macro-scale and micro-scale spatial structures. The prediction at a given scale is conditioned on the features extracted from a lower scale and past observations from the current scale. In order to overcome the blurry issue caused by the mean square error loss, we add a critic model with Wasserstein distance based adversarial loss to complement MSE. We compare our spatio-temporal pyramid model against a single scale convolution network as well as a model with multiple spatial scales only, and demonstrate that our pyramid structure performs better for predicting up to 24 future frames.

KW - ECoG

KW - Machine learning

KW - Video prediction

UR - http://www.scopus.com/inward/record.url?scp=85048092150&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85048092150&partnerID=8YFLogxK

U2 - 10.1109/ISBI.2018.8363813

DO - 10.1109/ISBI.2018.8363813

M3 - Conference contribution

AN - SCOPUS:85048092150

VL - 2018-April

SP - 1313

EP - 1317

BT - 2018 IEEE 15th International Symposium on Biomedical Imaging, ISBI 2018

PB - IEEE Computer Society

ER -