Gated feedback recurrent neural networks

Junyoung Chung, Caglar Gulcehre, Kyunghyun Cho, Yoshua Bengio

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

In this work, we propose a novel recurrent neural network (RNN) architecture. The proposed RNN, gated-feedback RNN (GF-RNN), extends the existing approach of stacking multiple recurrent layers by allowing and controlling signals flowing from upper recurrent layers to lower layers using a global gating unit for each pair of layers. The recurrent signals exchanged between layers are gated adaptively based on the previous hidden states and the current input. We evaluated the proposed GF-RNN with different types of recurrent units, such as tanh, long short-term memory and gated recurrent units, on the tasks of character-level language modeling and Python program evaluation. Our empirical evaluation of different RNN units, revealed that in both tasks, the GF-RNN outperforms the conventional approaches to build deep stacked RNNs. We suggest that the improvement arises because the GF-RNN can adaptively assign different layers to different timescales and layer-to-layer interactions (including the top-down ones which are not usually present in a stacked RNN) by learning to gate these interactions.

Original languageEnglish (US)
Title of host publication32nd International Conference on Machine Learning, ICML 2015
PublisherInternational Machine Learning Society (IMLS)
Pages2067-2075
Number of pages9
Volume3
ISBN (Print)9781510810587
StatePublished - 2015
Event32nd International Conference on Machine Learning, ICML 2015 - Lile, France
Duration: Jul 6 2015Jul 11 2015

Other

Other32nd International Conference on Machine Learning, ICML 2015
CountryFrance
CityLile
Period7/6/157/11/15

Fingerprint

Recurrent neural networks
Feedback
Network architecture

ASJC Scopus subject areas

  • Human-Computer Interaction
  • Computer Science Applications

Cite this

Chung, J., Gulcehre, C., Cho, K., & Bengio, Y. (2015). Gated feedback recurrent neural networks. In 32nd International Conference on Machine Learning, ICML 2015 (Vol. 3, pp. 2067-2075). International Machine Learning Society (IMLS).

Gated feedback recurrent neural networks. / Chung, Junyoung; Gulcehre, Caglar; Cho, Kyunghyun; Bengio, Yoshua.

32nd International Conference on Machine Learning, ICML 2015. Vol. 3 International Machine Learning Society (IMLS), 2015. p. 2067-2075.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Chung, J, Gulcehre, C, Cho, K & Bengio, Y 2015, Gated feedback recurrent neural networks. in 32nd International Conference on Machine Learning, ICML 2015. vol. 3, International Machine Learning Society (IMLS), pp. 2067-2075, 32nd International Conference on Machine Learning, ICML 2015, Lile, France, 7/6/15.
Chung J, Gulcehre C, Cho K, Bengio Y. Gated feedback recurrent neural networks. In 32nd International Conference on Machine Learning, ICML 2015. Vol. 3. International Machine Learning Society (IMLS). 2015. p. 2067-2075
Chung, Junyoung ; Gulcehre, Caglar ; Cho, Kyunghyun ; Bengio, Yoshua. / Gated feedback recurrent neural networks. 32nd International Conference on Machine Learning, ICML 2015. Vol. 3 International Machine Learning Society (IMLS), 2015. pp. 2067-2075
@inproceedings{46b5990c26c142c399252ad42b78874c,
title = "Gated feedback recurrent neural networks",
abstract = "In this work, we propose a novel recurrent neural network (RNN) architecture. The proposed RNN, gated-feedback RNN (GF-RNN), extends the existing approach of stacking multiple recurrent layers by allowing and controlling signals flowing from upper recurrent layers to lower layers using a global gating unit for each pair of layers. The recurrent signals exchanged between layers are gated adaptively based on the previous hidden states and the current input. We evaluated the proposed GF-RNN with different types of recurrent units, such as tanh, long short-term memory and gated recurrent units, on the tasks of character-level language modeling and Python program evaluation. Our empirical evaluation of different RNN units, revealed that in both tasks, the GF-RNN outperforms the conventional approaches to build deep stacked RNNs. We suggest that the improvement arises because the GF-RNN can adaptively assign different layers to different timescales and layer-to-layer interactions (including the top-down ones which are not usually present in a stacked RNN) by learning to gate these interactions.",
author = "Junyoung Chung and Caglar Gulcehre and Kyunghyun Cho and Yoshua Bengio",
year = "2015",
language = "English (US)",
isbn = "9781510810587",
volume = "3",
pages = "2067--2075",
booktitle = "32nd International Conference on Machine Learning, ICML 2015",
publisher = "International Machine Learning Society (IMLS)",

}

TY - GEN

T1 - Gated feedback recurrent neural networks

AU - Chung, Junyoung

AU - Gulcehre, Caglar

AU - Cho, Kyunghyun

AU - Bengio, Yoshua

PY - 2015

Y1 - 2015

N2 - In this work, we propose a novel recurrent neural network (RNN) architecture. The proposed RNN, gated-feedback RNN (GF-RNN), extends the existing approach of stacking multiple recurrent layers by allowing and controlling signals flowing from upper recurrent layers to lower layers using a global gating unit for each pair of layers. The recurrent signals exchanged between layers are gated adaptively based on the previous hidden states and the current input. We evaluated the proposed GF-RNN with different types of recurrent units, such as tanh, long short-term memory and gated recurrent units, on the tasks of character-level language modeling and Python program evaluation. Our empirical evaluation of different RNN units, revealed that in both tasks, the GF-RNN outperforms the conventional approaches to build deep stacked RNNs. We suggest that the improvement arises because the GF-RNN can adaptively assign different layers to different timescales and layer-to-layer interactions (including the top-down ones which are not usually present in a stacked RNN) by learning to gate these interactions.

AB - In this work, we propose a novel recurrent neural network (RNN) architecture. The proposed RNN, gated-feedback RNN (GF-RNN), extends the existing approach of stacking multiple recurrent layers by allowing and controlling signals flowing from upper recurrent layers to lower layers using a global gating unit for each pair of layers. The recurrent signals exchanged between layers are gated adaptively based on the previous hidden states and the current input. We evaluated the proposed GF-RNN with different types of recurrent units, such as tanh, long short-term memory and gated recurrent units, on the tasks of character-level language modeling and Python program evaluation. Our empirical evaluation of different RNN units, revealed that in both tasks, the GF-RNN outperforms the conventional approaches to build deep stacked RNNs. We suggest that the improvement arises because the GF-RNN can adaptively assign different layers to different timescales and layer-to-layer interactions (including the top-down ones which are not usually present in a stacked RNN) by learning to gate these interactions.

UR - http://www.scopus.com/inward/record.url?scp=84969972350&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84969972350&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:84969972350

SN - 9781510810587

VL - 3

SP - 2067

EP - 2075

BT - 32nd International Conference on Machine Learning, ICML 2015

PB - International Machine Learning Society (IMLS)

ER -