Noisin: Unbiased Regularization for Recurrent Neural Networks

Adji B. Dieng, Rajesh Ranganath, Jaan Altosaar, David M. Blei

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Recurrent neural networks (RNNS) are powerful models of sequential data. They have been successfully used in domains such as text and speech. However, RNNs are susceptible to over- fitting; regularization is important. In this paper we develop Noisin, a new method for regularizing RNNs. Noisin injects random noise into the hidden states of the RNN and then maximizes the corresponding marginal likelihood of the data. We show how Noisin applies to any RNN and we study many different types of noise. Noisin is unbiased-it preserves the underlying RNN on average. We characterize how Noisin regularizes its RNN both theoretically and empirically. On language modeling benchmarks, Noisin improves over dropout by as much as 12.2% on the Penn Treebank and 9.4% on the Wikitext-2 dataset. We also compared the state-of-the-art language model of Yang et al. 2017, both with and without Noisin. On the Penn Treebank, the method with Noisin more quickly reaches state- of-the-art performance.

Original languageEnglish (US)
Title of host publication35th International Conference on Machine Learning, ICML 2018
EditorsAndreas Krause, Jennifer Dy
PublisherInternational Machine Learning Society (IMLS)
Pages2030-2039
Number of pages10
ISBN (Electronic)9781510867963
StatePublished - Jan 1 2018
Event35th International Conference on Machine Learning, ICML 2018 - Stockholm, Sweden
Duration: Jul 10 2018Jul 15 2018

Publication series

Name35th International Conference on Machine Learning, ICML 2018
Volume3

Other

Other35th International Conference on Machine Learning, ICML 2018
CountrySweden
CityStockholm
Period7/10/187/15/18

Fingerprint

Recurrent neural networks

ASJC Scopus subject areas

  • Computational Theory and Mathematics
  • Human-Computer Interaction
  • Software

Cite this

Dieng, A. B., Ranganath, R., Altosaar, J., & Blei, D. M. (2018). Noisin: Unbiased Regularization for Recurrent Neural Networks. In A. Krause, & J. Dy (Eds.), 35th International Conference on Machine Learning, ICML 2018 (pp. 2030-2039). (35th International Conference on Machine Learning, ICML 2018; Vol. 3). International Machine Learning Society (IMLS).

Noisin : Unbiased Regularization for Recurrent Neural Networks. / Dieng, Adji B.; Ranganath, Rajesh; Altosaar, Jaan; Blei, David M.

35th International Conference on Machine Learning, ICML 2018. ed. / Andreas Krause; Jennifer Dy. International Machine Learning Society (IMLS), 2018. p. 2030-2039 (35th International Conference on Machine Learning, ICML 2018; Vol. 3).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Dieng, AB, Ranganath, R, Altosaar, J & Blei, DM 2018, Noisin: Unbiased Regularization for Recurrent Neural Networks. in A Krause & J Dy (eds), 35th International Conference on Machine Learning, ICML 2018. 35th International Conference on Machine Learning, ICML 2018, vol. 3, International Machine Learning Society (IMLS), pp. 2030-2039, 35th International Conference on Machine Learning, ICML 2018, Stockholm, Sweden, 7/10/18.
Dieng AB, Ranganath R, Altosaar J, Blei DM. Noisin: Unbiased Regularization for Recurrent Neural Networks. In Krause A, Dy J, editors, 35th International Conference on Machine Learning, ICML 2018. International Machine Learning Society (IMLS). 2018. p. 2030-2039. (35th International Conference on Machine Learning, ICML 2018).
Dieng, Adji B. ; Ranganath, Rajesh ; Altosaar, Jaan ; Blei, David M. / Noisin : Unbiased Regularization for Recurrent Neural Networks. 35th International Conference on Machine Learning, ICML 2018. editor / Andreas Krause ; Jennifer Dy. International Machine Learning Society (IMLS), 2018. pp. 2030-2039 (35th International Conference on Machine Learning, ICML 2018).
@inproceedings{d7651613f08547ddb604585edb498d4f,
title = "Noisin: Unbiased Regularization for Recurrent Neural Networks",
abstract = "Recurrent neural networks (RNNS) are powerful models of sequential data. They have been successfully used in domains such as text and speech. However, RNNs are susceptible to over- fitting; regularization is important. In this paper we develop Noisin, a new method for regularizing RNNs. Noisin injects random noise into the hidden states of the RNN and then maximizes the corresponding marginal likelihood of the data. We show how Noisin applies to any RNN and we study many different types of noise. Noisin is unbiased-it preserves the underlying RNN on average. We characterize how Noisin regularizes its RNN both theoretically and empirically. On language modeling benchmarks, Noisin improves over dropout by as much as 12.2{\%} on the Penn Treebank and 9.4{\%} on the Wikitext-2 dataset. We also compared the state-of-the-art language model of Yang et al. 2017, both with and without Noisin. On the Penn Treebank, the method with Noisin more quickly reaches state- of-the-art performance.",
author = "Dieng, {Adji B.} and Rajesh Ranganath and Jaan Altosaar and Blei, {David M.}",
year = "2018",
month = "1",
day = "1",
language = "English (US)",
series = "35th International Conference on Machine Learning, ICML 2018",
publisher = "International Machine Learning Society (IMLS)",
pages = "2030--2039",
editor = "Andreas Krause and Jennifer Dy",
booktitle = "35th International Conference on Machine Learning, ICML 2018",

}

TY - GEN

T1 - Noisin

T2 - Unbiased Regularization for Recurrent Neural Networks

AU - Dieng, Adji B.

AU - Ranganath, Rajesh

AU - Altosaar, Jaan

AU - Blei, David M.

PY - 2018/1/1

Y1 - 2018/1/1

N2 - Recurrent neural networks (RNNS) are powerful models of sequential data. They have been successfully used in domains such as text and speech. However, RNNs are susceptible to over- fitting; regularization is important. In this paper we develop Noisin, a new method for regularizing RNNs. Noisin injects random noise into the hidden states of the RNN and then maximizes the corresponding marginal likelihood of the data. We show how Noisin applies to any RNN and we study many different types of noise. Noisin is unbiased-it preserves the underlying RNN on average. We characterize how Noisin regularizes its RNN both theoretically and empirically. On language modeling benchmarks, Noisin improves over dropout by as much as 12.2% on the Penn Treebank and 9.4% on the Wikitext-2 dataset. We also compared the state-of-the-art language model of Yang et al. 2017, both with and without Noisin. On the Penn Treebank, the method with Noisin more quickly reaches state- of-the-art performance.

AB - Recurrent neural networks (RNNS) are powerful models of sequential data. They have been successfully used in domains such as text and speech. However, RNNs are susceptible to over- fitting; regularization is important. In this paper we develop Noisin, a new method for regularizing RNNs. Noisin injects random noise into the hidden states of the RNN and then maximizes the corresponding marginal likelihood of the data. We show how Noisin applies to any RNN and we study many different types of noise. Noisin is unbiased-it preserves the underlying RNN on average. We characterize how Noisin regularizes its RNN both theoretically and empirically. On language modeling benchmarks, Noisin improves over dropout by as much as 12.2% on the Penn Treebank and 9.4% on the Wikitext-2 dataset. We also compared the state-of-the-art language model of Yang et al. 2017, both with and without Noisin. On the Penn Treebank, the method with Noisin more quickly reaches state- of-the-art performance.

UR - http://www.scopus.com/inward/record.url?scp=85057259330&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85057259330&partnerID=8YFLogxK

M3 - Conference contribution

T3 - 35th International Conference on Machine Learning, ICML 2018

SP - 2030

EP - 2039

BT - 35th International Conference on Machine Learning, ICML 2018

A2 - Krause, Andreas

A2 - Dy, Jennifer

PB - International Machine Learning Society (IMLS)

ER -