An adaptive learning rate for stochastic variational inference

Rajesh Ranganath, Chong Wang, David M. Blei, Eric P. Xing

Research output: Contribution to conferencePaper

Abstract

Stochastic variational inference finds good posterior approximations of probabilistic models with very large data sets. It optimizes the variational objective with stochastic optimization, following noisy estimates of the natural gradient. Operationally, stochastic inference iteratively subsamples from the data, analyzes the subsample, and updates parameters with a decreasing learning rate. However, the algorithm is sensitive to that rate, which usually requires hand-tuning to each application. We solve this problem by developing an adaptive learning rate for stochastic variational inference. Our method requires no tuning and is easily implemented with computations already made in the algorithm. We demonstrate our approach with latent Dirichlet allocation applied to three large text corpora. Inference with the adaptive learning rate converges faster and to a better approximation than the best settings of hand-tuned rates.

Original languageEnglish (US)
Pages957-965
Number of pages9
StatePublished - Jan 1 2013
Event30th International Conference on Machine Learning, ICML 2013 - Atlanta, GA, United States
Duration: Jun 16 2013Jun 21 2013

Other

Other30th International Conference on Machine Learning, ICML 2013
CountryUnited States
CityAtlanta, GA
Period6/16/136/21/13

Fingerprint

Tuning
learning
Statistical Models

ASJC Scopus subject areas

  • Human-Computer Interaction
  • Sociology and Political Science

Cite this

Ranganath, R., Wang, C., Blei, D. M., & Xing, E. P. (2013). An adaptive learning rate for stochastic variational inference. 957-965. Paper presented at 30th International Conference on Machine Learning, ICML 2013, Atlanta, GA, United States.

An adaptive learning rate for stochastic variational inference. / Ranganath, Rajesh; Wang, Chong; Blei, David M.; Xing, Eric P.

2013. 957-965 Paper presented at 30th International Conference on Machine Learning, ICML 2013, Atlanta, GA, United States.

Research output: Contribution to conferencePaper

Ranganath, R, Wang, C, Blei, DM & Xing, EP 2013, 'An adaptive learning rate for stochastic variational inference', Paper presented at 30th International Conference on Machine Learning, ICML 2013, Atlanta, GA, United States, 6/16/13 - 6/21/13 pp. 957-965.
Ranganath R, Wang C, Blei DM, Xing EP. An adaptive learning rate for stochastic variational inference. 2013. Paper presented at 30th International Conference on Machine Learning, ICML 2013, Atlanta, GA, United States.
Ranganath, Rajesh ; Wang, Chong ; Blei, David M. ; Xing, Eric P. / An adaptive learning rate for stochastic variational inference. Paper presented at 30th International Conference on Machine Learning, ICML 2013, Atlanta, GA, United States.9 p.
@conference{365b5fdecc124d40a162c9529a47a023,
title = "An adaptive learning rate for stochastic variational inference",
abstract = "Stochastic variational inference finds good posterior approximations of probabilistic models with very large data sets. It optimizes the variational objective with stochastic optimization, following noisy estimates of the natural gradient. Operationally, stochastic inference iteratively subsamples from the data, analyzes the subsample, and updates parameters with a decreasing learning rate. However, the algorithm is sensitive to that rate, which usually requires hand-tuning to each application. We solve this problem by developing an adaptive learning rate for stochastic variational inference. Our method requires no tuning and is easily implemented with computations already made in the algorithm. We demonstrate our approach with latent Dirichlet allocation applied to three large text corpora. Inference with the adaptive learning rate converges faster and to a better approximation than the best settings of hand-tuned rates.",
author = "Rajesh Ranganath and Chong Wang and Blei, {David M.} and Xing, {Eric P.}",
year = "2013",
month = "1",
day = "1",
language = "English (US)",
pages = "957--965",
note = "30th International Conference on Machine Learning, ICML 2013 ; Conference date: 16-06-2013 Through 21-06-2013",

}

TY - CONF

T1 - An adaptive learning rate for stochastic variational inference

AU - Ranganath, Rajesh

AU - Wang, Chong

AU - Blei, David M.

AU - Xing, Eric P.

PY - 2013/1/1

Y1 - 2013/1/1

N2 - Stochastic variational inference finds good posterior approximations of probabilistic models with very large data sets. It optimizes the variational objective with stochastic optimization, following noisy estimates of the natural gradient. Operationally, stochastic inference iteratively subsamples from the data, analyzes the subsample, and updates parameters with a decreasing learning rate. However, the algorithm is sensitive to that rate, which usually requires hand-tuning to each application. We solve this problem by developing an adaptive learning rate for stochastic variational inference. Our method requires no tuning and is easily implemented with computations already made in the algorithm. We demonstrate our approach with latent Dirichlet allocation applied to three large text corpora. Inference with the adaptive learning rate converges faster and to a better approximation than the best settings of hand-tuned rates.

AB - Stochastic variational inference finds good posterior approximations of probabilistic models with very large data sets. It optimizes the variational objective with stochastic optimization, following noisy estimates of the natural gradient. Operationally, stochastic inference iteratively subsamples from the data, analyzes the subsample, and updates parameters with a decreasing learning rate. However, the algorithm is sensitive to that rate, which usually requires hand-tuning to each application. We solve this problem by developing an adaptive learning rate for stochastic variational inference. Our method requires no tuning and is easily implemented with computations already made in the algorithm. We demonstrate our approach with latent Dirichlet allocation applied to three large text corpora. Inference with the adaptive learning rate converges faster and to a better approximation than the best settings of hand-tuned rates.

UR - http://www.scopus.com/inward/record.url?scp=84897505299&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84897505299&partnerID=8YFLogxK

M3 - Paper

AN - SCOPUS:84897505299

SP - 957

EP - 965

ER -