Automatic variational inference in Stan

Alp Kucukelbir, Rajesh Ranganath, Andrew Gelman, David M. Blei

Research output: Contribution to journalConference article

Abstract

Variational inference is a scalable technique for approximate Bayesian inference. Deriving variational inference algorithms requires tedious model-specific calculations; this makes it difficult for non-experts to use. We propose an automatic variational inference algorithm, automatic differentiation variational inference (ADVI); we implement it in Stan (code available), a probabilistic programming system. In ADVI the user provides a Bayesian model and a dataset, nothing else. We make no conjugacy assumptions and support a broad class of models. The algorithm automatically determines an appropriate variational family and optimizes the variational objective. We compare ADVI to MCMC sampling across hierarchical generalized linear models, nonconjugate matrix factorization, and a mixture model. We train the mixture model on a quarter million images. With ADVI we can use variational inference on any model we write in Stan.

Original languageEnglish (US)
Pages (from-to)568-576
Number of pages9
JournalAdvances in Neural Information Processing Systems
Volume2015-January
StatePublished - Jan 1 2015

Fingerprint

Computer systems programming
Factorization
Sampling

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Information Systems
  • Signal Processing

Cite this

Kucukelbir, A., Ranganath, R., Gelman, A., & Blei, D. M. (2015). Automatic variational inference in Stan. Advances in Neural Information Processing Systems, 2015-January, 568-576.

Automatic variational inference in Stan. / Kucukelbir, Alp; Ranganath, Rajesh; Gelman, Andrew; Blei, David M.

In: Advances in Neural Information Processing Systems, Vol. 2015-January, 01.01.2015, p. 568-576.

Research output: Contribution to journalConference article

Kucukelbir, A, Ranganath, R, Gelman, A & Blei, DM 2015, 'Automatic variational inference in Stan', Advances in Neural Information Processing Systems, vol. 2015-January, pp. 568-576.
Kucukelbir A, Ranganath R, Gelman A, Blei DM. Automatic variational inference in Stan. Advances in Neural Information Processing Systems. 2015 Jan 1;2015-January:568-576.
Kucukelbir, Alp ; Ranganath, Rajesh ; Gelman, Andrew ; Blei, David M. / Automatic variational inference in Stan. In: Advances in Neural Information Processing Systems. 2015 ; Vol. 2015-January. pp. 568-576.
@article{07b85b33ce8f4c16a2a9bd8cb7133003,
title = "Automatic variational inference in Stan",
abstract = "Variational inference is a scalable technique for approximate Bayesian inference. Deriving variational inference algorithms requires tedious model-specific calculations; this makes it difficult for non-experts to use. We propose an automatic variational inference algorithm, automatic differentiation variational inference (ADVI); we implement it in Stan (code available), a probabilistic programming system. In ADVI the user provides a Bayesian model and a dataset, nothing else. We make no conjugacy assumptions and support a broad class of models. The algorithm automatically determines an appropriate variational family and optimizes the variational objective. We compare ADVI to MCMC sampling across hierarchical generalized linear models, nonconjugate matrix factorization, and a mixture model. We train the mixture model on a quarter million images. With ADVI we can use variational inference on any model we write in Stan.",
author = "Alp Kucukelbir and Rajesh Ranganath and Andrew Gelman and Blei, {David M.}",
year = "2015",
month = "1",
day = "1",
language = "English (US)",
volume = "2015-January",
pages = "568--576",
journal = "Advances in Neural Information Processing Systems",
issn = "1049-5258",

}

TY - JOUR

T1 - Automatic variational inference in Stan

AU - Kucukelbir, Alp

AU - Ranganath, Rajesh

AU - Gelman, Andrew

AU - Blei, David M.

PY - 2015/1/1

Y1 - 2015/1/1

N2 - Variational inference is a scalable technique for approximate Bayesian inference. Deriving variational inference algorithms requires tedious model-specific calculations; this makes it difficult for non-experts to use. We propose an automatic variational inference algorithm, automatic differentiation variational inference (ADVI); we implement it in Stan (code available), a probabilistic programming system. In ADVI the user provides a Bayesian model and a dataset, nothing else. We make no conjugacy assumptions and support a broad class of models. The algorithm automatically determines an appropriate variational family and optimizes the variational objective. We compare ADVI to MCMC sampling across hierarchical generalized linear models, nonconjugate matrix factorization, and a mixture model. We train the mixture model on a quarter million images. With ADVI we can use variational inference on any model we write in Stan.

AB - Variational inference is a scalable technique for approximate Bayesian inference. Deriving variational inference algorithms requires tedious model-specific calculations; this makes it difficult for non-experts to use. We propose an automatic variational inference algorithm, automatic differentiation variational inference (ADVI); we implement it in Stan (code available), a probabilistic programming system. In ADVI the user provides a Bayesian model and a dataset, nothing else. We make no conjugacy assumptions and support a broad class of models. The algorithm automatically determines an appropriate variational family and optimizes the variational objective. We compare ADVI to MCMC sampling across hierarchical generalized linear models, nonconjugate matrix factorization, and a mixture model. We train the mixture model on a quarter million images. With ADVI we can use variational inference on any model we write in Stan.

UR - http://www.scopus.com/inward/record.url?scp=84965158671&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84965158671&partnerID=8YFLogxK

M3 - Conference article

AN - SCOPUS:84965158671

VL - 2015-January

SP - 568

EP - 576

JO - Advances in Neural Information Processing Systems

JF - Advances in Neural Information Processing Systems

SN - 1049-5258

ER -