Operator variational inference

Rajesh Ranganath, Jaan Altosaar, Dustin Tran, David M. Blei

Research output: Contribution to journalConference article

Abstract

Variational inference is an umbrella term for algorithms which cast Bayesian inference as optimization. Classically, variational inference uses the Kullback-Leibler divergence to define the optimization. Though this divergence has been widely used, the resultant posterior approximation can suffer from undesirable statistical properties. To address this, we reexamine variational inference from its roots as an optimization problem. We use operators, or functions of functions, to design variational objectives. As one example, we design a variational objective with a Langevin-Stein operator. We develop a black box algorithm, operator variational inference (OPVI), for optimizing any operator objective. Importantly, operators enable us to make explicit the statistical and computational tradeoffs for variational inference. We can characterize different properties of variational objectives, such as objectives that admit data subsampling-allowing inference to scale to massive data-as well as objectives that admit variational programs-a rich class of posterior approximations that does not require a tractable density. We illustrate the benefits of OPVI on a mixture model and a generative model of images.

Original languageEnglish (US)
Pages (from-to)496-504
Number of pages9
JournalAdvances in Neural Information Processing Systems
StatePublished - Jan 1 2016

Fingerprint

Mathematical operators

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Information Systems
  • Signal Processing

Cite this

Ranganath, R., Altosaar, J., Tran, D., & Blei, D. M. (2016). Operator variational inference. Advances in Neural Information Processing Systems, 496-504.

Operator variational inference. / Ranganath, Rajesh; Altosaar, Jaan; Tran, Dustin; Blei, David M.

In: Advances in Neural Information Processing Systems, 01.01.2016, p. 496-504.

Research output: Contribution to journalConference article

Ranganath, R, Altosaar, J, Tran, D & Blei, DM 2016, 'Operator variational inference', Advances in Neural Information Processing Systems, pp. 496-504.
Ranganath R, Altosaar J, Tran D, Blei DM. Operator variational inference. Advances in Neural Information Processing Systems. 2016 Jan 1;496-504.
Ranganath, Rajesh ; Altosaar, Jaan ; Tran, Dustin ; Blei, David M. / Operator variational inference. In: Advances in Neural Information Processing Systems. 2016 ; pp. 496-504.
@article{65356d0bd6de4d50a2d1cd3d60fb2209,
title = "Operator variational inference",
abstract = "Variational inference is an umbrella term for algorithms which cast Bayesian inference as optimization. Classically, variational inference uses the Kullback-Leibler divergence to define the optimization. Though this divergence has been widely used, the resultant posterior approximation can suffer from undesirable statistical properties. To address this, we reexamine variational inference from its roots as an optimization problem. We use operators, or functions of functions, to design variational objectives. As one example, we design a variational objective with a Langevin-Stein operator. We develop a black box algorithm, operator variational inference (OPVI), for optimizing any operator objective. Importantly, operators enable us to make explicit the statistical and computational tradeoffs for variational inference. We can characterize different properties of variational objectives, such as objectives that admit data subsampling-allowing inference to scale to massive data-as well as objectives that admit variational programs-a rich class of posterior approximations that does not require a tractable density. We illustrate the benefits of OPVI on a mixture model and a generative model of images.",
author = "Rajesh Ranganath and Jaan Altosaar and Dustin Tran and Blei, {David M.}",
year = "2016",
month = "1",
day = "1",
language = "English (US)",
pages = "496--504",
journal = "Advances in Neural Information Processing Systems",
issn = "1049-5258",

}

TY - JOUR

T1 - Operator variational inference

AU - Ranganath, Rajesh

AU - Altosaar, Jaan

AU - Tran, Dustin

AU - Blei, David M.

PY - 2016/1/1

Y1 - 2016/1/1

N2 - Variational inference is an umbrella term for algorithms which cast Bayesian inference as optimization. Classically, variational inference uses the Kullback-Leibler divergence to define the optimization. Though this divergence has been widely used, the resultant posterior approximation can suffer from undesirable statistical properties. To address this, we reexamine variational inference from its roots as an optimization problem. We use operators, or functions of functions, to design variational objectives. As one example, we design a variational objective with a Langevin-Stein operator. We develop a black box algorithm, operator variational inference (OPVI), for optimizing any operator objective. Importantly, operators enable us to make explicit the statistical and computational tradeoffs for variational inference. We can characterize different properties of variational objectives, such as objectives that admit data subsampling-allowing inference to scale to massive data-as well as objectives that admit variational programs-a rich class of posterior approximations that does not require a tractable density. We illustrate the benefits of OPVI on a mixture model and a generative model of images.

AB - Variational inference is an umbrella term for algorithms which cast Bayesian inference as optimization. Classically, variational inference uses the Kullback-Leibler divergence to define the optimization. Though this divergence has been widely used, the resultant posterior approximation can suffer from undesirable statistical properties. To address this, we reexamine variational inference from its roots as an optimization problem. We use operators, or functions of functions, to design variational objectives. As one example, we design a variational objective with a Langevin-Stein operator. We develop a black box algorithm, operator variational inference (OPVI), for optimizing any operator objective. Importantly, operators enable us to make explicit the statistical and computational tradeoffs for variational inference. We can characterize different properties of variational objectives, such as objectives that admit data subsampling-allowing inference to scale to massive data-as well as objectives that admit variational programs-a rich class of posterior approximations that does not require a tractable density. We illustrate the benefits of OPVI on a mixture model and a generative model of images.

UR - http://www.scopus.com/inward/record.url?scp=85019244640&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85019244640&partnerID=8YFLogxK

M3 - Conference article

SP - 496

EP - 504

JO - Advances in Neural Information Processing Systems

JF - Advances in Neural Information Processing Systems

SN - 1049-5258

ER -