Variational tempering

Stephan Mandt, James McInerney, Farhan Abrol, Rajesh Ranganath, David Blei

Research output: Contribution to conferencePaper

Abstract

Variational inference (VI) combined with data subsampling enables approximate posterior inference over large data sets, but suffers from poor local optima. We first formulate a deterministic annealing approach for the generic class of conditionally conjugate exponential family models. This approach uses a decreasing temperature parameter which deterministically deforms the objective during the course of the optimization. A well-known drawback to this annealing approach is the choice of the cooling schedule. We therefore introduce variational tempering, a variational algorithm that introduces a temperature latent variable to the model. In contrast to related work in the Markov chain Monte Carlo literature, this algorithm results in adaptive annealing schedules. Lastly, we develop local variational tempering, which assigns a latent temperature to each data point; this allows for dynamic annealing that varies across data. Compared to the traditional VI, all proposed approaches find improved predictive likelihoods on held-out data.

Original languageEnglish (US)
Pages704-712
Number of pages9
StatePublished - Jan 1 2016
Event19th International Conference on Artificial Intelligence and Statistics, AISTATS 2016 - Cadiz, Spain
Duration: May 9 2016May 11 2016

Conference

Conference19th International Conference on Artificial Intelligence and Statistics, AISTATS 2016
CountrySpain
CityCadiz
Period5/9/165/11/16

Fingerprint

Tempering
Annealing
Schedule
Subsampling
Markov processes
Temperature
Exponential Family
Latent Variables
Markov Chain Monte Carlo
Large Data Sets
Assign
Cooling
Likelihood
Vary
Optimization
Model

ASJC Scopus subject areas

  • Artificial Intelligence
  • Statistics and Probability

Cite this

Mandt, S., McInerney, J., Abrol, F., Ranganath, R., & Blei, D. (2016). Variational tempering. 704-712. Paper presented at 19th International Conference on Artificial Intelligence and Statistics, AISTATS 2016, Cadiz, Spain.

Variational tempering. / Mandt, Stephan; McInerney, James; Abrol, Farhan; Ranganath, Rajesh; Blei, David.

2016. 704-712 Paper presented at 19th International Conference on Artificial Intelligence and Statistics, AISTATS 2016, Cadiz, Spain.

Research output: Contribution to conferencePaper

Mandt, S, McInerney, J, Abrol, F, Ranganath, R & Blei, D 2016, 'Variational tempering', Paper presented at 19th International Conference on Artificial Intelligence and Statistics, AISTATS 2016, Cadiz, Spain, 5/9/16 - 5/11/16 pp. 704-712.
Mandt S, McInerney J, Abrol F, Ranganath R, Blei D. Variational tempering. 2016. Paper presented at 19th International Conference on Artificial Intelligence and Statistics, AISTATS 2016, Cadiz, Spain.
Mandt, Stephan ; McInerney, James ; Abrol, Farhan ; Ranganath, Rajesh ; Blei, David. / Variational tempering. Paper presented at 19th International Conference on Artificial Intelligence and Statistics, AISTATS 2016, Cadiz, Spain.9 p.
@conference{ed586d45339243dcbf9d780cd6ada77f,
title = "Variational tempering",
abstract = "Variational inference (VI) combined with data subsampling enables approximate posterior inference over large data sets, but suffers from poor local optima. We first formulate a deterministic annealing approach for the generic class of conditionally conjugate exponential family models. This approach uses a decreasing temperature parameter which deterministically deforms the objective during the course of the optimization. A well-known drawback to this annealing approach is the choice of the cooling schedule. We therefore introduce variational tempering, a variational algorithm that introduces a temperature latent variable to the model. In contrast to related work in the Markov chain Monte Carlo literature, this algorithm results in adaptive annealing schedules. Lastly, we develop local variational tempering, which assigns a latent temperature to each data point; this allows for dynamic annealing that varies across data. Compared to the traditional VI, all proposed approaches find improved predictive likelihoods on held-out data.",
author = "Stephan Mandt and James McInerney and Farhan Abrol and Rajesh Ranganath and David Blei",
year = "2016",
month = "1",
day = "1",
language = "English (US)",
pages = "704--712",
note = "19th International Conference on Artificial Intelligence and Statistics, AISTATS 2016 ; Conference date: 09-05-2016 Through 11-05-2016",

}

TY - CONF

T1 - Variational tempering

AU - Mandt, Stephan

AU - McInerney, James

AU - Abrol, Farhan

AU - Ranganath, Rajesh

AU - Blei, David

PY - 2016/1/1

Y1 - 2016/1/1

N2 - Variational inference (VI) combined with data subsampling enables approximate posterior inference over large data sets, but suffers from poor local optima. We first formulate a deterministic annealing approach for the generic class of conditionally conjugate exponential family models. This approach uses a decreasing temperature parameter which deterministically deforms the objective during the course of the optimization. A well-known drawback to this annealing approach is the choice of the cooling schedule. We therefore introduce variational tempering, a variational algorithm that introduces a temperature latent variable to the model. In contrast to related work in the Markov chain Monte Carlo literature, this algorithm results in adaptive annealing schedules. Lastly, we develop local variational tempering, which assigns a latent temperature to each data point; this allows for dynamic annealing that varies across data. Compared to the traditional VI, all proposed approaches find improved predictive likelihoods on held-out data.

AB - Variational inference (VI) combined with data subsampling enables approximate posterior inference over large data sets, but suffers from poor local optima. We first formulate a deterministic annealing approach for the generic class of conditionally conjugate exponential family models. This approach uses a decreasing temperature parameter which deterministically deforms the objective during the course of the optimization. A well-known drawback to this annealing approach is the choice of the cooling schedule. We therefore introduce variational tempering, a variational algorithm that introduces a temperature latent variable to the model. In contrast to related work in the Markov chain Monte Carlo literature, this algorithm results in adaptive annealing schedules. Lastly, we develop local variational tempering, which assigns a latent temperature to each data point; this allows for dynamic annealing that varies across data. Compared to the traditional VI, all proposed approaches find improved predictive likelihoods on held-out data.

UR - http://www.scopus.com/inward/record.url?scp=85010209459&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85010209459&partnerID=8YFLogxK

M3 - Paper

AN - SCOPUS:85010209459

SP - 704

EP - 712

ER -