Proximity variational inference

Jaan Altosaar, Rajesh Ranganath, David M. Blei

Research output: Contribution to conferencePaper

Abstract

Variational inference is a powerful approach for approximate posterior inference. However, it is sensitive to initialization and can be subject to poor local optima. In this paper, we develop proximity variational inference (pvi). pvi is a new method for optimizing the variational objective that constrains subsequent iterates of the variational parameters to robustify the optimization path. Consequently, pvi is less sensitive to initialization and optimization quirks and finds better local optima. We demonstrate our method on four proximity statistics. We study pvi on a Bernoulli factor model and sigmoid belief network fit to real and synthetic data and compare to deterministic annealing (Katahira et al., 2008). We highlight the flexibility of pvi by designing a proximity statistic for Bayesian deep learning models such as the variational autoencoder (Kingma and Welling, 2014; Rezende et al., 2014) and show that it gives better performance by reducing overpruning. pvi also yields improved predictions in a deep generative model of text. Empirically, we show that pvi consistently finds better local optima and gives better predictive performance.

Original languageEnglish (US)
Pages1961-1969
Number of pages9
StatePublished - Jan 1 2018
Event21st International Conference on Artificial Intelligence and Statistics, AISTATS 2018 - Playa Blanca, Lanzarote, Canary Islands, Spain
Duration: Apr 9 2018Apr 11 2018

Conference

Conference21st International Conference on Artificial Intelligence and Statistics, AISTATS 2018
CountrySpain
CityPlaya Blanca, Lanzarote, Canary Islands
Period4/9/184/11/18

Fingerprint

Proximity
Statistics
Bayesian networks
Annealing
Initialization
Belief Networks
Bayesian Learning
Generative Models
Optimization
Factor Models
Synthetic Data
Iterate
Bernoulli
Statistic
Flexibility
Path
Prediction
Deep learning

ASJC Scopus subject areas

  • Statistics and Probability
  • Artificial Intelligence

Cite this

Altosaar, J., Ranganath, R., & Blei, D. M. (2018). Proximity variational inference. 1961-1969. Paper presented at 21st International Conference on Artificial Intelligence and Statistics, AISTATS 2018, Playa Blanca, Lanzarote, Canary Islands, Spain.

Proximity variational inference. / Altosaar, Jaan; Ranganath, Rajesh; Blei, David M.

2018. 1961-1969 Paper presented at 21st International Conference on Artificial Intelligence and Statistics, AISTATS 2018, Playa Blanca, Lanzarote, Canary Islands, Spain.

Research output: Contribution to conferencePaper

Altosaar, J, Ranganath, R & Blei, DM 2018, 'Proximity variational inference', Paper presented at 21st International Conference on Artificial Intelligence and Statistics, AISTATS 2018, Playa Blanca, Lanzarote, Canary Islands, Spain, 4/9/18 - 4/11/18 pp. 1961-1969.
Altosaar J, Ranganath R, Blei DM. Proximity variational inference. 2018. Paper presented at 21st International Conference on Artificial Intelligence and Statistics, AISTATS 2018, Playa Blanca, Lanzarote, Canary Islands, Spain.
Altosaar, Jaan ; Ranganath, Rajesh ; Blei, David M. / Proximity variational inference. Paper presented at 21st International Conference on Artificial Intelligence and Statistics, AISTATS 2018, Playa Blanca, Lanzarote, Canary Islands, Spain.9 p.
@conference{e5936ca94bc64d3da321af811ee35ab8,
title = "Proximity variational inference",
abstract = "Variational inference is a powerful approach for approximate posterior inference. However, it is sensitive to initialization and can be subject to poor local optima. In this paper, we develop proximity variational inference (pvi). pvi is a new method for optimizing the variational objective that constrains subsequent iterates of the variational parameters to robustify the optimization path. Consequently, pvi is less sensitive to initialization and optimization quirks and finds better local optima. We demonstrate our method on four proximity statistics. We study pvi on a Bernoulli factor model and sigmoid belief network fit to real and synthetic data and compare to deterministic annealing (Katahira et al., 2008). We highlight the flexibility of pvi by designing a proximity statistic for Bayesian deep learning models such as the variational autoencoder (Kingma and Welling, 2014; Rezende et al., 2014) and show that it gives better performance by reducing overpruning. pvi also yields improved predictions in a deep generative model of text. Empirically, we show that pvi consistently finds better local optima and gives better predictive performance.",
author = "Jaan Altosaar and Rajesh Ranganath and Blei, {David M.}",
year = "2018",
month = "1",
day = "1",
language = "English (US)",
pages = "1961--1969",
note = "21st International Conference on Artificial Intelligence and Statistics, AISTATS 2018 ; Conference date: 09-04-2018 Through 11-04-2018",

}

TY - CONF

T1 - Proximity variational inference

AU - Altosaar, Jaan

AU - Ranganath, Rajesh

AU - Blei, David M.

PY - 2018/1/1

Y1 - 2018/1/1

N2 - Variational inference is a powerful approach for approximate posterior inference. However, it is sensitive to initialization and can be subject to poor local optima. In this paper, we develop proximity variational inference (pvi). pvi is a new method for optimizing the variational objective that constrains subsequent iterates of the variational parameters to robustify the optimization path. Consequently, pvi is less sensitive to initialization and optimization quirks and finds better local optima. We demonstrate our method on four proximity statistics. We study pvi on a Bernoulli factor model and sigmoid belief network fit to real and synthetic data and compare to deterministic annealing (Katahira et al., 2008). We highlight the flexibility of pvi by designing a proximity statistic for Bayesian deep learning models such as the variational autoencoder (Kingma and Welling, 2014; Rezende et al., 2014) and show that it gives better performance by reducing overpruning. pvi also yields improved predictions in a deep generative model of text. Empirically, we show that pvi consistently finds better local optima and gives better predictive performance.

AB - Variational inference is a powerful approach for approximate posterior inference. However, it is sensitive to initialization and can be subject to poor local optima. In this paper, we develop proximity variational inference (pvi). pvi is a new method for optimizing the variational objective that constrains subsequent iterates of the variational parameters to robustify the optimization path. Consequently, pvi is less sensitive to initialization and optimization quirks and finds better local optima. We demonstrate our method on four proximity statistics. We study pvi on a Bernoulli factor model and sigmoid belief network fit to real and synthetic data and compare to deterministic annealing (Katahira et al., 2008). We highlight the flexibility of pvi by designing a proximity statistic for Bayesian deep learning models such as the variational autoencoder (Kingma and Welling, 2014; Rezende et al., 2014) and show that it gives better performance by reducing overpruning. pvi also yields improved predictions in a deep generative model of text. Empirically, we show that pvi consistently finds better local optima and gives better predictive performance.

UR - http://www.scopus.com/inward/record.url?scp=85067801474&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85067801474&partnerID=8YFLogxK

M3 - Paper

AN - SCOPUS:85067801474

SP - 1961

EP - 1969

ER -