Bounding the test log-likelihood of generative models

Yoshua Bengio, Li Yao, Kyunghyun Cho

Research output: Contribution to conferencePaper

Abstract

Several interesting generative learning algorithms involve a complex probability distribution over many random variables, involving intractable normalization constants or latent variable marginalization. Some of them may not have even an analytic expression for the unnormalized probability function and no tractable approximation. This makes it difficult to estimate the quality of these models, once they have been trained, or to monitor their quality (e.g. for early stopping) while training. A previously proposed method is based on constructing a non-parametric density estimator of the model’s probability function from samples generated by the model. We revisit this idea, propose a more efficient estimator, and prove that it provides a lower bound on the true test log-likelihood and an unbiased estimator as the number of generated samples goes to infinity, although one that incorporates the effect of poor mixing. We further propose a biased variant of the estimator that can be used reliably with a finite number of samples for the purpose of model comparison.

Original languageEnglish (US)
StatePublished - Jan 1 2014
Event2nd International Conference on Learning Representations, ICLR 2014 - Banff, Canada
Duration: Apr 14 2014Apr 16 2014

Conference

Conference2nd International Conference on Learning Representations, ICLR 2014
CountryCanada
CityBanff
Period4/14/144/16/14

Fingerprint

model comparison
normalization
Random variables
Probability distributions
Learning algorithms
learning
Generative
Infinity
Monitor
Marginalization
Approximation
Model Comparison
Normalization

ASJC Scopus subject areas

  • Linguistics and Language
  • Language and Linguistics
  • Education
  • Computer Science Applications

Cite this

Bengio, Y., Yao, L., & Cho, K. (2014). Bounding the test log-likelihood of generative models. Paper presented at 2nd International Conference on Learning Representations, ICLR 2014, Banff, Canada.

Bounding the test log-likelihood of generative models. / Bengio, Yoshua; Yao, Li; Cho, Kyunghyun.

2014. Paper presented at 2nd International Conference on Learning Representations, ICLR 2014, Banff, Canada.

Research output: Contribution to conferencePaper

Bengio, Y, Yao, L & Cho, K 2014, 'Bounding the test log-likelihood of generative models' Paper presented at 2nd International Conference on Learning Representations, ICLR 2014, Banff, Canada, 4/14/14 - 4/16/14, .
Bengio Y, Yao L, Cho K. Bounding the test log-likelihood of generative models. 2014. Paper presented at 2nd International Conference on Learning Representations, ICLR 2014, Banff, Canada.
Bengio, Yoshua ; Yao, Li ; Cho, Kyunghyun. / Bounding the test log-likelihood of generative models. Paper presented at 2nd International Conference on Learning Representations, ICLR 2014, Banff, Canada.
@conference{10f9cf996b514b12955a54991a1e74fe,
title = "Bounding the test log-likelihood of generative models",
abstract = "Several interesting generative learning algorithms involve a complex probability distribution over many random variables, involving intractable normalization constants or latent variable marginalization. Some of them may not have even an analytic expression for the unnormalized probability function and no tractable approximation. This makes it difficult to estimate the quality of these models, once they have been trained, or to monitor their quality (e.g. for early stopping) while training. A previously proposed method is based on constructing a non-parametric density estimator of the model’s probability function from samples generated by the model. We revisit this idea, propose a more efficient estimator, and prove that it provides a lower bound on the true test log-likelihood and an unbiased estimator as the number of generated samples goes to infinity, although one that incorporates the effect of poor mixing. We further propose a biased variant of the estimator that can be used reliably with a finite number of samples for the purpose of model comparison.",
author = "Yoshua Bengio and Li Yao and Kyunghyun Cho",
year = "2014",
month = "1",
day = "1",
language = "English (US)",
note = "2nd International Conference on Learning Representations, ICLR 2014 ; Conference date: 14-04-2014 Through 16-04-2014",

}

TY - CONF

T1 - Bounding the test log-likelihood of generative models

AU - Bengio, Yoshua

AU - Yao, Li

AU - Cho, Kyunghyun

PY - 2014/1/1

Y1 - 2014/1/1

N2 - Several interesting generative learning algorithms involve a complex probability distribution over many random variables, involving intractable normalization constants or latent variable marginalization. Some of them may not have even an analytic expression for the unnormalized probability function and no tractable approximation. This makes it difficult to estimate the quality of these models, once they have been trained, or to monitor their quality (e.g. for early stopping) while training. A previously proposed method is based on constructing a non-parametric density estimator of the model’s probability function from samples generated by the model. We revisit this idea, propose a more efficient estimator, and prove that it provides a lower bound on the true test log-likelihood and an unbiased estimator as the number of generated samples goes to infinity, although one that incorporates the effect of poor mixing. We further propose a biased variant of the estimator that can be used reliably with a finite number of samples for the purpose of model comparison.

AB - Several interesting generative learning algorithms involve a complex probability distribution over many random variables, involving intractable normalization constants or latent variable marginalization. Some of them may not have even an analytic expression for the unnormalized probability function and no tractable approximation. This makes it difficult to estimate the quality of these models, once they have been trained, or to monitor their quality (e.g. for early stopping) while training. A previously proposed method is based on constructing a non-parametric density estimator of the model’s probability function from samples generated by the model. We revisit this idea, propose a more efficient estimator, and prove that it provides a lower bound on the true test log-likelihood and an unbiased estimator as the number of generated samples goes to infinity, although one that incorporates the effect of poor mixing. We further propose a biased variant of the estimator that can be used reliably with a finite number of samples for the purpose of model comparison.

UR - http://www.scopus.com/inward/record.url?scp=85030463677&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85030463677&partnerID=8YFLogxK

M3 - Paper

AN - SCOPUS:85030463677

ER -