### Abstract

Several interesting generative learning algorithms involve a complex probability distribution over many random variables, involving intractable normalization constants or latent variable marginalization. Some of them may not have even an analytic expression for the unnormalized probability function and no tractable approximation. This makes it difficult to estimate the quality of these models, once they have been trained, or to monitor their quality (e.g. for early stopping) while training. A previously proposed method is based on constructing a non-parametric density estimator of the model’s probability function from samples generated by the model. We revisit this idea, propose a more efficient estimator, and prove that it provides a lower bound on the true test log-likelihood and an unbiased estimator as the number of generated samples goes to infinity, although one that incorporates the effect of poor mixing. We further propose a biased variant of the estimator that can be used reliably with a finite number of samples for the purpose of model comparison.

Original language | English (US) |
---|---|

State | Published - Jan 1 2014 |

Event | 2nd International Conference on Learning Representations, ICLR 2014 - Banff, Canada Duration: Apr 14 2014 → Apr 16 2014 |

### Conference

Conference | 2nd International Conference on Learning Representations, ICLR 2014 |
---|---|

Country | Canada |

City | Banff |

Period | 4/14/14 → 4/16/14 |

### Fingerprint

### ASJC Scopus subject areas

- Linguistics and Language
- Language and Linguistics
- Education
- Computer Science Applications

### Cite this

*Bounding the test log-likelihood of generative models*. Paper presented at 2nd International Conference on Learning Representations, ICLR 2014, Banff, Canada.

**Bounding the test log-likelihood of generative models.** / Bengio, Yoshua; Yao, Li; Cho, Kyunghyun.

Research output: Contribution to conference › Paper

}

TY - CONF

T1 - Bounding the test log-likelihood of generative models

AU - Bengio, Yoshua

AU - Yao, Li

AU - Cho, Kyunghyun

PY - 2014/1/1

Y1 - 2014/1/1

N2 - Several interesting generative learning algorithms involve a complex probability distribution over many random variables, involving intractable normalization constants or latent variable marginalization. Some of them may not have even an analytic expression for the unnormalized probability function and no tractable approximation. This makes it difficult to estimate the quality of these models, once they have been trained, or to monitor their quality (e.g. for early stopping) while training. A previously proposed method is based on constructing a non-parametric density estimator of the model’s probability function from samples generated by the model. We revisit this idea, propose a more efficient estimator, and prove that it provides a lower bound on the true test log-likelihood and an unbiased estimator as the number of generated samples goes to infinity, although one that incorporates the effect of poor mixing. We further propose a biased variant of the estimator that can be used reliably with a finite number of samples for the purpose of model comparison.

AB - Several interesting generative learning algorithms involve a complex probability distribution over many random variables, involving intractable normalization constants or latent variable marginalization. Some of them may not have even an analytic expression for the unnormalized probability function and no tractable approximation. This makes it difficult to estimate the quality of these models, once they have been trained, or to monitor their quality (e.g. for early stopping) while training. A previously proposed method is based on constructing a non-parametric density estimator of the model’s probability function from samples generated by the model. We revisit this idea, propose a more efficient estimator, and prove that it provides a lower bound on the true test log-likelihood and an unbiased estimator as the number of generated samples goes to infinity, although one that incorporates the effect of poor mixing. We further propose a biased variant of the estimator that can be used reliably with a finite number of samples for the purpose of model comparison.

UR - http://www.scopus.com/inward/record.url?scp=85030463677&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85030463677&partnerID=8YFLogxK

M3 - Paper

AN - SCOPUS:85030463677

ER -