Iterative neural autoregressive distribution estimator (NADE-k)

Tapani Raiko, Li Yao, Kyunghyun Cho, Yoshua Bengio

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Training of the neural autoregressive density estimator (NADE) can be viewed as doing one step of probabilistic inference on missing values in data. We propose a new model that extends this inference scheme to multiple steps, arguing that it is easier to learn to improve a reconstruction in k steps rather than to learn to reconstruct in a single inference step. The proposed model is an unsupervised building block for deep learning that combines the desirable properties of NADE and multi-prediction training: (1) Its test likelihood can be computed analytically, (2) it is easy to generate independent samples from it, and (3) it uses an inference engine that is a superset of variational inference for Boltzmann machines. The proposed NADE-k is competitive with the state-of-the-art in density estimation on the two datasets tested.

Original languageEnglish (US)
Title of host publicationAdvances in Neural Information Processing Systems
PublisherNeural information processing systems foundation
Pages325-333
Number of pages9
Volume1
EditionJanuary
StatePublished - 2014
Event28th Annual Conference on Neural Information Processing Systems 2014, NIPS 2014 - Montreal, Canada
Duration: Dec 8 2014Dec 13 2014

Other

Other28th Annual Conference on Neural Information Processing Systems 2014, NIPS 2014
CountryCanada
CityMontreal
Period12/8/1412/13/14

Fingerprint

Inference engines
Deep learning

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Information Systems
  • Signal Processing

Cite this

Raiko, T., Yao, L., Cho, K., & Bengio, Y. (2014). Iterative neural autoregressive distribution estimator (NADE-k). In Advances in Neural Information Processing Systems (January ed., Vol. 1, pp. 325-333). Neural information processing systems foundation.

Iterative neural autoregressive distribution estimator (NADE-k). / Raiko, Tapani; Yao, Li; Cho, Kyunghyun; Bengio, Yoshua.

Advances in Neural Information Processing Systems. Vol. 1 January. ed. Neural information processing systems foundation, 2014. p. 325-333.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Raiko, T, Yao, L, Cho, K & Bengio, Y 2014, Iterative neural autoregressive distribution estimator (NADE-k). in Advances in Neural Information Processing Systems. January edn, vol. 1, Neural information processing systems foundation, pp. 325-333, 28th Annual Conference on Neural Information Processing Systems 2014, NIPS 2014, Montreal, Canada, 12/8/14.
Raiko T, Yao L, Cho K, Bengio Y. Iterative neural autoregressive distribution estimator (NADE-k). In Advances in Neural Information Processing Systems. January ed. Vol. 1. Neural information processing systems foundation. 2014. p. 325-333
Raiko, Tapani ; Yao, Li ; Cho, Kyunghyun ; Bengio, Yoshua. / Iterative neural autoregressive distribution estimator (NADE-k). Advances in Neural Information Processing Systems. Vol. 1 January. ed. Neural information processing systems foundation, 2014. pp. 325-333
@inproceedings{0fe4472162a343c78601a58f81bb5796,
title = "Iterative neural autoregressive distribution estimator (NADE-k)",
abstract = "Training of the neural autoregressive density estimator (NADE) can be viewed as doing one step of probabilistic inference on missing values in data. We propose a new model that extends this inference scheme to multiple steps, arguing that it is easier to learn to improve a reconstruction in k steps rather than to learn to reconstruct in a single inference step. The proposed model is an unsupervised building block for deep learning that combines the desirable properties of NADE and multi-prediction training: (1) Its test likelihood can be computed analytically, (2) it is easy to generate independent samples from it, and (3) it uses an inference engine that is a superset of variational inference for Boltzmann machines. The proposed NADE-k is competitive with the state-of-the-art in density estimation on the two datasets tested.",
author = "Tapani Raiko and Li Yao and Kyunghyun Cho and Yoshua Bengio",
year = "2014",
language = "English (US)",
volume = "1",
pages = "325--333",
booktitle = "Advances in Neural Information Processing Systems",
publisher = "Neural information processing systems foundation",
edition = "January",

}

TY - GEN

T1 - Iterative neural autoregressive distribution estimator (NADE-k)

AU - Raiko, Tapani

AU - Yao, Li

AU - Cho, Kyunghyun

AU - Bengio, Yoshua

PY - 2014

Y1 - 2014

N2 - Training of the neural autoregressive density estimator (NADE) can be viewed as doing one step of probabilistic inference on missing values in data. We propose a new model that extends this inference scheme to multiple steps, arguing that it is easier to learn to improve a reconstruction in k steps rather than to learn to reconstruct in a single inference step. The proposed model is an unsupervised building block for deep learning that combines the desirable properties of NADE and multi-prediction training: (1) Its test likelihood can be computed analytically, (2) it is easy to generate independent samples from it, and (3) it uses an inference engine that is a superset of variational inference for Boltzmann machines. The proposed NADE-k is competitive with the state-of-the-art in density estimation on the two datasets tested.

AB - Training of the neural autoregressive density estimator (NADE) can be viewed as doing one step of probabilistic inference on missing values in data. We propose a new model that extends this inference scheme to multiple steps, arguing that it is easier to learn to improve a reconstruction in k steps rather than to learn to reconstruct in a single inference step. The proposed model is an unsupervised building block for deep learning that combines the desirable properties of NADE and multi-prediction training: (1) Its test likelihood can be computed analytically, (2) it is easy to generate independent samples from it, and (3) it uses an inference engine that is a superset of variational inference for Boltzmann machines. The proposed NADE-k is competitive with the state-of-the-art in density estimation on the two datasets tested.

UR - http://www.scopus.com/inward/record.url?scp=84937836555&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84937836555&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:84937836555

VL - 1

SP - 325

EP - 333

BT - Advances in Neural Information Processing Systems

PB - Neural information processing systems foundation

ER -