Gaussian-Bernoulli deep Boltzmann machine

Kyunghyun Cho, Tapani Raiko, Alexander Ilin

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

In this paper, we study a model that we call Gaussian-Bernoulli deep Boltzmann machine (GDBM) and discuss potential improvements in training the model. GDBM is designed to be applicable to continuous data and it is constructed from Gaussian-Bernoulli restricted Boltzmann machine (GRBM) by adding multiple layers of binary hidden neurons. The studied improvements of the learning algorithm for GDBM include parallel tempering, enhanced gradient, adaptive learning rate and layer-wise pretraining. We empirically show that they help avoid some of the common difficulties found in training deep Boltzmann machines such as divergence of learning, the difficulty in choosing right learning rate scheduling, and the existence of meaningless higher layers.

Original languageEnglish (US)
Title of host publication2013 International Joint Conference on Neural Networks, IJCNN 2013
DOIs
StatePublished - 2013
Event2013 International Joint Conference on Neural Networks, IJCNN 2013 - Dallas, TX, United States
Duration: Aug 4 2013Aug 9 2013

Other

Other2013 International Joint Conference on Neural Networks, IJCNN 2013
CountryUnited States
CityDallas, TX
Period8/4/138/9/13

Fingerprint

Tempering
Learning algorithms
Neurons
Scheduling

ASJC Scopus subject areas

  • Software
  • Artificial Intelligence

Cite this

Cho, K., Raiko, T., & Ilin, A. (2013). Gaussian-Bernoulli deep Boltzmann machine. In 2013 International Joint Conference on Neural Networks, IJCNN 2013 [6706831] https://doi.org/10.1109/IJCNN.2013.6706831

Gaussian-Bernoulli deep Boltzmann machine. / Cho, Kyunghyun; Raiko, Tapani; Ilin, Alexander.

2013 International Joint Conference on Neural Networks, IJCNN 2013. 2013. 6706831.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Cho, K, Raiko, T & Ilin, A 2013, Gaussian-Bernoulli deep Boltzmann machine. in 2013 International Joint Conference on Neural Networks, IJCNN 2013., 6706831, 2013 International Joint Conference on Neural Networks, IJCNN 2013, Dallas, TX, United States, 8/4/13. https://doi.org/10.1109/IJCNN.2013.6706831
Cho K, Raiko T, Ilin A. Gaussian-Bernoulli deep Boltzmann machine. In 2013 International Joint Conference on Neural Networks, IJCNN 2013. 2013. 6706831 https://doi.org/10.1109/IJCNN.2013.6706831
Cho, Kyunghyun ; Raiko, Tapani ; Ilin, Alexander. / Gaussian-Bernoulli deep Boltzmann machine. 2013 International Joint Conference on Neural Networks, IJCNN 2013. 2013.
@inproceedings{af2067d4e6ed4702ae10e02522138cf4,
title = "Gaussian-Bernoulli deep Boltzmann machine",
abstract = "In this paper, we study a model that we call Gaussian-Bernoulli deep Boltzmann machine (GDBM) and discuss potential improvements in training the model. GDBM is designed to be applicable to continuous data and it is constructed from Gaussian-Bernoulli restricted Boltzmann machine (GRBM) by adding multiple layers of binary hidden neurons. The studied improvements of the learning algorithm for GDBM include parallel tempering, enhanced gradient, adaptive learning rate and layer-wise pretraining. We empirically show that they help avoid some of the common difficulties found in training deep Boltzmann machines such as divergence of learning, the difficulty in choosing right learning rate scheduling, and the existence of meaningless higher layers.",
author = "Kyunghyun Cho and Tapani Raiko and Alexander Ilin",
year = "2013",
doi = "10.1109/IJCNN.2013.6706831",
language = "English (US)",
isbn = "9781467361293",
booktitle = "2013 International Joint Conference on Neural Networks, IJCNN 2013",

}

TY - GEN

T1 - Gaussian-Bernoulli deep Boltzmann machine

AU - Cho, Kyunghyun

AU - Raiko, Tapani

AU - Ilin, Alexander

PY - 2013

Y1 - 2013

N2 - In this paper, we study a model that we call Gaussian-Bernoulli deep Boltzmann machine (GDBM) and discuss potential improvements in training the model. GDBM is designed to be applicable to continuous data and it is constructed from Gaussian-Bernoulli restricted Boltzmann machine (GRBM) by adding multiple layers of binary hidden neurons. The studied improvements of the learning algorithm for GDBM include parallel tempering, enhanced gradient, adaptive learning rate and layer-wise pretraining. We empirically show that they help avoid some of the common difficulties found in training deep Boltzmann machines such as divergence of learning, the difficulty in choosing right learning rate scheduling, and the existence of meaningless higher layers.

AB - In this paper, we study a model that we call Gaussian-Bernoulli deep Boltzmann machine (GDBM) and discuss potential improvements in training the model. GDBM is designed to be applicable to continuous data and it is constructed from Gaussian-Bernoulli restricted Boltzmann machine (GRBM) by adding multiple layers of binary hidden neurons. The studied improvements of the learning algorithm for GDBM include parallel tempering, enhanced gradient, adaptive learning rate and layer-wise pretraining. We empirically show that they help avoid some of the common difficulties found in training deep Boltzmann machines such as divergence of learning, the difficulty in choosing right learning rate scheduling, and the existence of meaningless higher layers.

UR - http://www.scopus.com/inward/record.url?scp=84893549229&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84893549229&partnerID=8YFLogxK

U2 - 10.1109/IJCNN.2013.6706831

DO - 10.1109/IJCNN.2013.6706831

M3 - Conference contribution

SN - 9781467361293

BT - 2013 International Joint Conference on Neural Networks, IJCNN 2013

ER -