Improved learning of Gaussian-Bernoulli restricted Boltzmann machines

Kyunghyun Cho, Alexander Ilin, Tapani Raiko

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

We propose a few remedies to improve training of Gaussian-Bernoulli restricted Boltzmann machines (GBRBM), which is known to be difficult. Firstly, we use a different parameterization of the energy function, which allows for more intuitive interpretation of the parameters and facilitates learning. Secondly, we propose parallel tempering learning for GBRBM. Lastly, we use an adaptive learning rate which is selected automatically in order to stabilize training. Our extensive experiments show that the proposed improvements indeed remove most of the difficulties encountered when training GBRBMs using conventional methods.

Original languageEnglish (US)
Title of host publicationArtificial Neural Networks and Machine Learning, ICANN 2011 - 21st International Conference on Artificial Neural Networks, Proceedings
Pages10-17
Number of pages8
Volume6791 LNCS
EditionPART 1
DOIs
StatePublished - 2011
Event21st International Conference on Artificial Neural Networks, ICANN 2011 - Espoo, Finland
Duration: Jun 14 2011Jun 17 2011

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
NumberPART 1
Volume6791 LNCS
ISSN (Print)03029743
ISSN (Electronic)16113349

Other

Other21st International Conference on Artificial Neural Networks, ICANN 2011
CountryFinland
CityEspoo
Period6/14/116/17/11

Fingerprint

Boltzmann Machine
Tempering
Parameterization
Bernoulli
Parallel Tempering
Adaptive Learning
Learning Rate
Experiments
Energy Function
Intuitive
Experiment
Learning
Training

Keywords

  • Adaptive Learning Rate
  • Gaussian-Bernoulli Restricted Boltzmann Machine
  • Parallel Tempering
  • Restricted Boltzmann Machine

ASJC Scopus subject areas

  • Computer Science(all)
  • Theoretical Computer Science

Cite this

Cho, K., Ilin, A., & Raiko, T. (2011). Improved learning of Gaussian-Bernoulli restricted Boltzmann machines. In Artificial Neural Networks and Machine Learning, ICANN 2011 - 21st International Conference on Artificial Neural Networks, Proceedings (PART 1 ed., Vol. 6791 LNCS, pp. 10-17). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 6791 LNCS, No. PART 1). https://doi.org/10.1007/978-3-642-21735-7_2

Improved learning of Gaussian-Bernoulli restricted Boltzmann machines. / Cho, Kyunghyun; Ilin, Alexander; Raiko, Tapani.

Artificial Neural Networks and Machine Learning, ICANN 2011 - 21st International Conference on Artificial Neural Networks, Proceedings. Vol. 6791 LNCS PART 1. ed. 2011. p. 10-17 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 6791 LNCS, No. PART 1).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Cho, K, Ilin, A & Raiko, T 2011, Improved learning of Gaussian-Bernoulli restricted Boltzmann machines. in Artificial Neural Networks and Machine Learning, ICANN 2011 - 21st International Conference on Artificial Neural Networks, Proceedings. PART 1 edn, vol. 6791 LNCS, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), no. PART 1, vol. 6791 LNCS, pp. 10-17, 21st International Conference on Artificial Neural Networks, ICANN 2011, Espoo, Finland, 6/14/11. https://doi.org/10.1007/978-3-642-21735-7_2
Cho K, Ilin A, Raiko T. Improved learning of Gaussian-Bernoulli restricted Boltzmann machines. In Artificial Neural Networks and Machine Learning, ICANN 2011 - 21st International Conference on Artificial Neural Networks, Proceedings. PART 1 ed. Vol. 6791 LNCS. 2011. p. 10-17. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); PART 1). https://doi.org/10.1007/978-3-642-21735-7_2
Cho, Kyunghyun ; Ilin, Alexander ; Raiko, Tapani. / Improved learning of Gaussian-Bernoulli restricted Boltzmann machines. Artificial Neural Networks and Machine Learning, ICANN 2011 - 21st International Conference on Artificial Neural Networks, Proceedings. Vol. 6791 LNCS PART 1. ed. 2011. pp. 10-17 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); PART 1).
@inproceedings{351c08edec9747d28066a6b8feb7f8be,
title = "Improved learning of Gaussian-Bernoulli restricted Boltzmann machines",
abstract = "We propose a few remedies to improve training of Gaussian-Bernoulli restricted Boltzmann machines (GBRBM), which is known to be difficult. Firstly, we use a different parameterization of the energy function, which allows for more intuitive interpretation of the parameters and facilitates learning. Secondly, we propose parallel tempering learning for GBRBM. Lastly, we use an adaptive learning rate which is selected automatically in order to stabilize training. Our extensive experiments show that the proposed improvements indeed remove most of the difficulties encountered when training GBRBMs using conventional methods.",
keywords = "Adaptive Learning Rate, Gaussian-Bernoulli Restricted Boltzmann Machine, Parallel Tempering, Restricted Boltzmann Machine",
author = "Kyunghyun Cho and Alexander Ilin and Tapani Raiko",
year = "2011",
doi = "10.1007/978-3-642-21735-7_2",
language = "English (US)",
isbn = "9783642217340",
volume = "6791 LNCS",
series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",
number = "PART 1",
pages = "10--17",
booktitle = "Artificial Neural Networks and Machine Learning, ICANN 2011 - 21st International Conference on Artificial Neural Networks, Proceedings",
edition = "PART 1",

}

TY - GEN

T1 - Improved learning of Gaussian-Bernoulli restricted Boltzmann machines

AU - Cho, Kyunghyun

AU - Ilin, Alexander

AU - Raiko, Tapani

PY - 2011

Y1 - 2011

N2 - We propose a few remedies to improve training of Gaussian-Bernoulli restricted Boltzmann machines (GBRBM), which is known to be difficult. Firstly, we use a different parameterization of the energy function, which allows for more intuitive interpretation of the parameters and facilitates learning. Secondly, we propose parallel tempering learning for GBRBM. Lastly, we use an adaptive learning rate which is selected automatically in order to stabilize training. Our extensive experiments show that the proposed improvements indeed remove most of the difficulties encountered when training GBRBMs using conventional methods.

AB - We propose a few remedies to improve training of Gaussian-Bernoulli restricted Boltzmann machines (GBRBM), which is known to be difficult. Firstly, we use a different parameterization of the energy function, which allows for more intuitive interpretation of the parameters and facilitates learning. Secondly, we propose parallel tempering learning for GBRBM. Lastly, we use an adaptive learning rate which is selected automatically in order to stabilize training. Our extensive experiments show that the proposed improvements indeed remove most of the difficulties encountered when training GBRBMs using conventional methods.

KW - Adaptive Learning Rate

KW - Gaussian-Bernoulli Restricted Boltzmann Machine

KW - Parallel Tempering

KW - Restricted Boltzmann Machine

UR - http://www.scopus.com/inward/record.url?scp=79959342724&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=79959342724&partnerID=8YFLogxK

U2 - 10.1007/978-3-642-21735-7_2

DO - 10.1007/978-3-642-21735-7_2

M3 - Conference contribution

SN - 9783642217340

VL - 6791 LNCS

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 10

EP - 17

BT - Artificial Neural Networks and Machine Learning, ICANN 2011 - 21st International Conference on Artificial Neural Networks, Proceedings

ER -