Tikhonov-type regularization for restricted Boltzmann machines

Kyunghyun Cho, Alexander Ilin, Tapani Raiko

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

In this paper, we study a Tikhonov-type regularization for restricted Boltzmann machines (RBM). We present two alternative formulations of the Tikhonov-type regularization which encourage an RBM to learn a smoother probability distribution. Both formulations turn out to be combinations of the widely used weight-decay and sparsity regularization. We empirically evaluate the effect of the proposed regularization schemes and show that the use of them could help extracting better discriminative features with sparser hidden activation probabilities.

Original languageEnglish (US)
Title of host publicationArtificial Neural Networks and Machine Learning, ICANN 2012 - 22nd International Conference on Artificial Neural Networks, Proceedings
Pages81-88
Number of pages8
Volume7552 LNCS
EditionPART 1
DOIs
StatePublished - 2012
Event22nd International Conference on Artificial Neural Networks, ICANN 2012 - Lausanne, Switzerland
Duration: Sep 11 2012Sep 14 2012

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
NumberPART 1
Volume7552 LNCS
ISSN (Print)03029743
ISSN (Electronic)16113349

Other

Other22nd International Conference on Artificial Neural Networks, ICANN 2012
CountrySwitzerland
CityLausanne
Period9/11/129/14/12

Fingerprint

Boltzmann Machine
Probability distributions
Regularization
Chemical activation
Formulation
Sparsity
Activation
Probability Distribution
Decay
Evaluate
Alternatives

Keywords

  • Restricted Boltzmann Machine
  • Tikhonov Regularization

ASJC Scopus subject areas

  • Computer Science(all)
  • Theoretical Computer Science

Cite this

Cho, K., Ilin, A., & Raiko, T. (2012). Tikhonov-type regularization for restricted Boltzmann machines. In Artificial Neural Networks and Machine Learning, ICANN 2012 - 22nd International Conference on Artificial Neural Networks, Proceedings (PART 1 ed., Vol. 7552 LNCS, pp. 81-88). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 7552 LNCS, No. PART 1). https://doi.org/10.1007/978-3-642-33269-2_11

Tikhonov-type regularization for restricted Boltzmann machines. / Cho, Kyunghyun; Ilin, Alexander; Raiko, Tapani.

Artificial Neural Networks and Machine Learning, ICANN 2012 - 22nd International Conference on Artificial Neural Networks, Proceedings. Vol. 7552 LNCS PART 1. ed. 2012. p. 81-88 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 7552 LNCS, No. PART 1).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Cho, K, Ilin, A & Raiko, T 2012, Tikhonov-type regularization for restricted Boltzmann machines. in Artificial Neural Networks and Machine Learning, ICANN 2012 - 22nd International Conference on Artificial Neural Networks, Proceedings. PART 1 edn, vol. 7552 LNCS, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), no. PART 1, vol. 7552 LNCS, pp. 81-88, 22nd International Conference on Artificial Neural Networks, ICANN 2012, Lausanne, Switzerland, 9/11/12. https://doi.org/10.1007/978-3-642-33269-2_11
Cho K, Ilin A, Raiko T. Tikhonov-type regularization for restricted Boltzmann machines. In Artificial Neural Networks and Machine Learning, ICANN 2012 - 22nd International Conference on Artificial Neural Networks, Proceedings. PART 1 ed. Vol. 7552 LNCS. 2012. p. 81-88. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); PART 1). https://doi.org/10.1007/978-3-642-33269-2_11
Cho, Kyunghyun ; Ilin, Alexander ; Raiko, Tapani. / Tikhonov-type regularization for restricted Boltzmann machines. Artificial Neural Networks and Machine Learning, ICANN 2012 - 22nd International Conference on Artificial Neural Networks, Proceedings. Vol. 7552 LNCS PART 1. ed. 2012. pp. 81-88 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); PART 1).
@inproceedings{55b6ba6a6ce34b76b83afefec51024a9,
title = "Tikhonov-type regularization for restricted Boltzmann machines",
abstract = "In this paper, we study a Tikhonov-type regularization for restricted Boltzmann machines (RBM). We present two alternative formulations of the Tikhonov-type regularization which encourage an RBM to learn a smoother probability distribution. Both formulations turn out to be combinations of the widely used weight-decay and sparsity regularization. We empirically evaluate the effect of the proposed regularization schemes and show that the use of them could help extracting better discriminative features with sparser hidden activation probabilities.",
keywords = "Restricted Boltzmann Machine, Tikhonov Regularization",
author = "Kyunghyun Cho and Alexander Ilin and Tapani Raiko",
year = "2012",
doi = "10.1007/978-3-642-33269-2_11",
language = "English (US)",
isbn = "9783642332685",
volume = "7552 LNCS",
series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",
number = "PART 1",
pages = "81--88",
booktitle = "Artificial Neural Networks and Machine Learning, ICANN 2012 - 22nd International Conference on Artificial Neural Networks, Proceedings",
edition = "PART 1",

}

TY - GEN

T1 - Tikhonov-type regularization for restricted Boltzmann machines

AU - Cho, Kyunghyun

AU - Ilin, Alexander

AU - Raiko, Tapani

PY - 2012

Y1 - 2012

N2 - In this paper, we study a Tikhonov-type regularization for restricted Boltzmann machines (RBM). We present two alternative formulations of the Tikhonov-type regularization which encourage an RBM to learn a smoother probability distribution. Both formulations turn out to be combinations of the widely used weight-decay and sparsity regularization. We empirically evaluate the effect of the proposed regularization schemes and show that the use of them could help extracting better discriminative features with sparser hidden activation probabilities.

AB - In this paper, we study a Tikhonov-type regularization for restricted Boltzmann machines (RBM). We present two alternative formulations of the Tikhonov-type regularization which encourage an RBM to learn a smoother probability distribution. Both formulations turn out to be combinations of the widely used weight-decay and sparsity regularization. We empirically evaluate the effect of the proposed regularization schemes and show that the use of them could help extracting better discriminative features with sparser hidden activation probabilities.

KW - Restricted Boltzmann Machine

KW - Tikhonov Regularization

UR - http://www.scopus.com/inward/record.url?scp=84867671137&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84867671137&partnerID=8YFLogxK

U2 - 10.1007/978-3-642-33269-2_11

DO - 10.1007/978-3-642-33269-2_11

M3 - Conference contribution

AN - SCOPUS:84867671137

SN - 9783642332685

VL - 7552 LNCS

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 81

EP - 88

BT - Artificial Neural Networks and Machine Learning, ICANN 2012 - 22nd International Conference on Artificial Neural Networks, Proceedings

ER -