Measuring the usefulness of hidden units in Boltzmann machines with mutual information

Mathias Berglund, Tapani Raiko, Kyunghyun Cho

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Restricted Boltzmann machines (RBMs) and deep Boltzmann machines (DBMs) are important models in deep learning, but it is often difficult to measure their performance in general, or measure the importance of individual hidden units in specific. We propose to use mutual information to measure the usefulness of individual hidden units in Boltzmann machines. The measure serves as an upper bound for the information the neuron can pass on, enabling detection of a particular kind of poor training results.We confirm experimentally, that the proposed measure is telling how much the performance of the model drops when some of the units of an RBM are pruned away. Our experiments on DBMs highlight differences among different pretraining options.

Original languageEnglish (US)
Title of host publicationNeural Information Processing - 20th International Conference, ICONIP 2013, Proceedings
Pages482-489
Number of pages8
Volume8226 LNCS
EditionPART 1
DOIs
StatePublished - 2013
Event20th International Conference on Neural Information Processing, ICONIP 2013 - Daegu, Korea, Republic of
Duration: Nov 3 2013Nov 7 2013

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
NumberPART 1
Volume8226 LNCS
ISSN (Print)03029743
ISSN (Electronic)16113349

Other

Other20th International Conference on Neural Information Processing, ICONIP 2013
CountryKorea, Republic of
CityDaegu
Period11/3/1311/7/13

Fingerprint

Boltzmann Machine
Mutual Information
Unit
Information use
Neurons
Performance Measures
Experiments
Neuron
Upper bound
Model
Experiment
Deep learning

Keywords

  • Deep boltzmann machine
  • Deep learning
  • Mutual information
  • Pruning
  • Restricted boltzmann machine
  • Structural learning

ASJC Scopus subject areas

  • Computer Science(all)
  • Theoretical Computer Science

Cite this

Berglund, M., Raiko, T., & Cho, K. (2013). Measuring the usefulness of hidden units in Boltzmann machines with mutual information. In Neural Information Processing - 20th International Conference, ICONIP 2013, Proceedings (PART 1 ed., Vol. 8226 LNCS, pp. 482-489). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 8226 LNCS, No. PART 1). https://doi.org/10.1007/978-3-642-42054-2_60

Measuring the usefulness of hidden units in Boltzmann machines with mutual information. / Berglund, Mathias; Raiko, Tapani; Cho, Kyunghyun.

Neural Information Processing - 20th International Conference, ICONIP 2013, Proceedings. Vol. 8226 LNCS PART 1. ed. 2013. p. 482-489 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 8226 LNCS, No. PART 1).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Berglund, M, Raiko, T & Cho, K 2013, Measuring the usefulness of hidden units in Boltzmann machines with mutual information. in Neural Information Processing - 20th International Conference, ICONIP 2013, Proceedings. PART 1 edn, vol. 8226 LNCS, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), no. PART 1, vol. 8226 LNCS, pp. 482-489, 20th International Conference on Neural Information Processing, ICONIP 2013, Daegu, Korea, Republic of, 11/3/13. https://doi.org/10.1007/978-3-642-42054-2_60
Berglund M, Raiko T, Cho K. Measuring the usefulness of hidden units in Boltzmann machines with mutual information. In Neural Information Processing - 20th International Conference, ICONIP 2013, Proceedings. PART 1 ed. Vol. 8226 LNCS. 2013. p. 482-489. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); PART 1). https://doi.org/10.1007/978-3-642-42054-2_60
Berglund, Mathias ; Raiko, Tapani ; Cho, Kyunghyun. / Measuring the usefulness of hidden units in Boltzmann machines with mutual information. Neural Information Processing - 20th International Conference, ICONIP 2013, Proceedings. Vol. 8226 LNCS PART 1. ed. 2013. pp. 482-489 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); PART 1).
@inproceedings{32f0f334a5eb4a21a9d50cb4a64bdb6e,
title = "Measuring the usefulness of hidden units in Boltzmann machines with mutual information",
abstract = "Restricted Boltzmann machines (RBMs) and deep Boltzmann machines (DBMs) are important models in deep learning, but it is often difficult to measure their performance in general, or measure the importance of individual hidden units in specific. We propose to use mutual information to measure the usefulness of individual hidden units in Boltzmann machines. The measure serves as an upper bound for the information the neuron can pass on, enabling detection of a particular kind of poor training results.We confirm experimentally, that the proposed measure is telling how much the performance of the model drops when some of the units of an RBM are pruned away. Our experiments on DBMs highlight differences among different pretraining options.",
keywords = "Deep boltzmann machine, Deep learning, Mutual information, Pruning, Restricted boltzmann machine, Structural learning",
author = "Mathias Berglund and Tapani Raiko and Kyunghyun Cho",
year = "2013",
doi = "10.1007/978-3-642-42054-2_60",
language = "English (US)",
isbn = "9783642420535",
volume = "8226 LNCS",
series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",
number = "PART 1",
pages = "482--489",
booktitle = "Neural Information Processing - 20th International Conference, ICONIP 2013, Proceedings",
edition = "PART 1",

}

TY - GEN

T1 - Measuring the usefulness of hidden units in Boltzmann machines with mutual information

AU - Berglund, Mathias

AU - Raiko, Tapani

AU - Cho, Kyunghyun

PY - 2013

Y1 - 2013

N2 - Restricted Boltzmann machines (RBMs) and deep Boltzmann machines (DBMs) are important models in deep learning, but it is often difficult to measure their performance in general, or measure the importance of individual hidden units in specific. We propose to use mutual information to measure the usefulness of individual hidden units in Boltzmann machines. The measure serves as an upper bound for the information the neuron can pass on, enabling detection of a particular kind of poor training results.We confirm experimentally, that the proposed measure is telling how much the performance of the model drops when some of the units of an RBM are pruned away. Our experiments on DBMs highlight differences among different pretraining options.

AB - Restricted Boltzmann machines (RBMs) and deep Boltzmann machines (DBMs) are important models in deep learning, but it is often difficult to measure their performance in general, or measure the importance of individual hidden units in specific. We propose to use mutual information to measure the usefulness of individual hidden units in Boltzmann machines. The measure serves as an upper bound for the information the neuron can pass on, enabling detection of a particular kind of poor training results.We confirm experimentally, that the proposed measure is telling how much the performance of the model drops when some of the units of an RBM are pruned away. Our experiments on DBMs highlight differences among different pretraining options.

KW - Deep boltzmann machine

KW - Deep learning

KW - Mutual information

KW - Pruning

KW - Restricted boltzmann machine

KW - Structural learning

UR - http://www.scopus.com/inward/record.url?scp=84893348709&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84893348709&partnerID=8YFLogxK

U2 - 10.1007/978-3-642-42054-2_60

DO - 10.1007/978-3-642-42054-2_60

M3 - Conference contribution

SN - 9783642420535

VL - 8226 LNCS

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 482

EP - 489

BT - Neural Information Processing - 20th International Conference, ICONIP 2013, Proceedings

ER -