Measuring the usefulness of hidden units in Boltzmann machines with mutual information

Mathias Berglund, Tapani Raiko, Kyunghyun Cho

Research output: Contribution to journalArticle

Abstract

Restricted Boltzmann machines (RBMs) and deep Boltzmann machines (DBMs) are important models in deep learning, but it is often difficult to measure their performance in general, or measure the importance of individual hidden units in specific. We propose to use mutual information to measure the usefulness of individual hidden units in Boltzmann machines. The measure is fast to compute, and serves as an upper bound for the information the neuron can pass on, enabling detection of a particular kind of poor training results. We confirm experimentally that the proposed measure indicates how much the performance of the model drops when some of the units of an RBM are pruned away. We demonstrate the usefulness of the measure for early detection of poor training in DBMs.

Original languageEnglish (US)
Pages (from-to)12-18
Number of pages7
JournalNeural Networks
Volume64
DOIs
StatePublished - Apr 1 2015

Fingerprint

Learning
Neurons
Information use
Deep learning

Keywords

  • Deep Boltzmann machine
  • Deep learning
  • Mutual information
  • Pruning
  • Restricted Boltzmann machine
  • Structural learning

ASJC Scopus subject areas

  • Artificial Intelligence
  • Cognitive Neuroscience
  • Medicine(all)

Cite this

Measuring the usefulness of hidden units in Boltzmann machines with mutual information. / Berglund, Mathias; Raiko, Tapani; Cho, Kyunghyun.

In: Neural Networks, Vol. 64, 01.04.2015, p. 12-18.

Research output: Contribution to journalArticle

@article{3b89eaecad13414b90e1a57b9d0f8687,
title = "Measuring the usefulness of hidden units in Boltzmann machines with mutual information",
abstract = "Restricted Boltzmann machines (RBMs) and deep Boltzmann machines (DBMs) are important models in deep learning, but it is often difficult to measure their performance in general, or measure the importance of individual hidden units in specific. We propose to use mutual information to measure the usefulness of individual hidden units in Boltzmann machines. The measure is fast to compute, and serves as an upper bound for the information the neuron can pass on, enabling detection of a particular kind of poor training results. We confirm experimentally that the proposed measure indicates how much the performance of the model drops when some of the units of an RBM are pruned away. We demonstrate the usefulness of the measure for early detection of poor training in DBMs.",
keywords = "Deep Boltzmann machine, Deep learning, Mutual information, Pruning, Restricted Boltzmann machine, Structural learning",
author = "Mathias Berglund and Tapani Raiko and Kyunghyun Cho",
year = "2015",
month = "4",
day = "1",
doi = "10.1016/j.neunet.2014.09.004",
language = "English (US)",
volume = "64",
pages = "12--18",
journal = "Neural Networks",
issn = "0893-6080",
publisher = "Elsevier Limited",

}

TY - JOUR

T1 - Measuring the usefulness of hidden units in Boltzmann machines with mutual information

AU - Berglund, Mathias

AU - Raiko, Tapani

AU - Cho, Kyunghyun

PY - 2015/4/1

Y1 - 2015/4/1

N2 - Restricted Boltzmann machines (RBMs) and deep Boltzmann machines (DBMs) are important models in deep learning, but it is often difficult to measure their performance in general, or measure the importance of individual hidden units in specific. We propose to use mutual information to measure the usefulness of individual hidden units in Boltzmann machines. The measure is fast to compute, and serves as an upper bound for the information the neuron can pass on, enabling detection of a particular kind of poor training results. We confirm experimentally that the proposed measure indicates how much the performance of the model drops when some of the units of an RBM are pruned away. We demonstrate the usefulness of the measure for early detection of poor training in DBMs.

AB - Restricted Boltzmann machines (RBMs) and deep Boltzmann machines (DBMs) are important models in deep learning, but it is often difficult to measure their performance in general, or measure the importance of individual hidden units in specific. We propose to use mutual information to measure the usefulness of individual hidden units in Boltzmann machines. The measure is fast to compute, and serves as an upper bound for the information the neuron can pass on, enabling detection of a particular kind of poor training results. We confirm experimentally that the proposed measure indicates how much the performance of the model drops when some of the units of an RBM are pruned away. We demonstrate the usefulness of the measure for early detection of poor training in DBMs.

KW - Deep Boltzmann machine

KW - Deep learning

KW - Mutual information

KW - Pruning

KW - Restricted Boltzmann machine

KW - Structural learning

UR - http://www.scopus.com/inward/record.url?scp=84922376529&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84922376529&partnerID=8YFLogxK

U2 - 10.1016/j.neunet.2014.09.004

DO - 10.1016/j.neunet.2014.09.004

M3 - Article

VL - 64

SP - 12

EP - 18

JO - Neural Networks

JF - Neural Networks

SN - 0893-6080

ER -