Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations

Honglak Lee, Roger Grosse, Rajesh Ranganath, Andrew Y. Ng

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

There has been much interest in unsupervised learning of hierarchical generative models such as deep belief networks. Scaling such models to full-sized, high-dimensional images remains a difficult problem. To address this problem, we present the convolutional deep belief network, a hierarchical generative model which scales to realistic image sizes. This model is translation-invariant and supports efficient bottom-up and top-down probabilistic inference. Key to our approach is probabilistic max-pooling, a novel technique which shrinks the representations of higher layers in a probabilistically sound way. Our experiments show that the algorithm learns useful high-level visual features, such as object parts, from unlabeled images of objects and natural scenes. We demonstrate excellent performance on several visual recognition tasks and show that our model can perform hierarchical (bottom-up and top-down) inference over full-sized images.

Original languageEnglish (US)
Title of host publicationProceedings of the 26th International Conference On Machine Learning, ICML 2009
Pages609-616
Number of pages8
StatePublished - Dec 9 2009
Event26th International Conference On Machine Learning, ICML 2009 - Montreal, QC, Canada
Duration: Jun 14 2009Jun 18 2009

Other

Other26th International Conference On Machine Learning, ICML 2009
CountryCanada
CityMontreal, QC
Period6/14/096/18/09

Fingerprint

Unsupervised learning
Bayesian networks
Acoustic waves
Experiments

ASJC Scopus subject areas

  • Artificial Intelligence
  • Computer Networks and Communications
  • Software

Cite this

Lee, H., Grosse, R., Ranganath, R., & Ng, A. Y. (2009). Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. In Proceedings of the 26th International Conference On Machine Learning, ICML 2009 (pp. 609-616)

Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. / Lee, Honglak; Grosse, Roger; Ranganath, Rajesh; Ng, Andrew Y.

Proceedings of the 26th International Conference On Machine Learning, ICML 2009. 2009. p. 609-616.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Lee, H, Grosse, R, Ranganath, R & Ng, AY 2009, Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. in Proceedings of the 26th International Conference On Machine Learning, ICML 2009. pp. 609-616, 26th International Conference On Machine Learning, ICML 2009, Montreal, QC, Canada, 6/14/09.
Lee H, Grosse R, Ranganath R, Ng AY. Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. In Proceedings of the 26th International Conference On Machine Learning, ICML 2009. 2009. p. 609-616
Lee, Honglak ; Grosse, Roger ; Ranganath, Rajesh ; Ng, Andrew Y. / Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. Proceedings of the 26th International Conference On Machine Learning, ICML 2009. 2009. pp. 609-616
@inproceedings{ef26d2100c974e01b1009ecaa80ddc1c,
title = "Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations",
abstract = "There has been much interest in unsupervised learning of hierarchical generative models such as deep belief networks. Scaling such models to full-sized, high-dimensional images remains a difficult problem. To address this problem, we present the convolutional deep belief network, a hierarchical generative model which scales to realistic image sizes. This model is translation-invariant and supports efficient bottom-up and top-down probabilistic inference. Key to our approach is probabilistic max-pooling, a novel technique which shrinks the representations of higher layers in a probabilistically sound way. Our experiments show that the algorithm learns useful high-level visual features, such as object parts, from unlabeled images of objects and natural scenes. We demonstrate excellent performance on several visual recognition tasks and show that our model can perform hierarchical (bottom-up and top-down) inference over full-sized images.",
author = "Honglak Lee and Roger Grosse and Rajesh Ranganath and Ng, {Andrew Y.}",
year = "2009",
month = "12",
day = "9",
language = "English (US)",
isbn = "9781605585161",
pages = "609--616",
booktitle = "Proceedings of the 26th International Conference On Machine Learning, ICML 2009",

}

TY - GEN

T1 - Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations

AU - Lee, Honglak

AU - Grosse, Roger

AU - Ranganath, Rajesh

AU - Ng, Andrew Y.

PY - 2009/12/9

Y1 - 2009/12/9

N2 - There has been much interest in unsupervised learning of hierarchical generative models such as deep belief networks. Scaling such models to full-sized, high-dimensional images remains a difficult problem. To address this problem, we present the convolutional deep belief network, a hierarchical generative model which scales to realistic image sizes. This model is translation-invariant and supports efficient bottom-up and top-down probabilistic inference. Key to our approach is probabilistic max-pooling, a novel technique which shrinks the representations of higher layers in a probabilistically sound way. Our experiments show that the algorithm learns useful high-level visual features, such as object parts, from unlabeled images of objects and natural scenes. We demonstrate excellent performance on several visual recognition tasks and show that our model can perform hierarchical (bottom-up and top-down) inference over full-sized images.

AB - There has been much interest in unsupervised learning of hierarchical generative models such as deep belief networks. Scaling such models to full-sized, high-dimensional images remains a difficult problem. To address this problem, we present the convolutional deep belief network, a hierarchical generative model which scales to realistic image sizes. This model is translation-invariant and supports efficient bottom-up and top-down probabilistic inference. Key to our approach is probabilistic max-pooling, a novel technique which shrinks the representations of higher layers in a probabilistically sound way. Our experiments show that the algorithm learns useful high-level visual features, such as object parts, from unlabeled images of objects and natural scenes. We demonstrate excellent performance on several visual recognition tasks and show that our model can perform hierarchical (bottom-up and top-down) inference over full-sized images.

UR - http://www.scopus.com/inward/record.url?scp=71149119164&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=71149119164&partnerID=8YFLogxK

M3 - Conference contribution

SN - 9781605585161

SP - 609

EP - 616

BT - Proceedings of the 26th International Conference On Machine Learning, ICML 2009

ER -