Efficient learning of sparse representations with an energy-based model

Marc Aurelio Ranzato, Christopher Poultney, Sumit Chopra, Yann LeCun

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

We describe a novel unsupervised method for learning sparse, overcomplete features. The model uses a linear encoder, and a linear decoder preceded by a sparsifying non-linearity that turns a code vector into a quasi-binary sparse code vector. Given an input, the optimal code minimizes the distance between the output of the decoder and the input patch while being as similar as possible to the encoder output. Learning proceeds in a two-phase EM-like fashion: (1) compute the minimum-energy code vector, (2) adjust the parameters of the encoder and decoder so as to decrease the energy. The model produces "stroke detectors" when trained on handwritten numerals, and Gabor-like filters when trained on natural image patches. Inference and learning are very fast, requiring no preprocessing, and no expensive sampling. Using the proposed unsupervised method to initialize the first layer of a convolutional network, we achieved an error rate slightly lower than the best reported result on the MNIST dataset. Finally, an extension of the method is described to learn topographical filter maps.

Original languageEnglish (US)
Title of host publicationAdvances in Neural Information Processing Systems 19 - Proceedings of the 2006 Conference
Pages1137-1144
Number of pages8
StatePublished - 2007
Event20th Annual Conference on Neural Information Processing Systems, NIPS 2006 - Vancouver, BC, Canada
Duration: Dec 4 2006Dec 7 2006

Other

Other20th Annual Conference on Neural Information Processing Systems, NIPS 2006
CountryCanada
CityVancouver, BC
Period12/4/0612/7/06

Fingerprint

Sampling
Detectors

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Information Systems
  • Signal Processing

Cite this

Ranzato, M. A., Poultney, C., Chopra, S., & LeCun, Y. (2007). Efficient learning of sparse representations with an energy-based model. In Advances in Neural Information Processing Systems 19 - Proceedings of the 2006 Conference (pp. 1137-1144)

Efficient learning of sparse representations with an energy-based model. / Ranzato, Marc Aurelio; Poultney, Christopher; Chopra, Sumit; LeCun, Yann.

Advances in Neural Information Processing Systems 19 - Proceedings of the 2006 Conference. 2007. p. 1137-1144.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Ranzato, MA, Poultney, C, Chopra, S & LeCun, Y 2007, Efficient learning of sparse representations with an energy-based model. in Advances in Neural Information Processing Systems 19 - Proceedings of the 2006 Conference. pp. 1137-1144, 20th Annual Conference on Neural Information Processing Systems, NIPS 2006, Vancouver, BC, Canada, 12/4/06.
Ranzato MA, Poultney C, Chopra S, LeCun Y. Efficient learning of sparse representations with an energy-based model. In Advances in Neural Information Processing Systems 19 - Proceedings of the 2006 Conference. 2007. p. 1137-1144
Ranzato, Marc Aurelio ; Poultney, Christopher ; Chopra, Sumit ; LeCun, Yann. / Efficient learning of sparse representations with an energy-based model. Advances in Neural Information Processing Systems 19 - Proceedings of the 2006 Conference. 2007. pp. 1137-1144
@inproceedings{8a94e88f3ff04a668f4cb269909636ae,
title = "Efficient learning of sparse representations with an energy-based model",
abstract = "We describe a novel unsupervised method for learning sparse, overcomplete features. The model uses a linear encoder, and a linear decoder preceded by a sparsifying non-linearity that turns a code vector into a quasi-binary sparse code vector. Given an input, the optimal code minimizes the distance between the output of the decoder and the input patch while being as similar as possible to the encoder output. Learning proceeds in a two-phase EM-like fashion: (1) compute the minimum-energy code vector, (2) adjust the parameters of the encoder and decoder so as to decrease the energy. The model produces {"}stroke detectors{"} when trained on handwritten numerals, and Gabor-like filters when trained on natural image patches. Inference and learning are very fast, requiring no preprocessing, and no expensive sampling. Using the proposed unsupervised method to initialize the first layer of a convolutional network, we achieved an error rate slightly lower than the best reported result on the MNIST dataset. Finally, an extension of the method is described to learn topographical filter maps.",
author = "Ranzato, {Marc Aurelio} and Christopher Poultney and Sumit Chopra and Yann LeCun",
year = "2007",
language = "English (US)",
isbn = "9780262195683",
pages = "1137--1144",
booktitle = "Advances in Neural Information Processing Systems 19 - Proceedings of the 2006 Conference",

}

TY - GEN

T1 - Efficient learning of sparse representations with an energy-based model

AU - Ranzato, Marc Aurelio

AU - Poultney, Christopher

AU - Chopra, Sumit

AU - LeCun, Yann

PY - 2007

Y1 - 2007

N2 - We describe a novel unsupervised method for learning sparse, overcomplete features. The model uses a linear encoder, and a linear decoder preceded by a sparsifying non-linearity that turns a code vector into a quasi-binary sparse code vector. Given an input, the optimal code minimizes the distance between the output of the decoder and the input patch while being as similar as possible to the encoder output. Learning proceeds in a two-phase EM-like fashion: (1) compute the minimum-energy code vector, (2) adjust the parameters of the encoder and decoder so as to decrease the energy. The model produces "stroke detectors" when trained on handwritten numerals, and Gabor-like filters when trained on natural image patches. Inference and learning are very fast, requiring no preprocessing, and no expensive sampling. Using the proposed unsupervised method to initialize the first layer of a convolutional network, we achieved an error rate slightly lower than the best reported result on the MNIST dataset. Finally, an extension of the method is described to learn topographical filter maps.

AB - We describe a novel unsupervised method for learning sparse, overcomplete features. The model uses a linear encoder, and a linear decoder preceded by a sparsifying non-linearity that turns a code vector into a quasi-binary sparse code vector. Given an input, the optimal code minimizes the distance between the output of the decoder and the input patch while being as similar as possible to the encoder output. Learning proceeds in a two-phase EM-like fashion: (1) compute the minimum-energy code vector, (2) adjust the parameters of the encoder and decoder so as to decrease the energy. The model produces "stroke detectors" when trained on handwritten numerals, and Gabor-like filters when trained on natural image patches. Inference and learning are very fast, requiring no preprocessing, and no expensive sampling. Using the proposed unsupervised method to initialize the first layer of a convolutional network, we achieved an error rate slightly lower than the best reported result on the MNIST dataset. Finally, an extension of the method is described to learn topographical filter maps.

UR - http://www.scopus.com/inward/record.url?scp=84864069017&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84864069017&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:84864069017

SN - 9780262195683

SP - 1137

EP - 1144

BT - Advances in Neural Information Processing Systems 19 - Proceedings of the 2006 Conference

ER -