Learning fast approximations of sparse coding

Karol Gregor, Yann LeCun

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

In Sparse Coding (SC), input vectors are reconstructed using a sparse linear combination of basis vectors. SC has become a popular method for extracting features from data. For a given input, SC minimizes a quadratic reconstruction error with an L1 penalty term on the code. The process is often too slow for applications such as real-time pattern recognition. We proposed two versions of a very fast algorithm that produces approximate estimates of the sparse code that can be used to compute good visual features, or to initialize exact iterative algorithms. The main idea is to train a non-linear, feed-forward predictor with a specific architecture and a fixed depth to produce the best possible approximation of the sparse code. A version of the method, which can be seen as a trainable version of Li and Osher's coordinate descent method, is shown to produce approximate solutions with 10 times less computation than Li and Osher's for the same approximation error. Unlike previous proposals for sparse code predictors, the system allows a kind of approximate "explaining away" to take place during inference. The resulting predictor is differ- entiable and can be included into globally-trained recognition systems.

Original languageEnglish (US)
Title of host publicationICML 2010 - Proceedings, 27th International Conference on Machine Learning
Pages399-406
Number of pages8
StatePublished - 2010
Event27th International Conference on Machine Learning, ICML 2010 - Haifa, Israel
Duration: Jun 21 2010Jun 25 2010

Other

Other27th International Conference on Machine Learning, ICML 2010
CountryIsrael
CityHaifa
Period6/21/106/25/10

Fingerprint

coding
learning
Pattern recognition
pattern recognition
penalty
reconstruction
time

ASJC Scopus subject areas

  • Artificial Intelligence
  • Education

Cite this

Gregor, K., & LeCun, Y. (2010). Learning fast approximations of sparse coding. In ICML 2010 - Proceedings, 27th International Conference on Machine Learning (pp. 399-406)

Learning fast approximations of sparse coding. / Gregor, Karol; LeCun, Yann.

ICML 2010 - Proceedings, 27th International Conference on Machine Learning. 2010. p. 399-406.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Gregor, K & LeCun, Y 2010, Learning fast approximations of sparse coding. in ICML 2010 - Proceedings, 27th International Conference on Machine Learning. pp. 399-406, 27th International Conference on Machine Learning, ICML 2010, Haifa, Israel, 6/21/10.
Gregor K, LeCun Y. Learning fast approximations of sparse coding. In ICML 2010 - Proceedings, 27th International Conference on Machine Learning. 2010. p. 399-406
Gregor, Karol ; LeCun, Yann. / Learning fast approximations of sparse coding. ICML 2010 - Proceedings, 27th International Conference on Machine Learning. 2010. pp. 399-406
@inproceedings{46f49a297c8542aa80d7fad85aed17e0,
title = "Learning fast approximations of sparse coding",
abstract = "In Sparse Coding (SC), input vectors are reconstructed using a sparse linear combination of basis vectors. SC has become a popular method for extracting features from data. For a given input, SC minimizes a quadratic reconstruction error with an L1 penalty term on the code. The process is often too slow for applications such as real-time pattern recognition. We proposed two versions of a very fast algorithm that produces approximate estimates of the sparse code that can be used to compute good visual features, or to initialize exact iterative algorithms. The main idea is to train a non-linear, feed-forward predictor with a specific architecture and a fixed depth to produce the best possible approximation of the sparse code. A version of the method, which can be seen as a trainable version of Li and Osher's coordinate descent method, is shown to produce approximate solutions with 10 times less computation than Li and Osher's for the same approximation error. Unlike previous proposals for sparse code predictors, the system allows a kind of approximate {"}explaining away{"} to take place during inference. The resulting predictor is differ- entiable and can be included into globally-trained recognition systems.",
author = "Karol Gregor and Yann LeCun",
year = "2010",
language = "English (US)",
isbn = "9781605589077",
pages = "399--406",
booktitle = "ICML 2010 - Proceedings, 27th International Conference on Machine Learning",

}

TY - GEN

T1 - Learning fast approximations of sparse coding

AU - Gregor, Karol

AU - LeCun, Yann

PY - 2010

Y1 - 2010

N2 - In Sparse Coding (SC), input vectors are reconstructed using a sparse linear combination of basis vectors. SC has become a popular method for extracting features from data. For a given input, SC minimizes a quadratic reconstruction error with an L1 penalty term on the code. The process is often too slow for applications such as real-time pattern recognition. We proposed two versions of a very fast algorithm that produces approximate estimates of the sparse code that can be used to compute good visual features, or to initialize exact iterative algorithms. The main idea is to train a non-linear, feed-forward predictor with a specific architecture and a fixed depth to produce the best possible approximation of the sparse code. A version of the method, which can be seen as a trainable version of Li and Osher's coordinate descent method, is shown to produce approximate solutions with 10 times less computation than Li and Osher's for the same approximation error. Unlike previous proposals for sparse code predictors, the system allows a kind of approximate "explaining away" to take place during inference. The resulting predictor is differ- entiable and can be included into globally-trained recognition systems.

AB - In Sparse Coding (SC), input vectors are reconstructed using a sparse linear combination of basis vectors. SC has become a popular method for extracting features from data. For a given input, SC minimizes a quadratic reconstruction error with an L1 penalty term on the code. The process is often too slow for applications such as real-time pattern recognition. We proposed two versions of a very fast algorithm that produces approximate estimates of the sparse code that can be used to compute good visual features, or to initialize exact iterative algorithms. The main idea is to train a non-linear, feed-forward predictor with a specific architecture and a fixed depth to produce the best possible approximation of the sparse code. A version of the method, which can be seen as a trainable version of Li and Osher's coordinate descent method, is shown to produce approximate solutions with 10 times less computation than Li and Osher's for the same approximation error. Unlike previous proposals for sparse code predictors, the system allows a kind of approximate "explaining away" to take place during inference. The resulting predictor is differ- entiable and can be included into globally-trained recognition systems.

UR - http://www.scopus.com/inward/record.url?scp=77956515664&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=77956515664&partnerID=8YFLogxK

M3 - Conference contribution

SN - 9781605589077

SP - 399

EP - 406

BT - ICML 2010 - Proceedings, 27th International Conference on Machine Learning

ER -