A unified energy-based framework for unsupervised learning

Marc'Aurelio Ranzato, Y. Lan Boureau, Yann LeCun, Sumit Chopra

Research output: Contribution to journalArticle

Abstract

We introduce a view of unsupervised learning that integrates probabilistic and non-probabilistic methods for clustering, dimensionality reduction, and feature extraction in a unified framework. In this framework, an energy function associates low energies to input points that are similar to training samples, and high energies to unobserved points. Learning consists in minimizing the energies of training samples while ensuring that the energies of unobserved ones are higher. Some traditional methods construct the architecture so that only a small number of points can have low energy, while other methods explicitly "pull up" on the energies of unobserved points. In probabilistic methods the energy of unobserved points is pulled by minimizing the log partition function, an expensive, and sometimes intractable process. We explore different and more efficient methods using an energy-based approach. In particular, we show that a simple solution is to restrict the amount of information contained in codes that represent the data. We demonstrate such a method by training it on natural image patches and by applying to image denoising.

Original languageEnglish (US)
Pages (from-to)371-379
Number of pages9
JournalJournal of Machine Learning Research
Volume2
StatePublished - 2007

Fingerprint

Unsupervised learning
Unsupervised Learning
Image denoising
Energy
Feature extraction
Training Samples
Image Denoising
Probabilistic Methods
Dimensionality Reduction
Framework
Energy Function
Partition Function
Feature Extraction
Patch
High Energy
Integrate
Clustering
Demonstrate

ASJC Scopus subject areas

  • Artificial Intelligence
  • Software
  • Control and Systems Engineering
  • Statistics and Probability

Cite this

A unified energy-based framework for unsupervised learning. / Ranzato, Marc'Aurelio; Boureau, Y. Lan; LeCun, Yann; Chopra, Sumit.

In: Journal of Machine Learning Research, Vol. 2, 2007, p. 371-379.

Research output: Contribution to journalArticle

Ranzato, Marc'Aurelio ; Boureau, Y. Lan ; LeCun, Yann ; Chopra, Sumit. / A unified energy-based framework for unsupervised learning. In: Journal of Machine Learning Research. 2007 ; Vol. 2. pp. 371-379.
@article{a117e06792214ec69f778c1628c68ad5,
title = "A unified energy-based framework for unsupervised learning",
abstract = "We introduce a view of unsupervised learning that integrates probabilistic and non-probabilistic methods for clustering, dimensionality reduction, and feature extraction in a unified framework. In this framework, an energy function associates low energies to input points that are similar to training samples, and high energies to unobserved points. Learning consists in minimizing the energies of training samples while ensuring that the energies of unobserved ones are higher. Some traditional methods construct the architecture so that only a small number of points can have low energy, while other methods explicitly {"}pull up{"} on the energies of unobserved points. In probabilistic methods the energy of unobserved points is pulled by minimizing the log partition function, an expensive, and sometimes intractable process. We explore different and more efficient methods using an energy-based approach. In particular, we show that a simple solution is to restrict the amount of information contained in codes that represent the data. We demonstrate such a method by training it on natural image patches and by applying to image denoising.",
author = "Marc'Aurelio Ranzato and Boureau, {Y. Lan} and Yann LeCun and Sumit Chopra",
year = "2007",
language = "English (US)",
volume = "2",
pages = "371--379",
journal = "Journal of Machine Learning Research",
issn = "1532-4435",
publisher = "Microtome Publishing",

}

TY - JOUR

T1 - A unified energy-based framework for unsupervised learning

AU - Ranzato, Marc'Aurelio

AU - Boureau, Y. Lan

AU - LeCun, Yann

AU - Chopra, Sumit

PY - 2007

Y1 - 2007

N2 - We introduce a view of unsupervised learning that integrates probabilistic and non-probabilistic methods for clustering, dimensionality reduction, and feature extraction in a unified framework. In this framework, an energy function associates low energies to input points that are similar to training samples, and high energies to unobserved points. Learning consists in minimizing the energies of training samples while ensuring that the energies of unobserved ones are higher. Some traditional methods construct the architecture so that only a small number of points can have low energy, while other methods explicitly "pull up" on the energies of unobserved points. In probabilistic methods the energy of unobserved points is pulled by minimizing the log partition function, an expensive, and sometimes intractable process. We explore different and more efficient methods using an energy-based approach. In particular, we show that a simple solution is to restrict the amount of information contained in codes that represent the data. We demonstrate such a method by training it on natural image patches and by applying to image denoising.

AB - We introduce a view of unsupervised learning that integrates probabilistic and non-probabilistic methods for clustering, dimensionality reduction, and feature extraction in a unified framework. In this framework, an energy function associates low energies to input points that are similar to training samples, and high energies to unobserved points. Learning consists in minimizing the energies of training samples while ensuring that the energies of unobserved ones are higher. Some traditional methods construct the architecture so that only a small number of points can have low energy, while other methods explicitly "pull up" on the energies of unobserved points. In probabilistic methods the energy of unobserved points is pulled by minimizing the log partition function, an expensive, and sometimes intractable process. We explore different and more efficient methods using an energy-based approach. In particular, we show that a simple solution is to restrict the amount of information contained in codes that represent the data. We demonstrate such a method by training it on natural image patches and by applying to image denoising.

UR - http://www.scopus.com/inward/record.url?scp=84862270523&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84862270523&partnerID=8YFLogxK

M3 - Article

VL - 2

SP - 371

EP - 379

JO - Journal of Machine Learning Research

JF - Journal of Machine Learning Research

SN - 1532-4435

ER -