Discriminative recurrent sparse auto-encoders

Jason Tyler Rolfe, Yann LeCun

Research output: Contribution to conferencePaper

Abstract

We present the discriminative recurrent sparse auto-encoder model, comprising a recurrent encoder of rectified linear units, unrolled for a fixed number of iterations, and connected to two linear decoders that reconstruct the input and predict its supervised classification. Training via backpropagation-through-time initially minimizes an unsupervised sparse reconstruction error; the loss function is then augmented with a discriminative term on the supervised classification. The depth implicit in the temporally-unrolled form allows the system to exhibit far more representational power, while keeping the number of trainable parameters fixed. From an initially unstructured network the hidden units differentiate into categorical-units, each of which represents an input prototype with a well-defined class; and part-units representing deformations of these prototypes. The learned organization of the recurrent encoder is hierarchical: part-units are driven directly by the input, whereas the activity of categorical-units builds up over time through interactions with the part-units. Even using a small number of hidden units per layer, discriminative recurrent sparse auto-encoders achieve excellent performance on MNIST.

Original languageEnglish (US)
StatePublished - Jan 1 2013
Event1st International Conference on Learning Representations, ICLR 2013 - Scottsdale, United States
Duration: May 2 2013May 4 2013

Conference

Conference1st International Conference on Learning Representations, ICLR 2013
CountryUnited States
CityScottsdale
Period5/2/135/4/13

Fingerprint

Backpropagation
reconstruction
organization
interaction
performance
time
Prototype
Categorical
Iteration
Interaction
Layer

ASJC Scopus subject areas

  • Education
  • Computer Science Applications
  • Linguistics and Language
  • Language and Linguistics

Cite this

Rolfe, J. T., & LeCun, Y. (2013). Discriminative recurrent sparse auto-encoders. Paper presented at 1st International Conference on Learning Representations, ICLR 2013, Scottsdale, United States.

Discriminative recurrent sparse auto-encoders. / Rolfe, Jason Tyler; LeCun, Yann.

2013. Paper presented at 1st International Conference on Learning Representations, ICLR 2013, Scottsdale, United States.

Research output: Contribution to conferencePaper

Rolfe, JT & LeCun, Y 2013, 'Discriminative recurrent sparse auto-encoders' Paper presented at 1st International Conference on Learning Representations, ICLR 2013, Scottsdale, United States, 5/2/13 - 5/4/13, .
Rolfe JT, LeCun Y. Discriminative recurrent sparse auto-encoders. 2013. Paper presented at 1st International Conference on Learning Representations, ICLR 2013, Scottsdale, United States.
Rolfe, Jason Tyler ; LeCun, Yann. / Discriminative recurrent sparse auto-encoders. Paper presented at 1st International Conference on Learning Representations, ICLR 2013, Scottsdale, United States.
@conference{ed9bc61539cf453f9fc483a3b1a270a6,
title = "Discriminative recurrent sparse auto-encoders",
abstract = "We present the discriminative recurrent sparse auto-encoder model, comprising a recurrent encoder of rectified linear units, unrolled for a fixed number of iterations, and connected to two linear decoders that reconstruct the input and predict its supervised classification. Training via backpropagation-through-time initially minimizes an unsupervised sparse reconstruction error; the loss function is then augmented with a discriminative term on the supervised classification. The depth implicit in the temporally-unrolled form allows the system to exhibit far more representational power, while keeping the number of trainable parameters fixed. From an initially unstructured network the hidden units differentiate into categorical-units, each of which represents an input prototype with a well-defined class; and part-units representing deformations of these prototypes. The learned organization of the recurrent encoder is hierarchical: part-units are driven directly by the input, whereas the activity of categorical-units builds up over time through interactions with the part-units. Even using a small number of hidden units per layer, discriminative recurrent sparse auto-encoders achieve excellent performance on MNIST.",
author = "Rolfe, {Jason Tyler} and Yann LeCun",
year = "2013",
month = "1",
day = "1",
language = "English (US)",
note = "1st International Conference on Learning Representations, ICLR 2013 ; Conference date: 02-05-2013 Through 04-05-2013",

}

TY - CONF

T1 - Discriminative recurrent sparse auto-encoders

AU - Rolfe, Jason Tyler

AU - LeCun, Yann

PY - 2013/1/1

Y1 - 2013/1/1

N2 - We present the discriminative recurrent sparse auto-encoder model, comprising a recurrent encoder of rectified linear units, unrolled for a fixed number of iterations, and connected to two linear decoders that reconstruct the input and predict its supervised classification. Training via backpropagation-through-time initially minimizes an unsupervised sparse reconstruction error; the loss function is then augmented with a discriminative term on the supervised classification. The depth implicit in the temporally-unrolled form allows the system to exhibit far more representational power, while keeping the number of trainable parameters fixed. From an initially unstructured network the hidden units differentiate into categorical-units, each of which represents an input prototype with a well-defined class; and part-units representing deformations of these prototypes. The learned organization of the recurrent encoder is hierarchical: part-units are driven directly by the input, whereas the activity of categorical-units builds up over time through interactions with the part-units. Even using a small number of hidden units per layer, discriminative recurrent sparse auto-encoders achieve excellent performance on MNIST.

AB - We present the discriminative recurrent sparse auto-encoder model, comprising a recurrent encoder of rectified linear units, unrolled for a fixed number of iterations, and connected to two linear decoders that reconstruct the input and predict its supervised classification. Training via backpropagation-through-time initially minimizes an unsupervised sparse reconstruction error; the loss function is then augmented with a discriminative term on the supervised classification. The depth implicit in the temporally-unrolled form allows the system to exhibit far more representational power, while keeping the number of trainable parameters fixed. From an initially unstructured network the hidden units differentiate into categorical-units, each of which represents an input prototype with a well-defined class; and part-units representing deformations of these prototypes. The learned organization of the recurrent encoder is hierarchical: part-units are driven directly by the input, whereas the activity of categorical-units builds up over time through interactions with the part-units. Even using a small number of hidden units per layer, discriminative recurrent sparse auto-encoders achieve excellent performance on MNIST.

UR - http://www.scopus.com/inward/record.url?scp=85054233117&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85054233117&partnerID=8YFLogxK

M3 - Paper

ER -