On-line learning for very large data sets

Léon Bottou, Yann LeCun

Research output: Contribution to journalArticle

Abstract

The design of very large learning systems presents many unsolved challenges. Consider, for instance, a system that 'watches' television for a few weeks and learns to enumerate the objects present in these images. Most current learning algorithms do not scale well enough to handle such massive quantities of data. Experience suggests that the stochastic learning algorithms are best suited to such tasks. This is at first surprising because stochastic learning algorithms optimize the training error rather slowly. Our paper reconsiders the convergence speed in terms of how fast a learning algorithm optimizes the testing error. This reformulation shows the superiority of the well designed stochastic learning algorithm.

Original languageEnglish (US)
Pages (from-to)137-151
Number of pages15
JournalApplied Stochastic Models in Business and Industry
Volume21
Issue number2
DOIs
StatePublished - Mar 2005

Fingerprint

Large Data Sets
Learning algorithms
Learning Algorithm
Stochastic Algorithms
Optimise
Watches
Convergence Speed
Learning Systems
Television
Reformulation
Learning systems
Learning
Learning algorithm
Testing

Keywords

  • Convergence speed
  • Learning
  • Online learning
  • Stochastic optimization

ASJC Scopus subject areas

  • Applied Mathematics
  • Statistics and Probability
  • Management Science and Operations Research

Cite this

On-line learning for very large data sets. / Bottou, Léon; LeCun, Yann.

In: Applied Stochastic Models in Business and Industry, Vol. 21, No. 2, 03.2005, p. 137-151.

Research output: Contribution to journalArticle

@article{11915c740edf4c409154ea611c601fd0,
title = "On-line learning for very large data sets",
abstract = "The design of very large learning systems presents many unsolved challenges. Consider, for instance, a system that 'watches' television for a few weeks and learns to enumerate the objects present in these images. Most current learning algorithms do not scale well enough to handle such massive quantities of data. Experience suggests that the stochastic learning algorithms are best suited to such tasks. This is at first surprising because stochastic learning algorithms optimize the training error rather slowly. Our paper reconsiders the convergence speed in terms of how fast a learning algorithm optimizes the testing error. This reformulation shows the superiority of the well designed stochastic learning algorithm.",
keywords = "Convergence speed, Learning, Online learning, Stochastic optimization",
author = "L{\'e}on Bottou and Yann LeCun",
year = "2005",
month = "3",
doi = "10.1002/asmb.538",
language = "English (US)",
volume = "21",
pages = "137--151",
journal = "Applied Stochastic Models in Business and Industry",
issn = "1524-1904",
publisher = "John Wiley and Sons Ltd",
number = "2",

}

TY - JOUR

T1 - On-line learning for very large data sets

AU - Bottou, Léon

AU - LeCun, Yann

PY - 2005/3

Y1 - 2005/3

N2 - The design of very large learning systems presents many unsolved challenges. Consider, for instance, a system that 'watches' television for a few weeks and learns to enumerate the objects present in these images. Most current learning algorithms do not scale well enough to handle such massive quantities of data. Experience suggests that the stochastic learning algorithms are best suited to such tasks. This is at first surprising because stochastic learning algorithms optimize the training error rather slowly. Our paper reconsiders the convergence speed in terms of how fast a learning algorithm optimizes the testing error. This reformulation shows the superiority of the well designed stochastic learning algorithm.

AB - The design of very large learning systems presents many unsolved challenges. Consider, for instance, a system that 'watches' television for a few weeks and learns to enumerate the objects present in these images. Most current learning algorithms do not scale well enough to handle such massive quantities of data. Experience suggests that the stochastic learning algorithms are best suited to such tasks. This is at first surprising because stochastic learning algorithms optimize the training error rather slowly. Our paper reconsiders the convergence speed in terms of how fast a learning algorithm optimizes the testing error. This reformulation shows the superiority of the well designed stochastic learning algorithm.

KW - Convergence speed

KW - Learning

KW - Online learning

KW - Stochastic optimization

UR - http://www.scopus.com/inward/record.url?scp=17444425307&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=17444425307&partnerID=8YFLogxK

U2 - 10.1002/asmb.538

DO - 10.1002/asmb.538

M3 - Article

VL - 21

SP - 137

EP - 151

JO - Applied Stochastic Models in Business and Industry

JF - Applied Stochastic Models in Business and Industry

SN - 1524-1904

IS - 2

ER -