Universal halting times in optimization and machine learning

Levent Sagun, Thomas Trogdon, Yann LeCun

Research output: Contribution to journalArticle

Abstract

We present empirical evidence that the halting times for a class of optimization algorithms are universal. The algorithms we consider come from quadratic optimization, spin glasses and machine learning. A universality theorem is given in the case of the quadratic gradient descent flow. More precisely, given an algorithm, which we take to be both the optimization routine and the form of the random landscape, the fluctuations of the halting time of the algorithm follow a distribution that, after centering and scaling, appears invariant under changes in the distribution on the landscape - universality is present.

Original languageEnglish (US)
Pages (from-to)289-301
Number of pages13
JournalQuarterly of Applied Mathematics
Volume76
Issue number2
DOIs
StatePublished - Jan 1 2018

Fingerprint

Learning systems
Machine Learning
Universality
Optimization
Quadratic Optimization
Gradient Descent
Spin Glass
Spin glass
Optimization Algorithm
Scaling
Fluctuations
Invariant
Theorem
Class
Evidence
Form

ASJC Scopus subject areas

  • Applied Mathematics

Cite this

Universal halting times in optimization and machine learning. / Sagun, Levent; Trogdon, Thomas; LeCun, Yann.

In: Quarterly of Applied Mathematics, Vol. 76, No. 2, 01.01.2018, p. 289-301.

Research output: Contribution to journalArticle

Sagun, Levent ; Trogdon, Thomas ; LeCun, Yann. / Universal halting times in optimization and machine learning. In: Quarterly of Applied Mathematics. 2018 ; Vol. 76, No. 2. pp. 289-301.
@article{eb6a1e98381b44ce91f663dbe90fa0a8,
title = "Universal halting times in optimization and machine learning",
abstract = "We present empirical evidence that the halting times for a class of optimization algorithms are universal. The algorithms we consider come from quadratic optimization, spin glasses and machine learning. A universality theorem is given in the case of the quadratic gradient descent flow. More precisely, given an algorithm, which we take to be both the optimization routine and the form of the random landscape, the fluctuations of the halting time of the algorithm follow a distribution that, after centering and scaling, appears invariant under changes in the distribution on the landscape - universality is present.",
author = "Levent Sagun and Thomas Trogdon and Yann LeCun",
year = "2018",
month = "1",
day = "1",
doi = "10.1090/qam/1483",
language = "English (US)",
volume = "76",
pages = "289--301",
journal = "Quarterly of Applied Mathematics",
issn = "0033-569X",
publisher = "American Mathematical Society",
number = "2",

}

TY - JOUR

T1 - Universal halting times in optimization and machine learning

AU - Sagun, Levent

AU - Trogdon, Thomas

AU - LeCun, Yann

PY - 2018/1/1

Y1 - 2018/1/1

N2 - We present empirical evidence that the halting times for a class of optimization algorithms are universal. The algorithms we consider come from quadratic optimization, spin glasses and machine learning. A universality theorem is given in the case of the quadratic gradient descent flow. More precisely, given an algorithm, which we take to be both the optimization routine and the form of the random landscape, the fluctuations of the halting time of the algorithm follow a distribution that, after centering and scaling, appears invariant under changes in the distribution on the landscape - universality is present.

AB - We present empirical evidence that the halting times for a class of optimization algorithms are universal. The algorithms we consider come from quadratic optimization, spin glasses and machine learning. A universality theorem is given in the case of the quadratic gradient descent flow. More precisely, given an algorithm, which we take to be both the optimization routine and the form of the random landscape, the fluctuations of the halting time of the algorithm follow a distribution that, after centering and scaling, appears invariant under changes in the distribution on the landscape - universality is present.

UR - http://www.scopus.com/inward/record.url?scp=85039950078&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85039950078&partnerID=8YFLogxK

U2 - 10.1090/qam/1483

DO - 10.1090/qam/1483

M3 - Article

AN - SCOPUS:85039950078

VL - 76

SP - 289

EP - 301

JO - Quarterly of Applied Mathematics

JF - Quarterly of Applied Mathematics

SN - 0033-569X

IS - 2

ER -