Explorations on high dimensional landscapes

Levent Sagun, V. Ugur Güney, Gerard Ben Arous, Yann LeCun

Research output: Contribution to conferencePaper

Abstract

Finding minima of a real valued non-convex function over a high dimensional space is a major challenge in science. We provide evidence that some such functions that are defined on high dimensional domains have a narrow band of values whose pre-image contains the bulk of its critical points. This is in contrast with the low dimensional picture in which this band is wide. Our simulations agree with the previous theoretical work on spin glasses that proves the existence of such a band when the dimension of the domain tends to infinity. Furthermore our experiments on teacher-student networks with the MNIST dataset establish a similar phenomenon in deep networks. We finally observe that both the gradient descent and the stochastic gradient descent methods can reach this level within the same number of steps.

Original languageEnglish (US)
StatePublished - Jan 1 2015
Event3rd International Conference on Learning Representations, ICLR 2015 - San Diego, United States
Duration: May 7 2015May 9 2015

Conference

Conference3rd International Conference on Learning Representations, ICLR 2015
CountryUnited States
CitySan Diego
Period5/7/155/9/15

Fingerprint

Spin glass
student teacher
Students
simulation
experiment
science
evidence
Values
Experiments
Descent
Infinity
Bulk
Simulation
Experiment

ASJC Scopus subject areas

  • Education
  • Linguistics and Language
  • Language and Linguistics
  • Computer Science Applications

Cite this

Sagun, L., Ugur Güney, V., Ben Arous, G., & LeCun, Y. (2015). Explorations on high dimensional landscapes. Paper presented at 3rd International Conference on Learning Representations, ICLR 2015, San Diego, United States.

Explorations on high dimensional landscapes. / Sagun, Levent; Ugur Güney, V.; Ben Arous, Gerard; LeCun, Yann.

2015. Paper presented at 3rd International Conference on Learning Representations, ICLR 2015, San Diego, United States.

Research output: Contribution to conferencePaper

Sagun, L, Ugur Güney, V, Ben Arous, G & LeCun, Y 2015, 'Explorations on high dimensional landscapes', Paper presented at 3rd International Conference on Learning Representations, ICLR 2015, San Diego, United States, 5/7/15 - 5/9/15.
Sagun L, Ugur Güney V, Ben Arous G, LeCun Y. Explorations on high dimensional landscapes. 2015. Paper presented at 3rd International Conference on Learning Representations, ICLR 2015, San Diego, United States.
Sagun, Levent ; Ugur Güney, V. ; Ben Arous, Gerard ; LeCun, Yann. / Explorations on high dimensional landscapes. Paper presented at 3rd International Conference on Learning Representations, ICLR 2015, San Diego, United States.
@conference{a51b974f06ba45de85bed54bececfdcb,
title = "Explorations on high dimensional landscapes",
abstract = "Finding minima of a real valued non-convex function over a high dimensional space is a major challenge in science. We provide evidence that some such functions that are defined on high dimensional domains have a narrow band of values whose pre-image contains the bulk of its critical points. This is in contrast with the low dimensional picture in which this band is wide. Our simulations agree with the previous theoretical work on spin glasses that proves the existence of such a band when the dimension of the domain tends to infinity. Furthermore our experiments on teacher-student networks with the MNIST dataset establish a similar phenomenon in deep networks. We finally observe that both the gradient descent and the stochastic gradient descent methods can reach this level within the same number of steps.",
author = "Levent Sagun and {Ugur G{\"u}ney}, V. and {Ben Arous}, Gerard and Yann LeCun",
year = "2015",
month = "1",
day = "1",
language = "English (US)",
note = "3rd International Conference on Learning Representations, ICLR 2015 ; Conference date: 07-05-2015 Through 09-05-2015",

}

TY - CONF

T1 - Explorations on high dimensional landscapes

AU - Sagun, Levent

AU - Ugur Güney, V.

AU - Ben Arous, Gerard

AU - LeCun, Yann

PY - 2015/1/1

Y1 - 2015/1/1

N2 - Finding minima of a real valued non-convex function over a high dimensional space is a major challenge in science. We provide evidence that some such functions that are defined on high dimensional domains have a narrow band of values whose pre-image contains the bulk of its critical points. This is in contrast with the low dimensional picture in which this band is wide. Our simulations agree with the previous theoretical work on spin glasses that proves the existence of such a band when the dimension of the domain tends to infinity. Furthermore our experiments on teacher-student networks with the MNIST dataset establish a similar phenomenon in deep networks. We finally observe that both the gradient descent and the stochastic gradient descent methods can reach this level within the same number of steps.

AB - Finding minima of a real valued non-convex function over a high dimensional space is a major challenge in science. We provide evidence that some such functions that are defined on high dimensional domains have a narrow band of values whose pre-image contains the bulk of its critical points. This is in contrast with the low dimensional picture in which this band is wide. Our simulations agree with the previous theoretical work on spin glasses that proves the existence of such a band when the dimension of the domain tends to infinity. Furthermore our experiments on teacher-student networks with the MNIST dataset establish a similar phenomenon in deep networks. We finally observe that both the gradient descent and the stochastic gradient descent methods can reach this level within the same number of steps.

UR - http://www.scopus.com/inward/record.url?scp=85049682497&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85049682497&partnerID=8YFLogxK

M3 - Paper

AN - SCOPUS:85049682497

ER -