Comparing Dynamics: Deep Neural Networks versus Glassy Systems

Marco Baity-Jest, Levcnt Sagun, Geiger Mario, Stefano Spiglery, Gerard Ben Arous, Chiara Cammarota, Yann LeCun, Matthieu Vvyart, Giu Jio Biroli

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

We analyze numerically the training dynamics of deep neural networks (DNN) by using methods developed in statistical physics of glassy systems. The two main issues we address are (1) the complexity of the loss landscape and of the dynamics within it, and (2) to what extent DNNs share similarities with glassy systems. Our findings, obtained for different architectures and datasets, suggest that during the training process the dynamics slows down because of an increasingly large number of flat directions. At large limes, when the loss is approaching zero, the system diffuses at the bottom of the landscape. Despite some similarities with the dynamics of mean-field glassy systems, in particular, the absence of barrier crossing, we find distinctive dynamical behaviors in the two cases, showing that the statistical properties of the corresponding loss and energy landscapes arc different. In contrast, when the network is under-parametrized we observe a typical glassy behavior, thus suggesting the existence of different phases depending on whether the network is under-parametrized or over-parametrized.

Original languageEnglish (US)
Title of host publication35th International Conference on Machine Learning, ICML 2018
EditorsAndreas Krause, Jennifer Dy
PublisherInternational Machine Learning Society (IMLS)
Pages526-535
Number of pages10
Volume1
ISBN (Electronic)9781510867963
StatePublished - Jan 1 2018
Event35th International Conference on Machine Learning, ICML 2018 - Stockholm, Sweden
Duration: Jul 10 2018Jul 15 2018

Other

Other35th International Conference on Machine Learning, ICML 2018
CountrySweden
CityStockholm
Period7/10/187/15/18

    Fingerprint

ASJC Scopus subject areas

  • Computational Theory and Mathematics
  • Human-Computer Interaction
  • Software

Cite this

Baity-Jest, M., Sagun, L., Mario, G., Spiglery, S., Ben Arous, G., Cammarota, C., LeCun, Y., Vvyart, M., & Biroli, G. J. (2018). Comparing Dynamics: Deep Neural Networks versus Glassy Systems. In A. Krause, & J. Dy (Eds.), 35th International Conference on Machine Learning, ICML 2018 (Vol. 1, pp. 526-535). International Machine Learning Society (IMLS).