Eigenvalues of covariance matrices: Application to neural-network learning

Yann LeCun, Ido Kanter, Sara A. Solla

Research output: Contribution to journalArticle

Abstract

The learing time of a simple neural-network model is obtained through an analytic computation of the eigenvalue spectrum for the Hessian matrix, which describes the second-order properties of the objective function in the space of coupling coefficients. The results are generic for symmetric matrices obtained by summing outer products of random vectors. The form of the eigenvalue distribution suggests new techniques for accelerating the learning process, and provides a theoretical justification for the choice of centered versus biased state variables.

Original languageEnglish (US)
Pages (from-to)2396-2399
Number of pages4
JournalPhysical Review Letters
Volume66
Issue number18
DOIs
StatePublished - 1991

Fingerprint

learning
eigenvalues
Hessian matrices
coupling coefficients
products
matrices

ASJC Scopus subject areas

  • Physics and Astronomy(all)

Cite this

Eigenvalues of covariance matrices : Application to neural-network learning. / LeCun, Yann; Kanter, Ido; Solla, Sara A.

In: Physical Review Letters, Vol. 66, No. 18, 1991, p. 2396-2399.

Research output: Contribution to journalArticle

LeCun, Yann ; Kanter, Ido ; Solla, Sara A. / Eigenvalues of covariance matrices : Application to neural-network learning. In: Physical Review Letters. 1991 ; Vol. 66, No. 18. pp. 2396-2399.
@article{734ff179786b4ec5a0839c1c7e5143c0,
title = "Eigenvalues of covariance matrices: Application to neural-network learning",
abstract = "The learing time of a simple neural-network model is obtained through an analytic computation of the eigenvalue spectrum for the Hessian matrix, which describes the second-order properties of the objective function in the space of coupling coefficients. The results are generic for symmetric matrices obtained by summing outer products of random vectors. The form of the eigenvalue distribution suggests new techniques for accelerating the learning process, and provides a theoretical justification for the choice of centered versus biased state variables.",
author = "Yann LeCun and Ido Kanter and Solla, {Sara A.}",
year = "1991",
doi = "10.1103/PhysRevLett.66.2396",
language = "English (US)",
volume = "66",
pages = "2396--2399",
journal = "Physical Review Letters",
issn = "0031-9007",
publisher = "American Physical Society",
number = "18",

}

TY - JOUR

T1 - Eigenvalues of covariance matrices

T2 - Application to neural-network learning

AU - LeCun, Yann

AU - Kanter, Ido

AU - Solla, Sara A.

PY - 1991

Y1 - 1991

N2 - The learing time of a simple neural-network model is obtained through an analytic computation of the eigenvalue spectrum for the Hessian matrix, which describes the second-order properties of the objective function in the space of coupling coefficients. The results are generic for symmetric matrices obtained by summing outer products of random vectors. The form of the eigenvalue distribution suggests new techniques for accelerating the learning process, and provides a theoretical justification for the choice of centered versus biased state variables.

AB - The learing time of a simple neural-network model is obtained through an analytic computation of the eigenvalue spectrum for the Hessian matrix, which describes the second-order properties of the objective function in the space of coupling coefficients. The results are generic for symmetric matrices obtained by summing outer products of random vectors. The form of the eigenvalue distribution suggests new techniques for accelerating the learning process, and provides a theoretical justification for the choice of centered versus biased state variables.

UR - http://www.scopus.com/inward/record.url?scp=0000044667&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=0000044667&partnerID=8YFLogxK

U2 - 10.1103/PhysRevLett.66.2396

DO - 10.1103/PhysRevLett.66.2396

M3 - Article

AN - SCOPUS:0000044667

VL - 66

SP - 2396

EP - 2399

JO - Physical Review Letters

JF - Physical Review Letters

SN - 0031-9007

IS - 18

ER -