Double backpropagation increasing generalization performance

Harris Drucker, Yann LeCun

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

One test of a new training algorithm is how well the algorithm generalizes from the training data to the test data. It is shown that a new training algorithm termed double backpropagation improves generalization by simultaneously minimizing the normal energy term found in backpropagation and an additional energy term that is related to the sum of the squares of the input derivatives (gradients). In normal backpropagation training, minimizing the energy function tends to push the input gradient to zero. However, this is not always possible. Double backpropagation explicitly pushes the input gradients to zero, making the minimum broader, and increases the generalization on the test data. The authors show the improvement over normal backpropagation on four candidate architectures and a training set of 320 handwritten numbers and a test set of size 180.

Original languageEnglish (US)
Title of host publicationProceedings. IJCNN - International Joint Conference on Neural Networks
Editors Anon
PublisherPubl by IEEE
Pages145-150
Number of pages6
ISBN (Print)0780301641
StatePublished - 1992
EventInternational Joint Conference on Neural Networks - IJCNN-91-Seattle - Seattle, WA, USA
Duration: Jul 8 1991Jul 12 1991

Other

OtherInternational Joint Conference on Neural Networks - IJCNN-91-Seattle
CitySeattle, WA, USA
Period7/8/917/12/91

Fingerprint

Backpropagation
Derivatives

ASJC Scopus subject areas

  • Engineering(all)

Cite this

Drucker, H., & LeCun, Y. (1992). Double backpropagation increasing generalization performance. In Anon (Ed.), Proceedings. IJCNN - International Joint Conference on Neural Networks (pp. 145-150). Publ by IEEE.

Double backpropagation increasing generalization performance. / Drucker, Harris; LeCun, Yann.

Proceedings. IJCNN - International Joint Conference on Neural Networks. ed. / Anon. Publ by IEEE, 1992. p. 145-150.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Drucker, H & LeCun, Y 1992, Double backpropagation increasing generalization performance. in Anon (ed.), Proceedings. IJCNN - International Joint Conference on Neural Networks. Publ by IEEE, pp. 145-150, International Joint Conference on Neural Networks - IJCNN-91-Seattle, Seattle, WA, USA, 7/8/91.
Drucker H, LeCun Y. Double backpropagation increasing generalization performance. In Anon, editor, Proceedings. IJCNN - International Joint Conference on Neural Networks. Publ by IEEE. 1992. p. 145-150
Drucker, Harris ; LeCun, Yann. / Double backpropagation increasing generalization performance. Proceedings. IJCNN - International Joint Conference on Neural Networks. editor / Anon. Publ by IEEE, 1992. pp. 145-150
@inproceedings{570552d4fca942f5901d1aa6fe7c75c5,
title = "Double backpropagation increasing generalization performance",
abstract = "One test of a new training algorithm is how well the algorithm generalizes from the training data to the test data. It is shown that a new training algorithm termed double backpropagation improves generalization by simultaneously minimizing the normal energy term found in backpropagation and an additional energy term that is related to the sum of the squares of the input derivatives (gradients). In normal backpropagation training, minimizing the energy function tends to push the input gradient to zero. However, this is not always possible. Double backpropagation explicitly pushes the input gradients to zero, making the minimum broader, and increases the generalization on the test data. The authors show the improvement over normal backpropagation on four candidate architectures and a training set of 320 handwritten numbers and a test set of size 180.",
author = "Harris Drucker and Yann LeCun",
year = "1992",
language = "English (US)",
isbn = "0780301641",
pages = "145--150",
editor = "Anon",
booktitle = "Proceedings. IJCNN - International Joint Conference on Neural Networks",
publisher = "Publ by IEEE",

}

TY - GEN

T1 - Double backpropagation increasing generalization performance

AU - Drucker, Harris

AU - LeCun, Yann

PY - 1992

Y1 - 1992

N2 - One test of a new training algorithm is how well the algorithm generalizes from the training data to the test data. It is shown that a new training algorithm termed double backpropagation improves generalization by simultaneously minimizing the normal energy term found in backpropagation and an additional energy term that is related to the sum of the squares of the input derivatives (gradients). In normal backpropagation training, minimizing the energy function tends to push the input gradient to zero. However, this is not always possible. Double backpropagation explicitly pushes the input gradients to zero, making the minimum broader, and increases the generalization on the test data. The authors show the improvement over normal backpropagation on four candidate architectures and a training set of 320 handwritten numbers and a test set of size 180.

AB - One test of a new training algorithm is how well the algorithm generalizes from the training data to the test data. It is shown that a new training algorithm termed double backpropagation improves generalization by simultaneously minimizing the normal energy term found in backpropagation and an additional energy term that is related to the sum of the squares of the input derivatives (gradients). In normal backpropagation training, minimizing the energy function tends to push the input gradient to zero. However, this is not always possible. Double backpropagation explicitly pushes the input gradients to zero, making the minimum broader, and increases the generalization on the test data. The authors show the improvement over normal backpropagation on four candidate architectures and a training set of 320 handwritten numbers and a test set of size 180.

UR - http://www.scopus.com/inward/record.url?scp=0026711366&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=0026711366&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:0026711366

SN - 0780301641

SP - 145

EP - 150

BT - Proceedings. IJCNN - International Joint Conference on Neural Networks

A2 - Anon, null

PB - Publ by IEEE

ER -