### Abstract

One test of a new training algorithm is how well the algorithm generalizes from the training data to the test data. It is shown that a new training algorithm termed double backpropagation improves generalization by simultaneously minimizing the normal energy term found in backpropagation and an additional energy term that is related to the sum of the squares of the input derivatives (gradients). In normal backpropagation training, minimizing the energy function tends to push the input gradient to zero. However, this is not always possible. Double backpropagation explicitly pushes the input gradients to zero, making the minimum broader, and increases the generalization on the test data. The authors show the improvement over normal backpropagation on four candidate architectures and a training set of 320 handwritten numbers and a test set of size 180.

Original language | English (US) |
---|---|

Title of host publication | Proceedings. IJCNN - International Joint Conference on Neural Networks |

Editors | Anon |

Publisher | Publ by IEEE |

Pages | 145-150 |

Number of pages | 6 |

ISBN (Print) | 0780301641 |

State | Published - 1992 |

Event | International Joint Conference on Neural Networks - IJCNN-91-Seattle - Seattle, WA, USA Duration: Jul 8 1991 → Jul 12 1991 |

### Other

Other | International Joint Conference on Neural Networks - IJCNN-91-Seattle |
---|---|

City | Seattle, WA, USA |

Period | 7/8/91 → 7/12/91 |

### Fingerprint

### ASJC Scopus subject areas

- Engineering(all)

### Cite this

*Proceedings. IJCNN - International Joint Conference on Neural Networks*(pp. 145-150). Publ by IEEE.

**Double backpropagation increasing generalization performance.** / Drucker, Harris; LeCun, Yann.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

*Proceedings. IJCNN - International Joint Conference on Neural Networks.*Publ by IEEE, pp. 145-150, International Joint Conference on Neural Networks - IJCNN-91-Seattle, Seattle, WA, USA, 7/8/91.

}

TY - GEN

T1 - Double backpropagation increasing generalization performance

AU - Drucker, Harris

AU - LeCun, Yann

PY - 1992

Y1 - 1992

N2 - One test of a new training algorithm is how well the algorithm generalizes from the training data to the test data. It is shown that a new training algorithm termed double backpropagation improves generalization by simultaneously minimizing the normal energy term found in backpropagation and an additional energy term that is related to the sum of the squares of the input derivatives (gradients). In normal backpropagation training, minimizing the energy function tends to push the input gradient to zero. However, this is not always possible. Double backpropagation explicitly pushes the input gradients to zero, making the minimum broader, and increases the generalization on the test data. The authors show the improvement over normal backpropagation on four candidate architectures and a training set of 320 handwritten numbers and a test set of size 180.

AB - One test of a new training algorithm is how well the algorithm generalizes from the training data to the test data. It is shown that a new training algorithm termed double backpropagation improves generalization by simultaneously minimizing the normal energy term found in backpropagation and an additional energy term that is related to the sum of the squares of the input derivatives (gradients). In normal backpropagation training, minimizing the energy function tends to push the input gradient to zero. However, this is not always possible. Double backpropagation explicitly pushes the input gradients to zero, making the minimum broader, and increases the generalization on the test data. The authors show the improvement over normal backpropagation on four candidate architectures and a training set of 320 handwritten numbers and a test set of size 180.

UR - http://www.scopus.com/inward/record.url?scp=0026711366&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=0026711366&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:0026711366

SN - 0780301641

SP - 145

EP - 150

BT - Proceedings. IJCNN - International Joint Conference on Neural Networks

A2 - Anon, null

PB - Publ by IEEE

ER -