Evolving memory cell structures for sequence learning

Justin Bayer, Daan Wierstra, Julian Togelius, Jürgen Schmidhuber

    Research output: Chapter in Book/Report/Conference proceedingConference contribution

    Abstract

    Long Short-Term Memory (LSTM) is one of the best recent supervised sequence learning methods. Using gradient descent, it trains memory cells represented as differentiable computational graph structures. Interestingly, LSTM's cell structure seems somewhat arbitrary. In this paper we optimize its computational structure using a multi-objective evolutionary algorithm. The fitness function reflects the structure's usefulness for learning various formal languages. The evolved cells help to understand crucial features that aid sequence learning.

    Original languageEnglish (US)
    Title of host publicationArtificial Neural Networks - ICANN 2009 - 19th International Conference, Proceedings
    Pages755-764
    Number of pages10
    EditionPART 2
    DOIs
    StatePublished - Nov 27 2009
    Event19th International Conference on Artificial Neural Networks, ICANN 2009 - Limassol, Cyprus
    Duration: Sep 14 2009Sep 17 2009

    Publication series

    NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
    NumberPART 2
    Volume5769 LNCS
    ISSN (Print)0302-9743
    ISSN (Electronic)1611-3349

    Other

    Other19th International Conference on Artificial Neural Networks, ICANN 2009
    CountryCyprus
    CityLimassol
    Period9/14/099/17/09

      Fingerprint

    ASJC Scopus subject areas

    • Theoretical Computer Science
    • Computer Science(all)

    Cite this

    Bayer, J., Wierstra, D., Togelius, J., & Schmidhuber, J. (2009). Evolving memory cell structures for sequence learning. In Artificial Neural Networks - ICANN 2009 - 19th International Conference, Proceedings (PART 2 ed., pp. 755-764). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 5769 LNCS, No. PART 2). https://doi.org/10.1007/978-3-642-04277-5_76