Dynamic neural turing machine with continuous and discrete addressing schemes

Caglar Gulcehre, Sarath Chandar, Kyunghyun Cho, Yoshua Bengio

Research output: Contribution to journalArticle

Abstract

We extend the neural Turing machine (NTM) model into a dynamic neural Turing machine (D-NTM) by introducing trainable address vectors. This addressing scheme maintains for each memory cell twoseparate vectors, content and address vectors. This allows the D-NTM to learn a wide variety of location-based addressing strategies, including both linear and nonlinear ones.We implement theD-NTMwith both continuous and discrete read and write mechanisms.We investigate the mechanisms and effects of learning to read andwrite into a memory through experiments on Facebook bAbI tasks using both a feedforward and GRU controller. We provide extensive analysis of ourmodel and compare different variations of neural Turing machines on this task. We show that our model outperforms long short-term memory and NTM variants. We provide further experimental results on the sequential pMNIST, Stanford Natural Language Inference, associative recall, and copy tasks.

Original languageEnglish (US)
Pages (from-to)857-884
Number of pages28
JournalNeural computation
Volume30
Issue number4
DOIs
StatePublished - Apr 1 2018

    Fingerprint

ASJC Scopus subject areas

  • Arts and Humanities (miscellaneous)
  • Cognitive Neuroscience

Cite this