Tracking the world state with recurrent entity networks

Mikael Henaff, Jason Weston, Arthur Szlam, Antoine Bordes, Yann LeCun

Research output: Contribution to conferencePaper

Abstract

We introduce a new model, the Recurrent Entity Network (EntNet). It is equipped with a dynamic long-term memory which allows it to maintain and update a representation of the state of the world as it receives new data. For language understanding tasks, it can reason on-the-fly as it reads text, not just when it is required to answer a question or respond as is the case for a Memory Network (Sukhbaatar et al., 2015). Like a Neural Turing Machine or Differentiable Neural Computer (Graves et al., 2014; 2016) it maintains a fixed size memory and can learn to perform location and content-based read and write operations. However, unlike those models it has a simple parallel architecture in which several memory locations can be updated simultaneously. The EntNet sets a new state-of-the-art on the bAbI tasks, and is the first method to solve all the tasks in the 10k training examples setting. We also demonstrate that it can solve a reasoning task which requires a large number of supporting facts, which other methods are not able to solve, and can generalize past its training horizon. It can also be practically used on large scale datasets such as Children's Book Test, where it obtains competitive performance, reading the story in a single pass.

Original languageEnglish (US)
StatePublished - Jan 1 2019
Event5th International Conference on Learning Representations, ICLR 2017 - Toulon, France
Duration: Apr 24 2017Apr 26 2017

Conference

Conference5th International Conference on Learning Representations, ICLR 2017
CountryFrance
CityToulon
Period4/24/174/26/17

Fingerprint

Data storage equipment
Turing machines
Parallel architectures
Entity
language
performance
Language Understanding
Turing Machine
Long-term Memory

ASJC Scopus subject areas

  • Education
  • Computer Science Applications
  • Linguistics and Language
  • Language and Linguistics

Cite this

Henaff, M., Weston, J., Szlam, A., Bordes, A., & LeCun, Y. (2019). Tracking the world state with recurrent entity networks. Paper presented at 5th International Conference on Learning Representations, ICLR 2017, Toulon, France.

Tracking the world state with recurrent entity networks. / Henaff, Mikael; Weston, Jason; Szlam, Arthur; Bordes, Antoine; LeCun, Yann.

2019. Paper presented at 5th International Conference on Learning Representations, ICLR 2017, Toulon, France.

Research output: Contribution to conferencePaper

Henaff, M, Weston, J, Szlam, A, Bordes, A & LeCun, Y 2019, 'Tracking the world state with recurrent entity networks' Paper presented at 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, 4/24/17 - 4/26/17, .
Henaff M, Weston J, Szlam A, Bordes A, LeCun Y. Tracking the world state with recurrent entity networks. 2019. Paper presented at 5th International Conference on Learning Representations, ICLR 2017, Toulon, France.
Henaff, Mikael ; Weston, Jason ; Szlam, Arthur ; Bordes, Antoine ; LeCun, Yann. / Tracking the world state with recurrent entity networks. Paper presented at 5th International Conference on Learning Representations, ICLR 2017, Toulon, France.
@conference{e40e2147f6bb4e6e87c33740f8d96bb7,
title = "Tracking the world state with recurrent entity networks",
abstract = "We introduce a new model, the Recurrent Entity Network (EntNet). It is equipped with a dynamic long-term memory which allows it to maintain and update a representation of the state of the world as it receives new data. For language understanding tasks, it can reason on-the-fly as it reads text, not just when it is required to answer a question or respond as is the case for a Memory Network (Sukhbaatar et al., 2015). Like a Neural Turing Machine or Differentiable Neural Computer (Graves et al., 2014; 2016) it maintains a fixed size memory and can learn to perform location and content-based read and write operations. However, unlike those models it has a simple parallel architecture in which several memory locations can be updated simultaneously. The EntNet sets a new state-of-the-art on the bAbI tasks, and is the first method to solve all the tasks in the 10k training examples setting. We also demonstrate that it can solve a reasoning task which requires a large number of supporting facts, which other methods are not able to solve, and can generalize past its training horizon. It can also be practically used on large scale datasets such as Children's Book Test, where it obtains competitive performance, reading the story in a single pass.",
author = "Mikael Henaff and Jason Weston and Arthur Szlam and Antoine Bordes and Yann LeCun",
year = "2019",
month = "1",
day = "1",
language = "English (US)",
note = "5th International Conference on Learning Representations, ICLR 2017 ; Conference date: 24-04-2017 Through 26-04-2017",

}

TY - CONF

T1 - Tracking the world state with recurrent entity networks

AU - Henaff, Mikael

AU - Weston, Jason

AU - Szlam, Arthur

AU - Bordes, Antoine

AU - LeCun, Yann

PY - 2019/1/1

Y1 - 2019/1/1

N2 - We introduce a new model, the Recurrent Entity Network (EntNet). It is equipped with a dynamic long-term memory which allows it to maintain and update a representation of the state of the world as it receives new data. For language understanding tasks, it can reason on-the-fly as it reads text, not just when it is required to answer a question or respond as is the case for a Memory Network (Sukhbaatar et al., 2015). Like a Neural Turing Machine or Differentiable Neural Computer (Graves et al., 2014; 2016) it maintains a fixed size memory and can learn to perform location and content-based read and write operations. However, unlike those models it has a simple parallel architecture in which several memory locations can be updated simultaneously. The EntNet sets a new state-of-the-art on the bAbI tasks, and is the first method to solve all the tasks in the 10k training examples setting. We also demonstrate that it can solve a reasoning task which requires a large number of supporting facts, which other methods are not able to solve, and can generalize past its training horizon. It can also be practically used on large scale datasets such as Children's Book Test, where it obtains competitive performance, reading the story in a single pass.

AB - We introduce a new model, the Recurrent Entity Network (EntNet). It is equipped with a dynamic long-term memory which allows it to maintain and update a representation of the state of the world as it receives new data. For language understanding tasks, it can reason on-the-fly as it reads text, not just when it is required to answer a question or respond as is the case for a Memory Network (Sukhbaatar et al., 2015). Like a Neural Turing Machine or Differentiable Neural Computer (Graves et al., 2014; 2016) it maintains a fixed size memory and can learn to perform location and content-based read and write operations. However, unlike those models it has a simple parallel architecture in which several memory locations can be updated simultaneously. The EntNet sets a new state-of-the-art on the bAbI tasks, and is the first method to solve all the tasks in the 10k training examples setting. We also demonstrate that it can solve a reasoning task which requires a large number of supporting facts, which other methods are not able to solve, and can generalize past its training horizon. It can also be practically used on large scale datasets such as Children's Book Test, where it obtains competitive performance, reading the story in a single pass.

UR - http://www.scopus.com/inward/record.url?scp=85071044353&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85071044353&partnerID=8YFLogxK

M3 - Paper

ER -