The algorithmic anatomy of model-based evaluation

Nathaniel D. Daw, Peter Dayan

Research output: Contribution to journalArticle

Abstract

Despite many debates in the first half of the twentieth century, it is now largely a truism that humans and other animals build models of their environments and use them for prediction and control. However, model-based (MB) reasoning presents severe computational challenges. Alternative, computationally simpler, model-free (MF) schemes have been suggested in the reinforcement learning literature, and have afforded influential accounts of behavioural and neural data. Here, we study the realization of MB calculations, and the ways that this might be woven together with MF values and evaluation methods. There are as yet mostly only hints in the literature as to the resulting tapestry, so we offer more preview than review.

Original languageEnglish (US)
JournalPhilosophical Transactions of the Royal Society B: Biological Sciences
Volume369
Issue number1655
DOIs
StatePublished - Nov 5 2014

Fingerprint

Anatomy
Animal Models
Learning
Reinforcement learning
learning
animal models
Animals
prediction
Reinforcement (Psychology)
methodology

Keywords

  • Model-based reasoning
  • Model-free reasoning
  • Monte Carlo tree search
  • Orbitofrontal cortex
  • Reinforcement learning
  • Striatum

ASJC Scopus subject areas

  • Agricultural and Biological Sciences(all)
  • Biochemistry, Genetics and Molecular Biology(all)

Cite this

The algorithmic anatomy of model-based evaluation. / Daw, Nathaniel D.; Dayan, Peter.

In: Philosophical Transactions of the Royal Society B: Biological Sciences, Vol. 369, No. 1655, 05.11.2014.

Research output: Contribution to journalArticle

@article{bd1fcf78f1d64293b1c2fe23766eca28,
title = "The algorithmic anatomy of model-based evaluation",
abstract = "Despite many debates in the first half of the twentieth century, it is now largely a truism that humans and other animals build models of their environments and use them for prediction and control. However, model-based (MB) reasoning presents severe computational challenges. Alternative, computationally simpler, model-free (MF) schemes have been suggested in the reinforcement learning literature, and have afforded influential accounts of behavioural and neural data. Here, we study the realization of MB calculations, and the ways that this might be woven together with MF values and evaluation methods. There are as yet mostly only hints in the literature as to the resulting tapestry, so we offer more preview than review.",
keywords = "Model-based reasoning, Model-free reasoning, Monte Carlo tree search, Orbitofrontal cortex, Reinforcement learning, Striatum",
author = "Daw, {Nathaniel D.} and Peter Dayan",
year = "2014",
month = "11",
day = "5",
doi = "10.1098/rstb.2013.0478",
language = "English (US)",
volume = "369",
journal = "Philosophical Transactions of the Royal Society B: Biological Sciences",
issn = "0962-8436",
publisher = "Royal Society of London",
number = "1655",

}

TY - JOUR

T1 - The algorithmic anatomy of model-based evaluation

AU - Daw, Nathaniel D.

AU - Dayan, Peter

PY - 2014/11/5

Y1 - 2014/11/5

N2 - Despite many debates in the first half of the twentieth century, it is now largely a truism that humans and other animals build models of their environments and use them for prediction and control. However, model-based (MB) reasoning presents severe computational challenges. Alternative, computationally simpler, model-free (MF) schemes have been suggested in the reinforcement learning literature, and have afforded influential accounts of behavioural and neural data. Here, we study the realization of MB calculations, and the ways that this might be woven together with MF values and evaluation methods. There are as yet mostly only hints in the literature as to the resulting tapestry, so we offer more preview than review.

AB - Despite many debates in the first half of the twentieth century, it is now largely a truism that humans and other animals build models of their environments and use them for prediction and control. However, model-based (MB) reasoning presents severe computational challenges. Alternative, computationally simpler, model-free (MF) schemes have been suggested in the reinforcement learning literature, and have afforded influential accounts of behavioural and neural data. Here, we study the realization of MB calculations, and the ways that this might be woven together with MF values and evaluation methods. There are as yet mostly only hints in the literature as to the resulting tapestry, so we offer more preview than review.

KW - Model-based reasoning

KW - Model-free reasoning

KW - Monte Carlo tree search

KW - Orbitofrontal cortex

KW - Reinforcement learning

KW - Striatum

UR - http://www.scopus.com/inward/record.url?scp=84907545889&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84907545889&partnerID=8YFLogxK

U2 - 10.1098/rstb.2013.0478

DO - 10.1098/rstb.2013.0478

M3 - Article

VL - 369

JO - Philosophical Transactions of the Royal Society B: Biological Sciences

JF - Philosophical Transactions of the Royal Society B: Biological Sciences

SN - 0962-8436

IS - 1655

ER -