The Curse of Planning

Dissecting Multiple Reinforcement-Learning Systems by Taxing the Central Executive

A. Ross Otto, Samuel J. Gershman, Arthur B. Markman, Nathaniel D. Daw

Research output: Contribution to journalArticle

Abstract

A number of accounts of human and animal behavior posit the operation of parallel and competing valuation systems in the control of choice behavior. In these accounts, a flexible but computationally expensive model-based reinforcement-learning system has been contrasted with a less flexible but more efficient model-free reinforcement-learning system. The factors governing which system controls behavior-and under what circumstances-are still unclear. Following the hypothesis that model-based reinforcement learning requires cognitive resources, we demonstrated that having human decision makers perform a demanding secondary task engenders increased reliance on a model-free reinforcement-learning strategy. Further, we showed that, across trials, people negotiate the trade-off between the two systems dynamically as a function of concurrent executive-function demands, and people's choice latencies reflect the computational expenses of the strategy they employ. These results demonstrate that competition between multiple learning systems can be controlled on a trial-by-trial basis by modulating the availability of cognitive resources.

Original languageEnglish (US)
Pages (from-to)751-761
Number of pages11
JournalPsychological Science
Volume24
Issue number5
DOIs
StatePublished - May 2013

Fingerprint

Learning
Choice Behavior
Animal Behavior
Behavior Control
Executive Function
Reinforcement (Psychology)
Central Executive
Planning
Learning Systems
Curse
Reinforcement Learning
Cognitive Resources
Learning Strategies
Reliance
Computational
Human Behavior
Latency
Controlled

Keywords

  • cognitive neuroscience
  • decision making

ASJC Scopus subject areas

  • Psychology(all)
  • Arts and Humanities (miscellaneous)

Cite this

The Curse of Planning : Dissecting Multiple Reinforcement-Learning Systems by Taxing the Central Executive. / Otto, A. Ross; Gershman, Samuel J.; Markman, Arthur B.; Daw, Nathaniel D.

In: Psychological Science, Vol. 24, No. 5, 05.2013, p. 751-761.

Research output: Contribution to journalArticle

Otto, A. Ross ; Gershman, Samuel J. ; Markman, Arthur B. ; Daw, Nathaniel D. / The Curse of Planning : Dissecting Multiple Reinforcement-Learning Systems by Taxing the Central Executive. In: Psychological Science. 2013 ; Vol. 24, No. 5. pp. 751-761.
@article{7c486c91ccfd4ab7aa909b6cef41ac14,
title = "The Curse of Planning: Dissecting Multiple Reinforcement-Learning Systems by Taxing the Central Executive",
abstract = "A number of accounts of human and animal behavior posit the operation of parallel and competing valuation systems in the control of choice behavior. In these accounts, a flexible but computationally expensive model-based reinforcement-learning system has been contrasted with a less flexible but more efficient model-free reinforcement-learning system. The factors governing which system controls behavior-and under what circumstances-are still unclear. Following the hypothesis that model-based reinforcement learning requires cognitive resources, we demonstrated that having human decision makers perform a demanding secondary task engenders increased reliance on a model-free reinforcement-learning strategy. Further, we showed that, across trials, people negotiate the trade-off between the two systems dynamically as a function of concurrent executive-function demands, and people's choice latencies reflect the computational expenses of the strategy they employ. These results demonstrate that competition between multiple learning systems can be controlled on a trial-by-trial basis by modulating the availability of cognitive resources.",
keywords = "cognitive neuroscience, decision making",
author = "Otto, {A. Ross} and Gershman, {Samuel J.} and Markman, {Arthur B.} and Daw, {Nathaniel D.}",
year = "2013",
month = "5",
doi = "10.1177/0956797612463080",
language = "English (US)",
volume = "24",
pages = "751--761",
journal = "Psychological Science",
issn = "0956-7976",
publisher = "SAGE Publications Inc.",
number = "5",

}

TY - JOUR

T1 - The Curse of Planning

T2 - Dissecting Multiple Reinforcement-Learning Systems by Taxing the Central Executive

AU - Otto, A. Ross

AU - Gershman, Samuel J.

AU - Markman, Arthur B.

AU - Daw, Nathaniel D.

PY - 2013/5

Y1 - 2013/5

N2 - A number of accounts of human and animal behavior posit the operation of parallel and competing valuation systems in the control of choice behavior. In these accounts, a flexible but computationally expensive model-based reinforcement-learning system has been contrasted with a less flexible but more efficient model-free reinforcement-learning system. The factors governing which system controls behavior-and under what circumstances-are still unclear. Following the hypothesis that model-based reinforcement learning requires cognitive resources, we demonstrated that having human decision makers perform a demanding secondary task engenders increased reliance on a model-free reinforcement-learning strategy. Further, we showed that, across trials, people negotiate the trade-off between the two systems dynamically as a function of concurrent executive-function demands, and people's choice latencies reflect the computational expenses of the strategy they employ. These results demonstrate that competition between multiple learning systems can be controlled on a trial-by-trial basis by modulating the availability of cognitive resources.

AB - A number of accounts of human and animal behavior posit the operation of parallel and competing valuation systems in the control of choice behavior. In these accounts, a flexible but computationally expensive model-based reinforcement-learning system has been contrasted with a less flexible but more efficient model-free reinforcement-learning system. The factors governing which system controls behavior-and under what circumstances-are still unclear. Following the hypothesis that model-based reinforcement learning requires cognitive resources, we demonstrated that having human decision makers perform a demanding secondary task engenders increased reliance on a model-free reinforcement-learning strategy. Further, we showed that, across trials, people negotiate the trade-off between the two systems dynamically as a function of concurrent executive-function demands, and people's choice latencies reflect the computational expenses of the strategy they employ. These results demonstrate that competition between multiple learning systems can be controlled on a trial-by-trial basis by modulating the availability of cognitive resources.

KW - cognitive neuroscience

KW - decision making

UR - http://www.scopus.com/inward/record.url?scp=84877341847&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84877341847&partnerID=8YFLogxK

U2 - 10.1177/0956797612463080

DO - 10.1177/0956797612463080

M3 - Article

VL - 24

SP - 751

EP - 761

JO - Psychological Science

JF - Psychological Science

SN - 0956-7976

IS - 5

ER -