Cognitive Control Predicts Use of Model-based Reinforcement Learning

A. Ross Otto, Anya Skatova, Seth Madlon-Kay, Nathaniel D. Daw

Research output: Contribution to journalArticle

Abstract

Accounts of decision-making and its neural substrates have long posited the operation of separate, competing valuation systems in the control of choice behavior. Recent theoretical and experimental work suggest that this classic distinction between behaviorally and neurally dissociable systems for habitual and goal-directed (or more generally, automatic and controlled) choice may arise from two computational strategies for reinforcement learning (RL), called model-free and model-based RL, but the cognitive or computational processes by which one system may dominate over the other in the control of behavior is amatter of ongoing investigation. To elucidate this question, we leverage the theoretical framework of cognitive control, demonstrating that individual differences in utilization of goal-related contextual information—in the service of overcoming habitual, stimulusdriven responses—in established cognitive control paradigms predict model-based behavior in a separate, sequential choice task. The behavioral correspondence between cognitive control and model-based RL compellingly suggests that a common set of processes may underpin the two behaviors. In particular, computational mechanisms originally proposed to underlie controlled behavior may be applicable to understanding the interactions betweenmodel-based andmodel-free choice behavior.

Original languageEnglish (US)
Pages (from-to)319-333
Number of pages15
JournalJournal of Cognitive Neuroscience
Volume27
Issue number2
DOIs
StatePublished - Mar 1 2014

Fingerprint

reinforcement
Choice Behavior
Learning
learning
Behavior Control
Individuality
Decision Making
Reinforcement (Psychology)
Reinforcement Learning
Cognitive Control
utilization
paradigm
decision making
interaction
Computational
Habitual
Controlled

ASJC Scopus subject areas

  • Cognitive Neuroscience
  • Language and Linguistics
  • Linguistics and Language

Cite this

Cognitive Control Predicts Use of Model-based Reinforcement Learning. / Otto, A. Ross; Skatova, Anya; Madlon-Kay, Seth; Daw, Nathaniel D.

In: Journal of Cognitive Neuroscience, Vol. 27, No. 2, 01.03.2014, p. 319-333.

Research output: Contribution to journalArticle

Otto, A. Ross ; Skatova, Anya ; Madlon-Kay, Seth ; Daw, Nathaniel D. / Cognitive Control Predicts Use of Model-based Reinforcement Learning. In: Journal of Cognitive Neuroscience. 2014 ; Vol. 27, No. 2. pp. 319-333.
@article{de30fede0e694ea181e0570ddc78b608,
title = "Cognitive Control Predicts Use of Model-based Reinforcement Learning",
abstract = "Accounts of decision-making and its neural substrates have long posited the operation of separate, competing valuation systems in the control of choice behavior. Recent theoretical and experimental work suggest that this classic distinction between behaviorally and neurally dissociable systems for habitual and goal-directed (or more generally, automatic and controlled) choice may arise from two computational strategies for reinforcement learning (RL), called model-free and model-based RL, but the cognitive or computational processes by which one system may dominate over the other in the control of behavior is amatter of ongoing investigation. To elucidate this question, we leverage the theoretical framework of cognitive control, demonstrating that individual differences in utilization of goal-related contextual information—in the service of overcoming habitual, stimulusdriven responses—in established cognitive control paradigms predict model-based behavior in a separate, sequential choice task. The behavioral correspondence between cognitive control and model-based RL compellingly suggests that a common set of processes may underpin the two behaviors. In particular, computational mechanisms originally proposed to underlie controlled behavior may be applicable to understanding the interactions betweenmodel-based andmodel-free choice behavior.",
author = "Otto, {A. Ross} and Anya Skatova and Seth Madlon-Kay and Daw, {Nathaniel D.}",
year = "2014",
month = "3",
day = "1",
doi = "10.1162/jocn_a_00709",
language = "English (US)",
volume = "27",
pages = "319--333",
journal = "Journal of Cognitive Neuroscience",
issn = "0898-929X",
publisher = "MIT Press Journals",
number = "2",

}

TY - JOUR

T1 - Cognitive Control Predicts Use of Model-based Reinforcement Learning

AU - Otto, A. Ross

AU - Skatova, Anya

AU - Madlon-Kay, Seth

AU - Daw, Nathaniel D.

PY - 2014/3/1

Y1 - 2014/3/1

N2 - Accounts of decision-making and its neural substrates have long posited the operation of separate, competing valuation systems in the control of choice behavior. Recent theoretical and experimental work suggest that this classic distinction between behaviorally and neurally dissociable systems for habitual and goal-directed (or more generally, automatic and controlled) choice may arise from two computational strategies for reinforcement learning (RL), called model-free and model-based RL, but the cognitive or computational processes by which one system may dominate over the other in the control of behavior is amatter of ongoing investigation. To elucidate this question, we leverage the theoretical framework of cognitive control, demonstrating that individual differences in utilization of goal-related contextual information—in the service of overcoming habitual, stimulusdriven responses—in established cognitive control paradigms predict model-based behavior in a separate, sequential choice task. The behavioral correspondence between cognitive control and model-based RL compellingly suggests that a common set of processes may underpin the two behaviors. In particular, computational mechanisms originally proposed to underlie controlled behavior may be applicable to understanding the interactions betweenmodel-based andmodel-free choice behavior.

AB - Accounts of decision-making and its neural substrates have long posited the operation of separate, competing valuation systems in the control of choice behavior. Recent theoretical and experimental work suggest that this classic distinction between behaviorally and neurally dissociable systems for habitual and goal-directed (or more generally, automatic and controlled) choice may arise from two computational strategies for reinforcement learning (RL), called model-free and model-based RL, but the cognitive or computational processes by which one system may dominate over the other in the control of behavior is amatter of ongoing investigation. To elucidate this question, we leverage the theoretical framework of cognitive control, demonstrating that individual differences in utilization of goal-related contextual information—in the service of overcoming habitual, stimulusdriven responses—in established cognitive control paradigms predict model-based behavior in a separate, sequential choice task. The behavioral correspondence between cognitive control and model-based RL compellingly suggests that a common set of processes may underpin the two behaviors. In particular, computational mechanisms originally proposed to underlie controlled behavior may be applicable to understanding the interactions betweenmodel-based andmodel-free choice behavior.

UR - http://www.scopus.com/inward/record.url?scp=84919958707&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84919958707&partnerID=8YFLogxK

U2 - 10.1162/jocn_a_00709

DO - 10.1162/jocn_a_00709

M3 - Article

AN - SCOPUS:84919958707

VL - 27

SP - 319

EP - 333

JO - Journal of Cognitive Neuroscience

JF - Journal of Cognitive Neuroscience

SN - 0898-929X

IS - 2

ER -