Variability in dopamine genes dissociates model-based and model-free reinforcement learning

Bradley B. Doll, Kevin G. Bath, Nathaniel D. Daw, Michael J. Frank

Research output: Contribution to journalArticle

Abstract

Considerable evidence suggests that multiple learning systems can drive behavior. Choice can proceed reflexively from previous actions and their associated outcomes, as captured by “model-free” learning algorithms, or flexibly from prospective consideration of outcomes that might occur, as captured by “model-based” learning algorithms. However, differential contributions of dopamine to these systems are poorly understood. Dopamine is widely thought to support model-free learning by modulating plasticity in striatum. Model-based learning may also be affected by these striatal effects, or by other dopaminergic effects elsewhere, notably on prefrontal working memory function. Indeed, prominent demonstrations linking striatal dopamine to putatively model-free learning did not rule out model-based effects, whereas other studies have reported dopaminergic modulation of verifiably model-based learning, but without distinguishing a prefrontal versus striatal locus. To clarify the relationships between dopamine, neural systems, and learning strategies, we combine a genetic association approach in humans with two well-studied reinforcement learning tasks: one isolating model-based from model-free behavior and the other sensitive to key aspects of striatal plasticity. Prefrontal function was indexed by a polymorphism in the COMT gene, differences of which reflect dopamine levels in the prefrontal cortex. This polymorphism has been associated with differences in prefrontal activity and working memory. Striatal function was indexed by a gene coding for DARPP-32, which is densely expressed in the striatum where it is necessary for synaptic plasticity. We found evidence for our hypothesis that variations in prefrontal dopamine relate to model-based learning, whereas variations in striatal dopamine function relate to model-free learning.

Original languageEnglish (US)
Pages (from-to)1211-1222
Number of pages12
JournalJournal of Neuroscience
Volume36
Issue number4
DOIs
StatePublished - Jan 27 2016

Fingerprint

Dopamine
Learning
Corpus Striatum
Genes
Short-Term Memory
Reinforcement (Psychology)
Dopamine Agents
Neuronal Plasticity
Prefrontal Cortex

Keywords

  • Decision-making
  • Dopamine
  • Genetics
  • Reinforcement learning

ASJC Scopus subject areas

  • Neuroscience(all)

Cite this

Variability in dopamine genes dissociates model-based and model-free reinforcement learning. / Doll, Bradley B.; Bath, Kevin G.; Daw, Nathaniel D.; Frank, Michael J.

In: Journal of Neuroscience, Vol. 36, No. 4, 27.01.2016, p. 1211-1222.

Research output: Contribution to journalArticle

Doll, Bradley B. ; Bath, Kevin G. ; Daw, Nathaniel D. ; Frank, Michael J. / Variability in dopamine genes dissociates model-based and model-free reinforcement learning. In: Journal of Neuroscience. 2016 ; Vol. 36, No. 4. pp. 1211-1222.
@article{a4e115da1b854dafb3787803cf59aeff,
title = "Variability in dopamine genes dissociates model-based and model-free reinforcement learning",
abstract = "Considerable evidence suggests that multiple learning systems can drive behavior. Choice can proceed reflexively from previous actions and their associated outcomes, as captured by “model-free” learning algorithms, or flexibly from prospective consideration of outcomes that might occur, as captured by “model-based” learning algorithms. However, differential contributions of dopamine to these systems are poorly understood. Dopamine is widely thought to support model-free learning by modulating plasticity in striatum. Model-based learning may also be affected by these striatal effects, or by other dopaminergic effects elsewhere, notably on prefrontal working memory function. Indeed, prominent demonstrations linking striatal dopamine to putatively model-free learning did not rule out model-based effects, whereas other studies have reported dopaminergic modulation of verifiably model-based learning, but without distinguishing a prefrontal versus striatal locus. To clarify the relationships between dopamine, neural systems, and learning strategies, we combine a genetic association approach in humans with two well-studied reinforcement learning tasks: one isolating model-based from model-free behavior and the other sensitive to key aspects of striatal plasticity. Prefrontal function was indexed by a polymorphism in the COMT gene, differences of which reflect dopamine levels in the prefrontal cortex. This polymorphism has been associated with differences in prefrontal activity and working memory. Striatal function was indexed by a gene coding for DARPP-32, which is densely expressed in the striatum where it is necessary for synaptic plasticity. We found evidence for our hypothesis that variations in prefrontal dopamine relate to model-based learning, whereas variations in striatal dopamine function relate to model-free learning.",
keywords = "Decision-making, Dopamine, Genetics, Reinforcement learning",
author = "Doll, {Bradley B.} and Bath, {Kevin G.} and Daw, {Nathaniel D.} and Frank, {Michael J.}",
year = "2016",
month = "1",
day = "27",
doi = "10.1523/JNEUROSCI.1901-15.2016",
language = "English (US)",
volume = "36",
pages = "1211--1222",
journal = "Journal of Neuroscience",
issn = "0270-6474",
publisher = "Society for Neuroscience",
number = "4",

}

TY - JOUR

T1 - Variability in dopamine genes dissociates model-based and model-free reinforcement learning

AU - Doll, Bradley B.

AU - Bath, Kevin G.

AU - Daw, Nathaniel D.

AU - Frank, Michael J.

PY - 2016/1/27

Y1 - 2016/1/27

N2 - Considerable evidence suggests that multiple learning systems can drive behavior. Choice can proceed reflexively from previous actions and their associated outcomes, as captured by “model-free” learning algorithms, or flexibly from prospective consideration of outcomes that might occur, as captured by “model-based” learning algorithms. However, differential contributions of dopamine to these systems are poorly understood. Dopamine is widely thought to support model-free learning by modulating plasticity in striatum. Model-based learning may also be affected by these striatal effects, or by other dopaminergic effects elsewhere, notably on prefrontal working memory function. Indeed, prominent demonstrations linking striatal dopamine to putatively model-free learning did not rule out model-based effects, whereas other studies have reported dopaminergic modulation of verifiably model-based learning, but without distinguishing a prefrontal versus striatal locus. To clarify the relationships between dopamine, neural systems, and learning strategies, we combine a genetic association approach in humans with two well-studied reinforcement learning tasks: one isolating model-based from model-free behavior and the other sensitive to key aspects of striatal plasticity. Prefrontal function was indexed by a polymorphism in the COMT gene, differences of which reflect dopamine levels in the prefrontal cortex. This polymorphism has been associated with differences in prefrontal activity and working memory. Striatal function was indexed by a gene coding for DARPP-32, which is densely expressed in the striatum where it is necessary for synaptic plasticity. We found evidence for our hypothesis that variations in prefrontal dopamine relate to model-based learning, whereas variations in striatal dopamine function relate to model-free learning.

AB - Considerable evidence suggests that multiple learning systems can drive behavior. Choice can proceed reflexively from previous actions and their associated outcomes, as captured by “model-free” learning algorithms, or flexibly from prospective consideration of outcomes that might occur, as captured by “model-based” learning algorithms. However, differential contributions of dopamine to these systems are poorly understood. Dopamine is widely thought to support model-free learning by modulating plasticity in striatum. Model-based learning may also be affected by these striatal effects, or by other dopaminergic effects elsewhere, notably on prefrontal working memory function. Indeed, prominent demonstrations linking striatal dopamine to putatively model-free learning did not rule out model-based effects, whereas other studies have reported dopaminergic modulation of verifiably model-based learning, but without distinguishing a prefrontal versus striatal locus. To clarify the relationships between dopamine, neural systems, and learning strategies, we combine a genetic association approach in humans with two well-studied reinforcement learning tasks: one isolating model-based from model-free behavior and the other sensitive to key aspects of striatal plasticity. Prefrontal function was indexed by a polymorphism in the COMT gene, differences of which reflect dopamine levels in the prefrontal cortex. This polymorphism has been associated with differences in prefrontal activity and working memory. Striatal function was indexed by a gene coding for DARPP-32, which is densely expressed in the striatum where it is necessary for synaptic plasticity. We found evidence for our hypothesis that variations in prefrontal dopamine relate to model-based learning, whereas variations in striatal dopamine function relate to model-free learning.

KW - Decision-making

KW - Dopamine

KW - Genetics

KW - Reinforcement learning

UR - http://www.scopus.com/inward/record.url?scp=84956825053&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84956825053&partnerID=8YFLogxK

U2 - 10.1523/JNEUROSCI.1901-15.2016

DO - 10.1523/JNEUROSCI.1901-15.2016

M3 - Article

VL - 36

SP - 1211

EP - 1222

JO - Journal of Neuroscience

JF - Journal of Neuroscience

SN - 0270-6474

IS - 4

ER -