Tonic dopamine modulates exploitation of reward learning

Jeff A. Beeler, Nathaniel Daw, Cristianne R M Frazier, Xiaoxi Zhuang

Research output: Contribution to journalArticle

Abstract

The impact of dopamine on adaptive behavior in a naturalistic environment is largely unexamined. Experimental work suggests that phasic dopamine is central to reinforcement learning whereas tonic dopamine may modulate performance without altering learning per se; however, this idea has not been developed formally or integrated with computational models of dopamine function. We quantitatively evaluate the role of tonic dopamine in these functions by studying the behavior of hyperdopaminergic DAT knockdown mice in an instrumental task in a seminaturalistic homecage environment. In this "closed economy" paradigm, subjects earn all of their food by pressing either of two levers, but the relative cost for food on each lever shifts frequently. Compared to wild-type mice, hyperdopaminergic mice allocate more lever presses on high-cost levers, thus working harder to earn a given amount of food and maintain their body weight. However, both groups show a similarly quick reaction to shifts in lever cost, suggesting that the hyperdominergic mice are not slower at detecting changes, as with a learning deficit. We fit the lever choice data using reinforcement learning models to assess the distinction between acquisition and expression the models formalize. In these analyses, hyperdopaminergic mice displayed normal learning from recent reward history but diminished capacity to exploit this learning: a reduced coupling between choice and reward history. These data suggest that dopamine modulates the degree to which prior learning biases action selection and consequently alters the expression of learned, motivated behavior.

Original languageEnglish (US)
Article number170
JournalFrontiers in Behavioral Neuroscience
Volume4
Issue numberNOV
DOIs
StatePublished - Nov 4 2010

Fingerprint

Reward
Dopamine
Learning
Costs and Cost Analysis
Food
History
Selection Bias
Psychological Adaptation
Body Weight

Keywords

  • Behavioral flexibility
  • DAT knock-down
  • Dopamine
  • Environmental adaptation
  • Explore-exploit
  • Reinforcement learning

ASJC Scopus subject areas

  • Behavioral Neuroscience
  • Cognitive Neuroscience
  • Neuropsychology and Physiological Psychology

Cite this

Tonic dopamine modulates exploitation of reward learning. / Beeler, Jeff A.; Daw, Nathaniel; Frazier, Cristianne R M; Zhuang, Xiaoxi.

In: Frontiers in Behavioral Neuroscience, Vol. 4, No. NOV, 170, 04.11.2010.

Research output: Contribution to journalArticle

Beeler, Jeff A. ; Daw, Nathaniel ; Frazier, Cristianne R M ; Zhuang, Xiaoxi. / Tonic dopamine modulates exploitation of reward learning. In: Frontiers in Behavioral Neuroscience. 2010 ; Vol. 4, No. NOV.
@article{1db026e0a30d45c5b3daaa3885f55719,
title = "Tonic dopamine modulates exploitation of reward learning",
abstract = "The impact of dopamine on adaptive behavior in a naturalistic environment is largely unexamined. Experimental work suggests that phasic dopamine is central to reinforcement learning whereas tonic dopamine may modulate performance without altering learning per se; however, this idea has not been developed formally or integrated with computational models of dopamine function. We quantitatively evaluate the role of tonic dopamine in these functions by studying the behavior of hyperdopaminergic DAT knockdown mice in an instrumental task in a seminaturalistic homecage environment. In this {"}closed economy{"} paradigm, subjects earn all of their food by pressing either of two levers, but the relative cost for food on each lever shifts frequently. Compared to wild-type mice, hyperdopaminergic mice allocate more lever presses on high-cost levers, thus working harder to earn a given amount of food and maintain their body weight. However, both groups show a similarly quick reaction to shifts in lever cost, suggesting that the hyperdominergic mice are not slower at detecting changes, as with a learning deficit. We fit the lever choice data using reinforcement learning models to assess the distinction between acquisition and expression the models formalize. In these analyses, hyperdopaminergic mice displayed normal learning from recent reward history but diminished capacity to exploit this learning: a reduced coupling between choice and reward history. These data suggest that dopamine modulates the degree to which prior learning biases action selection and consequently alters the expression of learned, motivated behavior.",
keywords = "Behavioral flexibility, DAT knock-down, Dopamine, Environmental adaptation, Explore-exploit, Reinforcement learning",
author = "Beeler, {Jeff A.} and Nathaniel Daw and Frazier, {Cristianne R M} and Xiaoxi Zhuang",
year = "2010",
month = "11",
day = "4",
doi = "10.3389/fnbeh.2010.00170",
language = "English (US)",
volume = "4",
journal = "Frontiers in Behavioral Neuroscience",
issn = "1662-5153",
publisher = "Frontiers Research Foundation",
number = "NOV",

}

TY - JOUR

T1 - Tonic dopamine modulates exploitation of reward learning

AU - Beeler, Jeff A.

AU - Daw, Nathaniel

AU - Frazier, Cristianne R M

AU - Zhuang, Xiaoxi

PY - 2010/11/4

Y1 - 2010/11/4

N2 - The impact of dopamine on adaptive behavior in a naturalistic environment is largely unexamined. Experimental work suggests that phasic dopamine is central to reinforcement learning whereas tonic dopamine may modulate performance without altering learning per se; however, this idea has not been developed formally or integrated with computational models of dopamine function. We quantitatively evaluate the role of tonic dopamine in these functions by studying the behavior of hyperdopaminergic DAT knockdown mice in an instrumental task in a seminaturalistic homecage environment. In this "closed economy" paradigm, subjects earn all of their food by pressing either of two levers, but the relative cost for food on each lever shifts frequently. Compared to wild-type mice, hyperdopaminergic mice allocate more lever presses on high-cost levers, thus working harder to earn a given amount of food and maintain their body weight. However, both groups show a similarly quick reaction to shifts in lever cost, suggesting that the hyperdominergic mice are not slower at detecting changes, as with a learning deficit. We fit the lever choice data using reinforcement learning models to assess the distinction between acquisition and expression the models formalize. In these analyses, hyperdopaminergic mice displayed normal learning from recent reward history but diminished capacity to exploit this learning: a reduced coupling between choice and reward history. These data suggest that dopamine modulates the degree to which prior learning biases action selection and consequently alters the expression of learned, motivated behavior.

AB - The impact of dopamine on adaptive behavior in a naturalistic environment is largely unexamined. Experimental work suggests that phasic dopamine is central to reinforcement learning whereas tonic dopamine may modulate performance without altering learning per se; however, this idea has not been developed formally or integrated with computational models of dopamine function. We quantitatively evaluate the role of tonic dopamine in these functions by studying the behavior of hyperdopaminergic DAT knockdown mice in an instrumental task in a seminaturalistic homecage environment. In this "closed economy" paradigm, subjects earn all of their food by pressing either of two levers, but the relative cost for food on each lever shifts frequently. Compared to wild-type mice, hyperdopaminergic mice allocate more lever presses on high-cost levers, thus working harder to earn a given amount of food and maintain their body weight. However, both groups show a similarly quick reaction to shifts in lever cost, suggesting that the hyperdominergic mice are not slower at detecting changes, as with a learning deficit. We fit the lever choice data using reinforcement learning models to assess the distinction between acquisition and expression the models formalize. In these analyses, hyperdopaminergic mice displayed normal learning from recent reward history but diminished capacity to exploit this learning: a reduced coupling between choice and reward history. These data suggest that dopamine modulates the degree to which prior learning biases action selection and consequently alters the expression of learned, motivated behavior.

KW - Behavioral flexibility

KW - DAT knock-down

KW - Dopamine

KW - Environmental adaptation

KW - Explore-exploit

KW - Reinforcement learning

UR - http://www.scopus.com/inward/record.url?scp=79960284884&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=79960284884&partnerID=8YFLogxK

U2 - 10.3389/fnbeh.2010.00170

DO - 10.3389/fnbeh.2010.00170

M3 - Article

VL - 4

JO - Frontiers in Behavioral Neuroscience

JF - Frontiers in Behavioral Neuroscience

SN - 1662-5153

IS - NOV

M1 - 170

ER -