Generalization of value in reinforcement learning by humans

G. Elliott Wimmer, Nathaniel D. Daw, Daphna Shohamy

Research output: Contribution to journalArticle

Abstract

Research in decision-making has focused on the role of dopamine and its striatal targets in guiding choices via learned stimulus-reward or stimulus-response associations, behavior that is well described by reinforcement learning theories. However, basic reinforcement learning is relatively limited in scope and does not explain how learning about stimulus regularities or relations may guide decision-making. A candidate mechanism for this type of learning comes from the domain of memory, which has highlighted a role for the hippocampus in learning of stimulus-stimulus relations, typically dissociated from the role of the striatum in stimulus-response learning. Here, we used functional magnetic resonance imaging and computational model-based analyses to examine the joint contributions of these mechanisms to reinforcement learning. Humans performed a reinforcement learning task with added relational structure, modeled after tasks used to isolate hippocampal contributions to memory. On each trial participants chose one of four options, but the reward probabilities for pairs of options were correlated across trials. This (uninstructed) relationship between pairs of options potentially enabled an observer to learn about option values based on experience with the other options and to generalize across them. We observed blood oxygen level-dependent (BOLD) activity related to learning in the striatum and also in the hippocampus. By comparing a basic reinforcement learning model to one augmented to allow feedback to generalize between correlated options, we tested whether choice behavior and BOLD activity were influenced by the opportunity to generalize across correlated options. Although such generalization goes beyond standard computational accounts of reinforcement learning and striatal BOLD, both choices and striatal BOLD activity were better explained by the augmented model. Consistent with the hypothesized role for the hippocampus in this generalization, functional connectivity between the ventral striatum and hippocampus was modulated, across participants, by the ability of the augmented model to capture participants' choice. Our results thus point toward an interactive model in which striatal reinforcement learning systems may employ relational representations typically associated with the hippocampus.

Original languageEnglish (US)
Pages (from-to)1092-1104
Number of pages13
JournalEuropean Journal of Neuroscience
Volume35
Issue number7
DOIs
StatePublished - Apr 2012

Fingerprint

Learning
Corpus Striatum
Hippocampus
Oxygen
Reward
Reinforcement (Psychology)
Generalization (Psychology)
Decision Making
Choice Behavior
Aptitude
Dopamine
Magnetic Resonance Imaging
Research

Keywords

  • Computational model
  • Hippocampus
  • Memory
  • Reward
  • Ventral striatum

ASJC Scopus subject areas

  • Neuroscience(all)

Cite this

Generalization of value in reinforcement learning by humans. / Wimmer, G. Elliott; Daw, Nathaniel D.; Shohamy, Daphna.

In: European Journal of Neuroscience, Vol. 35, No. 7, 04.2012, p. 1092-1104.

Research output: Contribution to journalArticle

Wimmer, G. Elliott ; Daw, Nathaniel D. ; Shohamy, Daphna. / Generalization of value in reinforcement learning by humans. In: European Journal of Neuroscience. 2012 ; Vol. 35, No. 7. pp. 1092-1104.
@article{4f416f44e38a4994b20b843a1a4de80a,
title = "Generalization of value in reinforcement learning by humans",
abstract = "Research in decision-making has focused on the role of dopamine and its striatal targets in guiding choices via learned stimulus-reward or stimulus-response associations, behavior that is well described by reinforcement learning theories. However, basic reinforcement learning is relatively limited in scope and does not explain how learning about stimulus regularities or relations may guide decision-making. A candidate mechanism for this type of learning comes from the domain of memory, which has highlighted a role for the hippocampus in learning of stimulus-stimulus relations, typically dissociated from the role of the striatum in stimulus-response learning. Here, we used functional magnetic resonance imaging and computational model-based analyses to examine the joint contributions of these mechanisms to reinforcement learning. Humans performed a reinforcement learning task with added relational structure, modeled after tasks used to isolate hippocampal contributions to memory. On each trial participants chose one of four options, but the reward probabilities for pairs of options were correlated across trials. This (uninstructed) relationship between pairs of options potentially enabled an observer to learn about option values based on experience with the other options and to generalize across them. We observed blood oxygen level-dependent (BOLD) activity related to learning in the striatum and also in the hippocampus. By comparing a basic reinforcement learning model to one augmented to allow feedback to generalize between correlated options, we tested whether choice behavior and BOLD activity were influenced by the opportunity to generalize across correlated options. Although such generalization goes beyond standard computational accounts of reinforcement learning and striatal BOLD, both choices and striatal BOLD activity were better explained by the augmented model. Consistent with the hypothesized role for the hippocampus in this generalization, functional connectivity between the ventral striatum and hippocampus was modulated, across participants, by the ability of the augmented model to capture participants' choice. Our results thus point toward an interactive model in which striatal reinforcement learning systems may employ relational representations typically associated with the hippocampus.",
keywords = "Computational model, Hippocampus, Memory, Reward, Ventral striatum",
author = "Wimmer, {G. Elliott} and Daw, {Nathaniel D.} and Daphna Shohamy",
year = "2012",
month = "4",
doi = "10.1111/j.1460-9568.2012.08017.x",
language = "English (US)",
volume = "35",
pages = "1092--1104",
journal = "European Journal of Neuroscience",
issn = "0953-816X",
publisher = "Wiley-Blackwell",
number = "7",

}

TY - JOUR

T1 - Generalization of value in reinforcement learning by humans

AU - Wimmer, G. Elliott

AU - Daw, Nathaniel D.

AU - Shohamy, Daphna

PY - 2012/4

Y1 - 2012/4

N2 - Research in decision-making has focused on the role of dopamine and its striatal targets in guiding choices via learned stimulus-reward or stimulus-response associations, behavior that is well described by reinforcement learning theories. However, basic reinforcement learning is relatively limited in scope and does not explain how learning about stimulus regularities or relations may guide decision-making. A candidate mechanism for this type of learning comes from the domain of memory, which has highlighted a role for the hippocampus in learning of stimulus-stimulus relations, typically dissociated from the role of the striatum in stimulus-response learning. Here, we used functional magnetic resonance imaging and computational model-based analyses to examine the joint contributions of these mechanisms to reinforcement learning. Humans performed a reinforcement learning task with added relational structure, modeled after tasks used to isolate hippocampal contributions to memory. On each trial participants chose one of four options, but the reward probabilities for pairs of options were correlated across trials. This (uninstructed) relationship between pairs of options potentially enabled an observer to learn about option values based on experience with the other options and to generalize across them. We observed blood oxygen level-dependent (BOLD) activity related to learning in the striatum and also in the hippocampus. By comparing a basic reinforcement learning model to one augmented to allow feedback to generalize between correlated options, we tested whether choice behavior and BOLD activity were influenced by the opportunity to generalize across correlated options. Although such generalization goes beyond standard computational accounts of reinforcement learning and striatal BOLD, both choices and striatal BOLD activity were better explained by the augmented model. Consistent with the hypothesized role for the hippocampus in this generalization, functional connectivity between the ventral striatum and hippocampus was modulated, across participants, by the ability of the augmented model to capture participants' choice. Our results thus point toward an interactive model in which striatal reinforcement learning systems may employ relational representations typically associated with the hippocampus.

AB - Research in decision-making has focused on the role of dopamine and its striatal targets in guiding choices via learned stimulus-reward or stimulus-response associations, behavior that is well described by reinforcement learning theories. However, basic reinforcement learning is relatively limited in scope and does not explain how learning about stimulus regularities or relations may guide decision-making. A candidate mechanism for this type of learning comes from the domain of memory, which has highlighted a role for the hippocampus in learning of stimulus-stimulus relations, typically dissociated from the role of the striatum in stimulus-response learning. Here, we used functional magnetic resonance imaging and computational model-based analyses to examine the joint contributions of these mechanisms to reinforcement learning. Humans performed a reinforcement learning task with added relational structure, modeled after tasks used to isolate hippocampal contributions to memory. On each trial participants chose one of four options, but the reward probabilities for pairs of options were correlated across trials. This (uninstructed) relationship between pairs of options potentially enabled an observer to learn about option values based on experience with the other options and to generalize across them. We observed blood oxygen level-dependent (BOLD) activity related to learning in the striatum and also in the hippocampus. By comparing a basic reinforcement learning model to one augmented to allow feedback to generalize between correlated options, we tested whether choice behavior and BOLD activity were influenced by the opportunity to generalize across correlated options. Although such generalization goes beyond standard computational accounts of reinforcement learning and striatal BOLD, both choices and striatal BOLD activity were better explained by the augmented model. Consistent with the hypothesized role for the hippocampus in this generalization, functional connectivity between the ventral striatum and hippocampus was modulated, across participants, by the ability of the augmented model to capture participants' choice. Our results thus point toward an interactive model in which striatal reinforcement learning systems may employ relational representations typically associated with the hippocampus.

KW - Computational model

KW - Hippocampus

KW - Memory

KW - Reward

KW - Ventral striatum

UR - http://www.scopus.com/inward/record.url?scp=84859339117&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84859339117&partnerID=8YFLogxK

U2 - 10.1111/j.1460-9568.2012.08017.x

DO - 10.1111/j.1460-9568.2012.08017.x

M3 - Article

VL - 35

SP - 1092

EP - 1104

JO - European Journal of Neuroscience

JF - European Journal of Neuroscience

SN - 0953-816X

IS - 7

ER -