Cortical substrates for exploratory decisions in humans

Nathaniel D. Daw, John P. O'Doherty, Peter Dayan, Ben Seymour, Raymond J. Dolan

Research output: Contribution to journalArticle

Abstract

Decision making in an uncertain environment poses a conflict between the opposing demands of gathering and exploiting information. In a classic illustration of this 'exploration-exploitation' dilemma, a gambler choosing between multiple slot machines balances the desire to select what seems, on the basis of accumulated experience, the richest option, against the desire to choose a less familiar option that might turn out more advantageous (and thereby provide information for improving future decisions). Far from representing idle curiosity, such exploration is often critical for organisms to discover how best to harvest resources such as food and water. In appetitive choice, substantial experimental evidence, underpinned by computational reinforcement learning (RL) theory, indicates that a dopaminergic, striatal and medial prefrontal network mediates learning to exploit. In contrast, although exploration has been well studied from both theoretical and ethological perspectives, its neural substrates are much less clear. Here we show, in a gambling task, that human subjects' choices can be characterized by a computationally well-regarded strategy for addressing the explore/exploit dilemma. Furthermore, using this characterization to classify decisions as exploratory or exploitative, we employ functional magnetic resonance imaging to show that the frontopolar cortex and intraparietal sulcus are preferentially active during exploratory decisions. In contrast, regions of striatum and ventromedial prefrontal cortex exhibit activity characteristic of an involvement in value-based exploitative decision making. The results suggest a model of action selection under uncertainty that involves switching between exploratory and exploitative behavioural modes, and provide a computationally precise characterization of the contribution of key decision-related brain systems to each of these functions.

Original languageEnglish (US)
Pages (from-to)876-879
Number of pages4
JournalNature
Volume441
Issue number7095
DOIs
StatePublished - Jun 15 2006

Fingerprint

Decision Making
Learning
Corpus Striatum
Gambling
Parietal Lobe
Exploratory Behavior
Prefrontal Cortex
Uncertainty
Magnetic Resonance Imaging
Food
Water
Brain
Conflict (Psychology)
Reinforcement (Psychology)

ASJC Scopus subject areas

  • General

Cite this

Daw, N. D., O'Doherty, J. P., Dayan, P., Seymour, B., & Dolan, R. J. (2006). Cortical substrates for exploratory decisions in humans. Nature, 441(7095), 876-879. https://doi.org/10.1038/nature04766

Cortical substrates for exploratory decisions in humans. / Daw, Nathaniel D.; O'Doherty, John P.; Dayan, Peter; Seymour, Ben; Dolan, Raymond J.

In: Nature, Vol. 441, No. 7095, 15.06.2006, p. 876-879.

Research output: Contribution to journalArticle

Daw, ND, O'Doherty, JP, Dayan, P, Seymour, B & Dolan, RJ 2006, 'Cortical substrates for exploratory decisions in humans', Nature, vol. 441, no. 7095, pp. 876-879. https://doi.org/10.1038/nature04766
Daw ND, O'Doherty JP, Dayan P, Seymour B, Dolan RJ. Cortical substrates for exploratory decisions in humans. Nature. 2006 Jun 15;441(7095):876-879. https://doi.org/10.1038/nature04766
Daw, Nathaniel D. ; O'Doherty, John P. ; Dayan, Peter ; Seymour, Ben ; Dolan, Raymond J. / Cortical substrates for exploratory decisions in humans. In: Nature. 2006 ; Vol. 441, No. 7095. pp. 876-879.
@article{2981225f61b5411b940f4bb653240d32,
title = "Cortical substrates for exploratory decisions in humans",
abstract = "Decision making in an uncertain environment poses a conflict between the opposing demands of gathering and exploiting information. In a classic illustration of this 'exploration-exploitation' dilemma, a gambler choosing between multiple slot machines balances the desire to select what seems, on the basis of accumulated experience, the richest option, against the desire to choose a less familiar option that might turn out more advantageous (and thereby provide information for improving future decisions). Far from representing idle curiosity, such exploration is often critical for organisms to discover how best to harvest resources such as food and water. In appetitive choice, substantial experimental evidence, underpinned by computational reinforcement learning (RL) theory, indicates that a dopaminergic, striatal and medial prefrontal network mediates learning to exploit. In contrast, although exploration has been well studied from both theoretical and ethological perspectives, its neural substrates are much less clear. Here we show, in a gambling task, that human subjects' choices can be characterized by a computationally well-regarded strategy for addressing the explore/exploit dilemma. Furthermore, using this characterization to classify decisions as exploratory or exploitative, we employ functional magnetic resonance imaging to show that the frontopolar cortex and intraparietal sulcus are preferentially active during exploratory decisions. In contrast, regions of striatum and ventromedial prefrontal cortex exhibit activity characteristic of an involvement in value-based exploitative decision making. The results suggest a model of action selection under uncertainty that involves switching between exploratory and exploitative behavioural modes, and provide a computationally precise characterization of the contribution of key decision-related brain systems to each of these functions.",
author = "Daw, {Nathaniel D.} and O'Doherty, {John P.} and Peter Dayan and Ben Seymour and Dolan, {Raymond J.}",
year = "2006",
month = "6",
day = "15",
doi = "10.1038/nature04766",
language = "English (US)",
volume = "441",
pages = "876--879",
journal = "Nature Cell Biology",
issn = "1465-7392",
publisher = "Nature Publishing Group",
number = "7095",

}

TY - JOUR

T1 - Cortical substrates for exploratory decisions in humans

AU - Daw, Nathaniel D.

AU - O'Doherty, John P.

AU - Dayan, Peter

AU - Seymour, Ben

AU - Dolan, Raymond J.

PY - 2006/6/15

Y1 - 2006/6/15

N2 - Decision making in an uncertain environment poses a conflict between the opposing demands of gathering and exploiting information. In a classic illustration of this 'exploration-exploitation' dilemma, a gambler choosing between multiple slot machines balances the desire to select what seems, on the basis of accumulated experience, the richest option, against the desire to choose a less familiar option that might turn out more advantageous (and thereby provide information for improving future decisions). Far from representing idle curiosity, such exploration is often critical for organisms to discover how best to harvest resources such as food and water. In appetitive choice, substantial experimental evidence, underpinned by computational reinforcement learning (RL) theory, indicates that a dopaminergic, striatal and medial prefrontal network mediates learning to exploit. In contrast, although exploration has been well studied from both theoretical and ethological perspectives, its neural substrates are much less clear. Here we show, in a gambling task, that human subjects' choices can be characterized by a computationally well-regarded strategy for addressing the explore/exploit dilemma. Furthermore, using this characterization to classify decisions as exploratory or exploitative, we employ functional magnetic resonance imaging to show that the frontopolar cortex and intraparietal sulcus are preferentially active during exploratory decisions. In contrast, regions of striatum and ventromedial prefrontal cortex exhibit activity characteristic of an involvement in value-based exploitative decision making. The results suggest a model of action selection under uncertainty that involves switching between exploratory and exploitative behavioural modes, and provide a computationally precise characterization of the contribution of key decision-related brain systems to each of these functions.

AB - Decision making in an uncertain environment poses a conflict between the opposing demands of gathering and exploiting information. In a classic illustration of this 'exploration-exploitation' dilemma, a gambler choosing between multiple slot machines balances the desire to select what seems, on the basis of accumulated experience, the richest option, against the desire to choose a less familiar option that might turn out more advantageous (and thereby provide information for improving future decisions). Far from representing idle curiosity, such exploration is often critical for organisms to discover how best to harvest resources such as food and water. In appetitive choice, substantial experimental evidence, underpinned by computational reinforcement learning (RL) theory, indicates that a dopaminergic, striatal and medial prefrontal network mediates learning to exploit. In contrast, although exploration has been well studied from both theoretical and ethological perspectives, its neural substrates are much less clear. Here we show, in a gambling task, that human subjects' choices can be characterized by a computationally well-regarded strategy for addressing the explore/exploit dilemma. Furthermore, using this characterization to classify decisions as exploratory or exploitative, we employ functional magnetic resonance imaging to show that the frontopolar cortex and intraparietal sulcus are preferentially active during exploratory decisions. In contrast, regions of striatum and ventromedial prefrontal cortex exhibit activity characteristic of an involvement in value-based exploitative decision making. The results suggest a model of action selection under uncertainty that involves switching between exploratory and exploitative behavioural modes, and provide a computationally precise characterization of the contribution of key decision-related brain systems to each of these functions.

UR - http://www.scopus.com/inward/record.url?scp=33745223257&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=33745223257&partnerID=8YFLogxK

U2 - 10.1038/nature04766

DO - 10.1038/nature04766

M3 - Article

VL - 441

SP - 876

EP - 879

JO - Nature Cell Biology

JF - Nature Cell Biology

SN - 1465-7392

IS - 7095

ER -