Evaluating Amazon's Mechanical Turk as a Tool for Experimental Behavioral Research

Matthew J C Crump, John V. McDonnell, Todd M. Gureckis

Research output: Contribution to journalArticle

Abstract

Amazon Mechanical Turk (AMT) is an online crowdsourcing service where anonymous online workers complete web-based tasks for small sums of money. The service has attracted attention from experimental psychologists interested in gathering human subject data more efficiently. However, relative to traditional laboratory studies, many aspects of the testing environment are not under the experimenter's control. In this paper, we attempt to empirically evaluate the fidelity of the AMT system for use in cognitive behavioral experiments. These types of experiment differ from simple surveys in that they require multiple trials, sustained attention from participants, comprehension of complex instructions, and millisecond accuracy for response recording and stimulus presentation. We replicate a diverse body of tasks from experimental psychology including the Stroop, Switching, Flanker, Simon, Posner Cuing, attentional blink, subliminal priming, and category learning tasks using participants recruited using AMT. While most of replications were qualitatively successful and validated the approach of collecting data anonymously online using a web-browser, others revealed disparity between laboratory results and online results. A number of important lessons were encountered in the process of conducting these replications that should be of value to other researchers.

Original languageEnglish (US)
Article numbere57410
JournalPLoS One
Volume8
Issue number3
DOIs
StatePublished - Mar 13 2013

Fingerprint

Behavioral research
Behavioral Research
Crowdsourcing
Attentional Blink
Web Browser
Experimental Psychology
Web browsers
psychology
learning
researchers
Experiments
Research Personnel
Learning
Psychology
Testing
testing

ASJC Scopus subject areas

  • Agricultural and Biological Sciences(all)
  • Biochemistry, Genetics and Molecular Biology(all)
  • Medicine(all)

Cite this

Evaluating Amazon's Mechanical Turk as a Tool for Experimental Behavioral Research. / Crump, Matthew J C; McDonnell, John V.; Gureckis, Todd M.

In: PLoS One, Vol. 8, No. 3, e57410, 13.03.2013.

Research output: Contribution to journalArticle

Crump, Matthew J C ; McDonnell, John V. ; Gureckis, Todd M. / Evaluating Amazon's Mechanical Turk as a Tool for Experimental Behavioral Research. In: PLoS One. 2013 ; Vol. 8, No. 3.
@article{817958da8f834f129fea5ef10801e3da,
title = "Evaluating Amazon's Mechanical Turk as a Tool for Experimental Behavioral Research",
abstract = "Amazon Mechanical Turk (AMT) is an online crowdsourcing service where anonymous online workers complete web-based tasks for small sums of money. The service has attracted attention from experimental psychologists interested in gathering human subject data more efficiently. However, relative to traditional laboratory studies, many aspects of the testing environment are not under the experimenter's control. In this paper, we attempt to empirically evaluate the fidelity of the AMT system for use in cognitive behavioral experiments. These types of experiment differ from simple surveys in that they require multiple trials, sustained attention from participants, comprehension of complex instructions, and millisecond accuracy for response recording and stimulus presentation. We replicate a diverse body of tasks from experimental psychology including the Stroop, Switching, Flanker, Simon, Posner Cuing, attentional blink, subliminal priming, and category learning tasks using participants recruited using AMT. While most of replications were qualitatively successful and validated the approach of collecting data anonymously online using a web-browser, others revealed disparity between laboratory results and online results. A number of important lessons were encountered in the process of conducting these replications that should be of value to other researchers.",
author = "Crump, {Matthew J C} and McDonnell, {John V.} and Gureckis, {Todd M.}",
year = "2013",
month = "3",
day = "13",
doi = "10.1371/journal.pone.0057410",
language = "English (US)",
volume = "8",
journal = "PLoS One",
issn = "1932-6203",
publisher = "Public Library of Science",
number = "3",

}

TY - JOUR

T1 - Evaluating Amazon's Mechanical Turk as a Tool for Experimental Behavioral Research

AU - Crump, Matthew J C

AU - McDonnell, John V.

AU - Gureckis, Todd M.

PY - 2013/3/13

Y1 - 2013/3/13

N2 - Amazon Mechanical Turk (AMT) is an online crowdsourcing service where anonymous online workers complete web-based tasks for small sums of money. The service has attracted attention from experimental psychologists interested in gathering human subject data more efficiently. However, relative to traditional laboratory studies, many aspects of the testing environment are not under the experimenter's control. In this paper, we attempt to empirically evaluate the fidelity of the AMT system for use in cognitive behavioral experiments. These types of experiment differ from simple surveys in that they require multiple trials, sustained attention from participants, comprehension of complex instructions, and millisecond accuracy for response recording and stimulus presentation. We replicate a diverse body of tasks from experimental psychology including the Stroop, Switching, Flanker, Simon, Posner Cuing, attentional blink, subliminal priming, and category learning tasks using participants recruited using AMT. While most of replications were qualitatively successful and validated the approach of collecting data anonymously online using a web-browser, others revealed disparity between laboratory results and online results. A number of important lessons were encountered in the process of conducting these replications that should be of value to other researchers.

AB - Amazon Mechanical Turk (AMT) is an online crowdsourcing service where anonymous online workers complete web-based tasks for small sums of money. The service has attracted attention from experimental psychologists interested in gathering human subject data more efficiently. However, relative to traditional laboratory studies, many aspects of the testing environment are not under the experimenter's control. In this paper, we attempt to empirically evaluate the fidelity of the AMT system for use in cognitive behavioral experiments. These types of experiment differ from simple surveys in that they require multiple trials, sustained attention from participants, comprehension of complex instructions, and millisecond accuracy for response recording and stimulus presentation. We replicate a diverse body of tasks from experimental psychology including the Stroop, Switching, Flanker, Simon, Posner Cuing, attentional blink, subliminal priming, and category learning tasks using participants recruited using AMT. While most of replications were qualitatively successful and validated the approach of collecting data anonymously online using a web-browser, others revealed disparity between laboratory results and online results. A number of important lessons were encountered in the process of conducting these replications that should be of value to other researchers.

UR - http://www.scopus.com/inward/record.url?scp=84874841717&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84874841717&partnerID=8YFLogxK

U2 - 10.1371/journal.pone.0057410

DO - 10.1371/journal.pone.0057410

M3 - Article

C2 - 23516406

AN - SCOPUS:84874841717

VL - 8

JO - PLoS One

JF - PLoS One

SN - 1932-6203

IS - 3

M1 - e57410

ER -