Online crowdsourcing for efficient rating of speech

A validation study

Research output: Contribution to journalArticle

Abstract

Blinded listener ratings are essential for valid assessment of interventions for speech disorders, but collecting these ratings can be time-intensive and costly. This study evaluated the validity of speech ratings obtained through online crowdsourcing, a potentially more efficient approach. 100 words from children with /r/ misarticulation were electronically presented for binary rating by 35 phonetically trained listeners and 205 naïve listeners recruited through the Amazon Mechanical Turk (AMT) crowdsourcing platform. Bootstrapping was used to compare different-sized samples of AMT listeners against a "gold standard" (mode across all trained listeners) and an "industry standard" (mode across bootstrapped samples of three trained listeners). There was strong overall agreement between trained and AMT listeners. The "industry standard" level of performance was matched by bootstrapped samples with n = 9 AMT listeners. These results support the hypothesis that valid ratings of speech data can be obtained in an efficient manner through AMT. Researchers in communication disorders could benefit from increased awareness of this method. Learning outcomes: Readers will be able to (a) discuss advantages and disadvantages of data collection through the crowdsourcing platform Amazon Mechanical Turk (AMT), (b) describe the results of a validity study comparing samples of AMT listeners versus phonetically trained listeners in a speech-rating task.

Original languageEnglish (US)
Pages (from-to)70-83
Number of pages14
JournalJournal of Communication Disorders
Volume53
DOIs
StatePublished - Jan 1 2015

Fingerprint

Crowdsourcing
Validation Studies
listener
Turk
rating
Industry
Articulation Disorders
Communication Disorders
Speech Disorders
Reproducibility of Results
Research Personnel
Learning
communication disorder
gold standard
industry
speech disorder

Keywords

  • Crowdsourcing
  • Research methods
  • Speech perception
  • Speech rating
  • Speech sound disorders

ASJC Scopus subject areas

  • Linguistics and Language
  • Speech and Hearing
  • Cognitive Neuroscience
  • Experimental and Cognitive Psychology
  • LPN and LVN

Cite this

Online crowdsourcing for efficient rating of speech : A validation study. / McAllister Byun, Tara; Halpin, Peter F.; Szeredi, Daniel.

In: Journal of Communication Disorders, Vol. 53, 01.01.2015, p. 70-83.

Research output: Contribution to journalArticle

@article{defa78e4043d44a09e426ca43323e22b,
title = "Online crowdsourcing for efficient rating of speech: A validation study",
abstract = "Blinded listener ratings are essential for valid assessment of interventions for speech disorders, but collecting these ratings can be time-intensive and costly. This study evaluated the validity of speech ratings obtained through online crowdsourcing, a potentially more efficient approach. 100 words from children with /r/ misarticulation were electronically presented for binary rating by 35 phonetically trained listeners and 205 na{\"i}ve listeners recruited through the Amazon Mechanical Turk (AMT) crowdsourcing platform. Bootstrapping was used to compare different-sized samples of AMT listeners against a {"}gold standard{"} (mode across all trained listeners) and an {"}industry standard{"} (mode across bootstrapped samples of three trained listeners). There was strong overall agreement between trained and AMT listeners. The {"}industry standard{"} level of performance was matched by bootstrapped samples with n = 9 AMT listeners. These results support the hypothesis that valid ratings of speech data can be obtained in an efficient manner through AMT. Researchers in communication disorders could benefit from increased awareness of this method. Learning outcomes: Readers will be able to (a) discuss advantages and disadvantages of data collection through the crowdsourcing platform Amazon Mechanical Turk (AMT), (b) describe the results of a validity study comparing samples of AMT listeners versus phonetically trained listeners in a speech-rating task.",
keywords = "Crowdsourcing, Research methods, Speech perception, Speech rating, Speech sound disorders",
author = "{McAllister Byun}, Tara and Halpin, {Peter F.} and Daniel Szeredi",
year = "2015",
month = "1",
day = "1",
doi = "10.1016/j.jcomdis.2014.11.003",
language = "English (US)",
volume = "53",
pages = "70--83",
journal = "Journal of Communication Disorders",
issn = "0021-9924",
publisher = "Elsevier Inc.",

}

TY - JOUR

T1 - Online crowdsourcing for efficient rating of speech

T2 - A validation study

AU - McAllister Byun, Tara

AU - Halpin, Peter F.

AU - Szeredi, Daniel

PY - 2015/1/1

Y1 - 2015/1/1

N2 - Blinded listener ratings are essential for valid assessment of interventions for speech disorders, but collecting these ratings can be time-intensive and costly. This study evaluated the validity of speech ratings obtained through online crowdsourcing, a potentially more efficient approach. 100 words from children with /r/ misarticulation were electronically presented for binary rating by 35 phonetically trained listeners and 205 naïve listeners recruited through the Amazon Mechanical Turk (AMT) crowdsourcing platform. Bootstrapping was used to compare different-sized samples of AMT listeners against a "gold standard" (mode across all trained listeners) and an "industry standard" (mode across bootstrapped samples of three trained listeners). There was strong overall agreement between trained and AMT listeners. The "industry standard" level of performance was matched by bootstrapped samples with n = 9 AMT listeners. These results support the hypothesis that valid ratings of speech data can be obtained in an efficient manner through AMT. Researchers in communication disorders could benefit from increased awareness of this method. Learning outcomes: Readers will be able to (a) discuss advantages and disadvantages of data collection through the crowdsourcing platform Amazon Mechanical Turk (AMT), (b) describe the results of a validity study comparing samples of AMT listeners versus phonetically trained listeners in a speech-rating task.

AB - Blinded listener ratings are essential for valid assessment of interventions for speech disorders, but collecting these ratings can be time-intensive and costly. This study evaluated the validity of speech ratings obtained through online crowdsourcing, a potentially more efficient approach. 100 words from children with /r/ misarticulation were electronically presented for binary rating by 35 phonetically trained listeners and 205 naïve listeners recruited through the Amazon Mechanical Turk (AMT) crowdsourcing platform. Bootstrapping was used to compare different-sized samples of AMT listeners against a "gold standard" (mode across all trained listeners) and an "industry standard" (mode across bootstrapped samples of three trained listeners). There was strong overall agreement between trained and AMT listeners. The "industry standard" level of performance was matched by bootstrapped samples with n = 9 AMT listeners. These results support the hypothesis that valid ratings of speech data can be obtained in an efficient manner through AMT. Researchers in communication disorders could benefit from increased awareness of this method. Learning outcomes: Readers will be able to (a) discuss advantages and disadvantages of data collection through the crowdsourcing platform Amazon Mechanical Turk (AMT), (b) describe the results of a validity study comparing samples of AMT listeners versus phonetically trained listeners in a speech-rating task.

KW - Crowdsourcing

KW - Research methods

KW - Speech perception

KW - Speech rating

KW - Speech sound disorders

UR - http://www.scopus.com/inward/record.url?scp=84923095708&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84923095708&partnerID=8YFLogxK

U2 - 10.1016/j.jcomdis.2014.11.003

DO - 10.1016/j.jcomdis.2014.11.003

M3 - Article

VL - 53

SP - 70

EP - 83

JO - Journal of Communication Disorders

JF - Journal of Communication Disorders

SN - 0021-9924

ER -