Validating and optimizing a crowdsourced method for gradient measures of child speech

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

There is broad consensus that speech sound development is a gradual process, with acoustic measures frequently revealing covert contrast between sounds perceived as identical. Well-constructed perceptual tasks using Visual Analog Scaling (VAS) can draw out these gradient differences. However, this method has not seen widespread uptake in speech acquisition research, possibly due to the time-intensive character of VAS data collection. This project tested the validity of streamlined VAS data collection via crowdsourcing. It also addressed a methodological question that would be challenging to answer through conventional data collection: when collecting ratings of speech samples elicited from multiple individuals, should those samples be presented in fully random order, or grouped by speaker? 100 naïve listeners recruited through Amazon Mechanical Turk provided VAS ratings for 120 /r/ words produced by 4 children before, during, and after intervention. 50 listeners rated the stimuli in fully randomized order and 50 in grouped-by-speaker order. Mean click location was compared against an acoustic standard, and standard error of click location was used to index variability. In both conditions, mean click location was highly correlated with the acoustic measure, supporting the validity of speech ratings obtained via crowdsourcing. Lower variability was observed in the grouped presentation condition.

Original languageEnglish (US)
Title of host publicationProceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH
PublisherInternational Speech and Communication Association
Pages2834-2838
Number of pages5
Volume2015-January
StatePublished - 2015
Event16th Annual Conference of the International Speech Communication Association, INTERSPEECH 2015 - Dresden, Germany
Duration: Sep 6 2015Sep 10 2015

Other

Other16th Annual Conference of the International Speech Communication Association, INTERSPEECH 2015
CountryGermany
CityDresden
Period9/6/159/10/15

Fingerprint

Scaling
Gradient
Analogue
Acoustics
Acoustic waves
Standard error
Vision
Speech
Children
Data Collection
Rating
Sound
Listeners
Acquisition
Standards
Character
Presentation
Turks
Amazon
Nave

Keywords

  • Acquisition and disorders
  • Covert contrast
  • Crowdsourcing
  • Perceptual rating

ASJC Scopus subject areas

  • Language and Linguistics
  • Human-Computer Interaction
  • Signal Processing
  • Software
  • Modeling and Simulation

Cite this

Byun, T. M., Hitchcock, E., & Harel, D. (2015). Validating and optimizing a crowdsourced method for gradient measures of child speech. In Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH (Vol. 2015-January, pp. 2834-2838). International Speech and Communication Association.

Validating and optimizing a crowdsourced method for gradient measures of child speech. / Byun, Tara McAllister; Hitchcock, Elaine; Harel, Daphna.

Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH. Vol. 2015-January International Speech and Communication Association, 2015. p. 2834-2838.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Byun, TM, Hitchcock, E & Harel, D 2015, Validating and optimizing a crowdsourced method for gradient measures of child speech. in Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH. vol. 2015-January, International Speech and Communication Association, pp. 2834-2838, 16th Annual Conference of the International Speech Communication Association, INTERSPEECH 2015, Dresden, Germany, 9/6/15.
Byun TM, Hitchcock E, Harel D. Validating and optimizing a crowdsourced method for gradient measures of child speech. In Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH. Vol. 2015-January. International Speech and Communication Association. 2015. p. 2834-2838
Byun, Tara McAllister ; Hitchcock, Elaine ; Harel, Daphna. / Validating and optimizing a crowdsourced method for gradient measures of child speech. Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH. Vol. 2015-January International Speech and Communication Association, 2015. pp. 2834-2838
@inproceedings{598a081ce9af494aafa9c0a137ddea51,
title = "Validating and optimizing a crowdsourced method for gradient measures of child speech",
abstract = "There is broad consensus that speech sound development is a gradual process, with acoustic measures frequently revealing covert contrast between sounds perceived as identical. Well-constructed perceptual tasks using Visual Analog Scaling (VAS) can draw out these gradient differences. However, this method has not seen widespread uptake in speech acquisition research, possibly due to the time-intensive character of VAS data collection. This project tested the validity of streamlined VAS data collection via crowdsourcing. It also addressed a methodological question that would be challenging to answer through conventional data collection: when collecting ratings of speech samples elicited from multiple individuals, should those samples be presented in fully random order, or grouped by speaker? 100 na{\"i}ve listeners recruited through Amazon Mechanical Turk provided VAS ratings for 120 /r/ words produced by 4 children before, during, and after intervention. 50 listeners rated the stimuli in fully randomized order and 50 in grouped-by-speaker order. Mean click location was compared against an acoustic standard, and standard error of click location was used to index variability. In both conditions, mean click location was highly correlated with the acoustic measure, supporting the validity of speech ratings obtained via crowdsourcing. Lower variability was observed in the grouped presentation condition.",
keywords = "Acquisition and disorders, Covert contrast, Crowdsourcing, Perceptual rating",
author = "Byun, {Tara McAllister} and Elaine Hitchcock and Daphna Harel",
year = "2015",
language = "English (US)",
volume = "2015-January",
pages = "2834--2838",
booktitle = "Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH",
publisher = "International Speech and Communication Association",

}

TY - GEN

T1 - Validating and optimizing a crowdsourced method for gradient measures of child speech

AU - Byun, Tara McAllister

AU - Hitchcock, Elaine

AU - Harel, Daphna

PY - 2015

Y1 - 2015

N2 - There is broad consensus that speech sound development is a gradual process, with acoustic measures frequently revealing covert contrast between sounds perceived as identical. Well-constructed perceptual tasks using Visual Analog Scaling (VAS) can draw out these gradient differences. However, this method has not seen widespread uptake in speech acquisition research, possibly due to the time-intensive character of VAS data collection. This project tested the validity of streamlined VAS data collection via crowdsourcing. It also addressed a methodological question that would be challenging to answer through conventional data collection: when collecting ratings of speech samples elicited from multiple individuals, should those samples be presented in fully random order, or grouped by speaker? 100 naïve listeners recruited through Amazon Mechanical Turk provided VAS ratings for 120 /r/ words produced by 4 children before, during, and after intervention. 50 listeners rated the stimuli in fully randomized order and 50 in grouped-by-speaker order. Mean click location was compared against an acoustic standard, and standard error of click location was used to index variability. In both conditions, mean click location was highly correlated with the acoustic measure, supporting the validity of speech ratings obtained via crowdsourcing. Lower variability was observed in the grouped presentation condition.

AB - There is broad consensus that speech sound development is a gradual process, with acoustic measures frequently revealing covert contrast between sounds perceived as identical. Well-constructed perceptual tasks using Visual Analog Scaling (VAS) can draw out these gradient differences. However, this method has not seen widespread uptake in speech acquisition research, possibly due to the time-intensive character of VAS data collection. This project tested the validity of streamlined VAS data collection via crowdsourcing. It also addressed a methodological question that would be challenging to answer through conventional data collection: when collecting ratings of speech samples elicited from multiple individuals, should those samples be presented in fully random order, or grouped by speaker? 100 naïve listeners recruited through Amazon Mechanical Turk provided VAS ratings for 120 /r/ words produced by 4 children before, during, and after intervention. 50 listeners rated the stimuli in fully randomized order and 50 in grouped-by-speaker order. Mean click location was compared against an acoustic standard, and standard error of click location was used to index variability. In both conditions, mean click location was highly correlated with the acoustic measure, supporting the validity of speech ratings obtained via crowdsourcing. Lower variability was observed in the grouped presentation condition.

KW - Acquisition and disorders

KW - Covert contrast

KW - Crowdsourcing

KW - Perceptual rating

UR - http://www.scopus.com/inward/record.url?scp=84959113429&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84959113429&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:84959113429

VL - 2015-January

SP - 2834

EP - 2838

BT - Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH

PB - International Speech and Communication Association

ER -