A comparison of informal and formal acceptability judgments using a random sample from Linguistic Inquiry 2001-2010

Jon Sprouse, Carson T. Schütze, Diogo Almeida

Research output: Contribution to journalArticle

Abstract

The goal of the present study is to provide a direct comparison of the results of informal judgment collection methods with the results of formal judgment collection methods, as a first step in understanding the relative merits of each family of methods. Although previous studies have compared small samples of informal and formal results, this article presents the first large-scale comparison based on a random sample of phenomena from a leading theoretical journal (Linguistic Inquiry). We tested 296 data points from the approximately 1743 English data points that were published in Linguistic Inquiry between 2001 and 2010. We tested this sample with 936 naïve participants using three formal judgment tasks (magnitude estimation, 7-point Likert scale, and two-alternative forced-choice) and report five statistical analyses. The results suggest a convergence rate of 95% between informal and formal methods, with a margin of error of 5.3-5.8%. We discuss the implications of this convergence rate for the ongoing conversation about judgment collection methods, and lay out a set of questions for future research into syntactic methodology.

Original languageEnglish (US)
Pages (from-to)219-248
Number of pages30
JournalLingua
Volume134
DOIs
StatePublished - Sep 2 2013

Fingerprint

random sample
linguistics
conversation
Linguistic Inquiry
Acceptability Judgments
methodology
Datum Point

Keywords

  • Acceptability judgments
  • Experimental syntax
  • Grammaticality judgments
  • Methodology

ASJC Scopus subject areas

  • Language and Linguistics
  • Linguistics and Language

Cite this

A comparison of informal and formal acceptability judgments using a random sample from Linguistic Inquiry 2001-2010. / Sprouse, Jon; Schütze, Carson T.; Almeida, Diogo.

In: Lingua, Vol. 134, 02.09.2013, p. 219-248.

Research output: Contribution to journalArticle

@article{2152a421c62f4e36817149a0179482a2,
title = "A comparison of informal and formal acceptability judgments using a random sample from Linguistic Inquiry 2001-2010",
abstract = "The goal of the present study is to provide a direct comparison of the results of informal judgment collection methods with the results of formal judgment collection methods, as a first step in understanding the relative merits of each family of methods. Although previous studies have compared small samples of informal and formal results, this article presents the first large-scale comparison based on a random sample of phenomena from a leading theoretical journal (Linguistic Inquiry). We tested 296 data points from the approximately 1743 English data points that were published in Linguistic Inquiry between 2001 and 2010. We tested this sample with 936 na{\"i}ve participants using three formal judgment tasks (magnitude estimation, 7-point Likert scale, and two-alternative forced-choice) and report five statistical analyses. The results suggest a convergence rate of 95{\%} between informal and formal methods, with a margin of error of 5.3-5.8{\%}. We discuss the implications of this convergence rate for the ongoing conversation about judgment collection methods, and lay out a set of questions for future research into syntactic methodology.",
keywords = "Acceptability judgments, Experimental syntax, Grammaticality judgments, Methodology",
author = "Jon Sprouse and Sch{\"u}tze, {Carson T.} and Diogo Almeida",
year = "2013",
month = "9",
day = "2",
doi = "10.1016/j.lingua.2013.07.002",
language = "English (US)",
volume = "134",
pages = "219--248",
journal = "Lingua",
issn = "0024-3841",
publisher = "Elsevier",

}

TY - JOUR

T1 - A comparison of informal and formal acceptability judgments using a random sample from Linguistic Inquiry 2001-2010

AU - Sprouse, Jon

AU - Schütze, Carson T.

AU - Almeida, Diogo

PY - 2013/9/2

Y1 - 2013/9/2

N2 - The goal of the present study is to provide a direct comparison of the results of informal judgment collection methods with the results of formal judgment collection methods, as a first step in understanding the relative merits of each family of methods. Although previous studies have compared small samples of informal and formal results, this article presents the first large-scale comparison based on a random sample of phenomena from a leading theoretical journal (Linguistic Inquiry). We tested 296 data points from the approximately 1743 English data points that were published in Linguistic Inquiry between 2001 and 2010. We tested this sample with 936 naïve participants using three formal judgment tasks (magnitude estimation, 7-point Likert scale, and two-alternative forced-choice) and report five statistical analyses. The results suggest a convergence rate of 95% between informal and formal methods, with a margin of error of 5.3-5.8%. We discuss the implications of this convergence rate for the ongoing conversation about judgment collection methods, and lay out a set of questions for future research into syntactic methodology.

AB - The goal of the present study is to provide a direct comparison of the results of informal judgment collection methods with the results of formal judgment collection methods, as a first step in understanding the relative merits of each family of methods. Although previous studies have compared small samples of informal and formal results, this article presents the first large-scale comparison based on a random sample of phenomena from a leading theoretical journal (Linguistic Inquiry). We tested 296 data points from the approximately 1743 English data points that were published in Linguistic Inquiry between 2001 and 2010. We tested this sample with 936 naïve participants using three formal judgment tasks (magnitude estimation, 7-point Likert scale, and two-alternative forced-choice) and report five statistical analyses. The results suggest a convergence rate of 95% between informal and formal methods, with a margin of error of 5.3-5.8%. We discuss the implications of this convergence rate for the ongoing conversation about judgment collection methods, and lay out a set of questions for future research into syntactic methodology.

KW - Acceptability judgments

KW - Experimental syntax

KW - Grammaticality judgments

KW - Methodology

UR - http://www.scopus.com/inward/record.url?scp=84884354332&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84884354332&partnerID=8YFLogxK

U2 - 10.1016/j.lingua.2013.07.002

DO - 10.1016/j.lingua.2013.07.002

M3 - Article

AN - SCOPUS:84884354332

VL - 134

SP - 219

EP - 248

JO - Lingua

JF - Lingua

SN - 0024-3841

ER -