Investigating Differences between American and Indian Raters in Assessing TOEFL iBT Speaking Tasks

Jing Wei, Lorena Llosa

Research output: Contribution to journalArticle

Abstract

This article reports on an investigation of the role raters language background plays in raters assessment of test takers speaking ability. Specifically, this article examines differences between American and Indian raters in their scores and scoring processes when rating Indian test takers responses to the Test of English as a Foreign LanguageTM Internet-Based Test (TOEFL iBT®) Speaking tasks. Three American and three Indian raters were asked to score 60 speech samples from 10 Indian test takers responses to TOEFL iBT Speaking tasks and to perform think-aloud protocols while scoring. The data were analyzed with Multifaceted Rasch and verbal protocol analyses. Findings indicate that Indian raters were better than American raters at identifying and understanding features of Indian English in the test takers responses. However, Indian and American raters did not differ in their use of scoring criteria, their attitudes toward Indian English, or in the internal consistency and severity of the scores.

Original languageEnglish (US)
Pages (from-to)283-304
Number of pages22
JournalLanguage Assessment Quarterly
Volume12
Issue number3
DOIs
StatePublished - Jul 3 2015

Fingerprint

speaking
Internet
Raters
World Wide Web
rating
ability
language
Scoring

ASJC Scopus subject areas

  • Language and Linguistics
  • Linguistics and Language

Cite this

Investigating Differences between American and Indian Raters in Assessing TOEFL iBT Speaking Tasks. / Wei, Jing; Llosa, Lorena.

In: Language Assessment Quarterly, Vol. 12, No. 3, 03.07.2015, p. 283-304.

Research output: Contribution to journalArticle

@article{417b786d09214451a248d2486b457cb7,
title = "Investigating Differences between American and Indian Raters in Assessing TOEFL iBT Speaking Tasks",
abstract = "This article reports on an investigation of the role raters language background plays in raters assessment of test takers speaking ability. Specifically, this article examines differences between American and Indian raters in their scores and scoring processes when rating Indian test takers responses to the Test of English as a Foreign LanguageTM Internet-Based Test (TOEFL iBT{\circledR}) Speaking tasks. Three American and three Indian raters were asked to score 60 speech samples from 10 Indian test takers responses to TOEFL iBT Speaking tasks and to perform think-aloud protocols while scoring. The data were analyzed with Multifaceted Rasch and verbal protocol analyses. Findings indicate that Indian raters were better than American raters at identifying and understanding features of Indian English in the test takers responses. However, Indian and American raters did not differ in their use of scoring criteria, their attitudes toward Indian English, or in the internal consistency and severity of the scores.",
author = "Jing Wei and Lorena Llosa",
year = "2015",
month = "7",
day = "3",
doi = "10.1080/15434303.2015.1037446",
language = "English (US)",
volume = "12",
pages = "283--304",
journal = "Language Assessment Quarterly",
issn = "1543-4303",
publisher = "Routledge",
number = "3",

}

TY - JOUR

T1 - Investigating Differences between American and Indian Raters in Assessing TOEFL iBT Speaking Tasks

AU - Wei, Jing

AU - Llosa, Lorena

PY - 2015/7/3

Y1 - 2015/7/3

N2 - This article reports on an investigation of the role raters language background plays in raters assessment of test takers speaking ability. Specifically, this article examines differences between American and Indian raters in their scores and scoring processes when rating Indian test takers responses to the Test of English as a Foreign LanguageTM Internet-Based Test (TOEFL iBT®) Speaking tasks. Three American and three Indian raters were asked to score 60 speech samples from 10 Indian test takers responses to TOEFL iBT Speaking tasks and to perform think-aloud protocols while scoring. The data were analyzed with Multifaceted Rasch and verbal protocol analyses. Findings indicate that Indian raters were better than American raters at identifying and understanding features of Indian English in the test takers responses. However, Indian and American raters did not differ in their use of scoring criteria, their attitudes toward Indian English, or in the internal consistency and severity of the scores.

AB - This article reports on an investigation of the role raters language background plays in raters assessment of test takers speaking ability. Specifically, this article examines differences between American and Indian raters in their scores and scoring processes when rating Indian test takers responses to the Test of English as a Foreign LanguageTM Internet-Based Test (TOEFL iBT®) Speaking tasks. Three American and three Indian raters were asked to score 60 speech samples from 10 Indian test takers responses to TOEFL iBT Speaking tasks and to perform think-aloud protocols while scoring. The data were analyzed with Multifaceted Rasch and verbal protocol analyses. Findings indicate that Indian raters were better than American raters at identifying and understanding features of Indian English in the test takers responses. However, Indian and American raters did not differ in their use of scoring criteria, their attitudes toward Indian English, or in the internal consistency and severity of the scores.

UR - http://www.scopus.com/inward/record.url?scp=84939202618&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84939202618&partnerID=8YFLogxK

U2 - 10.1080/15434303.2015.1037446

DO - 10.1080/15434303.2015.1037446

M3 - Article

VL - 12

SP - 283

EP - 304

JO - Language Assessment Quarterly

JF - Language Assessment Quarterly

SN - 1543-4303

IS - 3

ER -