Building and supporting a validity argument for a standards-based classroom assessment of English proficiency based on teacher judgments

Research output: Contribution to journalArticle

Abstract

Using an argument-based approach to validation, this study examines the quality of teacher judgments in the context of a standards-based classroom assessment of English proficiency. Using Bachman's (2005) assessment use argument (AUA) as a framework for the investigation, this paper first articulates the claims, warrants, rebuttals, and backing needed to justify the link between teachers' scores on the English Language Development (ELD) Classroom Assessment and the interpretations made about students' language ability. Then the paper summarizes the findings of two studies - one quantitative and one qualitative - conducted to gather the necessary backing to support the warrants and, in particular, address the rebuttals about teacher judgments in the argument. The quantitative study examined the assessment in relation to another measure of the same ability - the California English Language Development Test - using confirmatory factor analysis of multitrait-multimethod data and provided evidence in support of the warrant that states that the ELD Classroom Assessment measures English proficiency as defined by the California ELD Standards. The qualitative study examined the processes teachers engaged in while scoring the classroom assessment using verbal protocol analysis. The findings of this study serve to support the rebuttals in the validity argument that state that there are inconsistencies in teachers' scoring. The paper concludes by providing an explanation for these seemingly contradictory findings using the AUA as a framework and discusses the implications of the findings for the use of standards-based classroom assessments based on teacher judgments.

Original languageEnglish (US)
Pages (from-to)32-42
Number of pages11
JournalEducational Measurement: Issues and Practice
Volume27
Issue number3
DOIs
StatePublished - Sep 2008

Fingerprint

classroom
teacher
English language
ability
factor analysis
interpretation
language
evidence
student

Keywords

  • English learners
  • Language proficiency
  • Teacher judgments
  • Validity

ASJC Scopus subject areas

  • Education

Cite this

@article{bdcfc6f3edd24cda933e7626dcb9117d,
title = "Building and supporting a validity argument for a standards-based classroom assessment of English proficiency based on teacher judgments",
abstract = "Using an argument-based approach to validation, this study examines the quality of teacher judgments in the context of a standards-based classroom assessment of English proficiency. Using Bachman's (2005) assessment use argument (AUA) as a framework for the investigation, this paper first articulates the claims, warrants, rebuttals, and backing needed to justify the link between teachers' scores on the English Language Development (ELD) Classroom Assessment and the interpretations made about students' language ability. Then the paper summarizes the findings of two studies - one quantitative and one qualitative - conducted to gather the necessary backing to support the warrants and, in particular, address the rebuttals about teacher judgments in the argument. The quantitative study examined the assessment in relation to another measure of the same ability - the California English Language Development Test - using confirmatory factor analysis of multitrait-multimethod data and provided evidence in support of the warrant that states that the ELD Classroom Assessment measures English proficiency as defined by the California ELD Standards. The qualitative study examined the processes teachers engaged in while scoring the classroom assessment using verbal protocol analysis. The findings of this study serve to support the rebuttals in the validity argument that state that there are inconsistencies in teachers' scoring. The paper concludes by providing an explanation for these seemingly contradictory findings using the AUA as a framework and discusses the implications of the findings for the use of standards-based classroom assessments based on teacher judgments.",
keywords = "English learners, Language proficiency, Teacher judgments, Validity",
author = "Lorena Llosa",
year = "2008",
month = "9",
doi = "10.1111/j.1745-3992.2008.00126.x",
language = "English (US)",
volume = "27",
pages = "32--42",
journal = "Educational Measurement: Issues and Practice",
issn = "0731-1745",
publisher = "Wiley-Blackwell",
number = "3",

}

TY - JOUR

T1 - Building and supporting a validity argument for a standards-based classroom assessment of English proficiency based on teacher judgments

AU - Llosa, Lorena

PY - 2008/9

Y1 - 2008/9

N2 - Using an argument-based approach to validation, this study examines the quality of teacher judgments in the context of a standards-based classroom assessment of English proficiency. Using Bachman's (2005) assessment use argument (AUA) as a framework for the investigation, this paper first articulates the claims, warrants, rebuttals, and backing needed to justify the link between teachers' scores on the English Language Development (ELD) Classroom Assessment and the interpretations made about students' language ability. Then the paper summarizes the findings of two studies - one quantitative and one qualitative - conducted to gather the necessary backing to support the warrants and, in particular, address the rebuttals about teacher judgments in the argument. The quantitative study examined the assessment in relation to another measure of the same ability - the California English Language Development Test - using confirmatory factor analysis of multitrait-multimethod data and provided evidence in support of the warrant that states that the ELD Classroom Assessment measures English proficiency as defined by the California ELD Standards. The qualitative study examined the processes teachers engaged in while scoring the classroom assessment using verbal protocol analysis. The findings of this study serve to support the rebuttals in the validity argument that state that there are inconsistencies in teachers' scoring. The paper concludes by providing an explanation for these seemingly contradictory findings using the AUA as a framework and discusses the implications of the findings for the use of standards-based classroom assessments based on teacher judgments.

AB - Using an argument-based approach to validation, this study examines the quality of teacher judgments in the context of a standards-based classroom assessment of English proficiency. Using Bachman's (2005) assessment use argument (AUA) as a framework for the investigation, this paper first articulates the claims, warrants, rebuttals, and backing needed to justify the link between teachers' scores on the English Language Development (ELD) Classroom Assessment and the interpretations made about students' language ability. Then the paper summarizes the findings of two studies - one quantitative and one qualitative - conducted to gather the necessary backing to support the warrants and, in particular, address the rebuttals about teacher judgments in the argument. The quantitative study examined the assessment in relation to another measure of the same ability - the California English Language Development Test - using confirmatory factor analysis of multitrait-multimethod data and provided evidence in support of the warrant that states that the ELD Classroom Assessment measures English proficiency as defined by the California ELD Standards. The qualitative study examined the processes teachers engaged in while scoring the classroom assessment using verbal protocol analysis. The findings of this study serve to support the rebuttals in the validity argument that state that there are inconsistencies in teachers' scoring. The paper concludes by providing an explanation for these seemingly contradictory findings using the AUA as a framework and discusses the implications of the findings for the use of standards-based classroom assessments based on teacher judgments.

KW - English learners

KW - Language proficiency

KW - Teacher judgments

KW - Validity

UR - http://www.scopus.com/inward/record.url?scp=50849111054&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=50849111054&partnerID=8YFLogxK

U2 - 10.1111/j.1745-3992.2008.00126.x

DO - 10.1111/j.1745-3992.2008.00126.x

M3 - Article

VL - 27

SP - 32

EP - 42

JO - Educational Measurement: Issues and Practice

JF - Educational Measurement: Issues and Practice

SN - 0731-1745

IS - 3

ER -