Toward a more nuanced approach to program effectiveness assessment: Hierarchical linear models in K-12 program evaluation

Xiaoxia A. Newton, Lorena Llosa

Research output: Contribution to journalArticle

Abstract

Most K-12 evaluations are designed to make inferences about how a program implemented at the classroom or school level affects student learning outcomes and such inferences inherently involve hierarchical data structure. One methodological challenge for evaluators is linking program implementation factors typically measured at the classroom or teacher level with student outcomes measured at the individual student level. Hierarchical linear modeling (HLM) is ideal for K-12 program evaluations because it can appropriately handle hierarchical data while allowing for a nuanced conceptualization of evaluation questions. The authors illustrate the advantages of HLM with an example of the evaluation of a technology-based reading program for elementary students. HLM enabled evaluators to have a deeper understanding of the relationship between program implementation and program outcomes by (a) providing a better and more proper estimate of the average program outcome, (b) providing an estimate of variation in student outcomes within classrooms, between classrooms, and between schools, and (c) most importantly, by providing a framework for probing what factors are related to variations in student outcomes. Through illustrating the potentials of HLM, the authors aim to advance and expand evaluative tools for establishing empirical evidence regarding program effectiveness in K-12 settings. The authors also hope to bring balance and additional insight into the current methodological discussions about the relationship between program implementation and program impact, rather than merely demonstrating the average effect of a program or policy intervention.

Original languageEnglish (US)
Pages (from-to)162-179
Number of pages18
JournalAmerican Journal of Evaluation
Volume31
Issue number2
DOIs
StatePublished - Jun 2010

Fingerprint

Program Evaluation
linear model
Linear Models
Students
evaluation
Hope
classroom
student
Program evaluation
Hierarchical linear models
Reading
Learning
Technology
school
Hierarchical linear modeling
Program implementation
Evaluation

Keywords

  • Hierarchical linear modeling
  • K-12 program evaluation
  • Quasi-experiment
  • What works

ASJC Scopus subject areas

  • Business and International Management
  • Strategy and Management
  • Social Psychology
  • Education
  • Health(social science)
  • Sociology and Political Science

Cite this

Toward a more nuanced approach to program effectiveness assessment : Hierarchical linear models in K-12 program evaluation. / Newton, Xiaoxia A.; Llosa, Lorena.

In: American Journal of Evaluation, Vol. 31, No. 2, 06.2010, p. 162-179.

Research output: Contribution to journalArticle

@article{b8b5b2338d024ec3a54c326fe12be830,
title = "Toward a more nuanced approach to program effectiveness assessment: Hierarchical linear models in K-12 program evaluation",
abstract = "Most K-12 evaluations are designed to make inferences about how a program implemented at the classroom or school level affects student learning outcomes and such inferences inherently involve hierarchical data structure. One methodological challenge for evaluators is linking program implementation factors typically measured at the classroom or teacher level with student outcomes measured at the individual student level. Hierarchical linear modeling (HLM) is ideal for K-12 program evaluations because it can appropriately handle hierarchical data while allowing for a nuanced conceptualization of evaluation questions. The authors illustrate the advantages of HLM with an example of the evaluation of a technology-based reading program for elementary students. HLM enabled evaluators to have a deeper understanding of the relationship between program implementation and program outcomes by (a) providing a better and more proper estimate of the average program outcome, (b) providing an estimate of variation in student outcomes within classrooms, between classrooms, and between schools, and (c) most importantly, by providing a framework for probing what factors are related to variations in student outcomes. Through illustrating the potentials of HLM, the authors aim to advance and expand evaluative tools for establishing empirical evidence regarding program effectiveness in K-12 settings. The authors also hope to bring balance and additional insight into the current methodological discussions about the relationship between program implementation and program impact, rather than merely demonstrating the average effect of a program or policy intervention.",
keywords = "Hierarchical linear modeling, K-12 program evaluation, Quasi-experiment, What works",
author = "Newton, {Xiaoxia A.} and Lorena Llosa",
year = "2010",
month = "6",
doi = "10.1177/1098214010363022",
language = "English (US)",
volume = "31",
pages = "162--179",
journal = "American Journal of Evaluation",
issn = "1098-2140",
publisher = "SAGE Publications Inc.",
number = "2",

}

TY - JOUR

T1 - Toward a more nuanced approach to program effectiveness assessment

T2 - Hierarchical linear models in K-12 program evaluation

AU - Newton, Xiaoxia A.

AU - Llosa, Lorena

PY - 2010/6

Y1 - 2010/6

N2 - Most K-12 evaluations are designed to make inferences about how a program implemented at the classroom or school level affects student learning outcomes and such inferences inherently involve hierarchical data structure. One methodological challenge for evaluators is linking program implementation factors typically measured at the classroom or teacher level with student outcomes measured at the individual student level. Hierarchical linear modeling (HLM) is ideal for K-12 program evaluations because it can appropriately handle hierarchical data while allowing for a nuanced conceptualization of evaluation questions. The authors illustrate the advantages of HLM with an example of the evaluation of a technology-based reading program for elementary students. HLM enabled evaluators to have a deeper understanding of the relationship between program implementation and program outcomes by (a) providing a better and more proper estimate of the average program outcome, (b) providing an estimate of variation in student outcomes within classrooms, between classrooms, and between schools, and (c) most importantly, by providing a framework for probing what factors are related to variations in student outcomes. Through illustrating the potentials of HLM, the authors aim to advance and expand evaluative tools for establishing empirical evidence regarding program effectiveness in K-12 settings. The authors also hope to bring balance and additional insight into the current methodological discussions about the relationship between program implementation and program impact, rather than merely demonstrating the average effect of a program or policy intervention.

AB - Most K-12 evaluations are designed to make inferences about how a program implemented at the classroom or school level affects student learning outcomes and such inferences inherently involve hierarchical data structure. One methodological challenge for evaluators is linking program implementation factors typically measured at the classroom or teacher level with student outcomes measured at the individual student level. Hierarchical linear modeling (HLM) is ideal for K-12 program evaluations because it can appropriately handle hierarchical data while allowing for a nuanced conceptualization of evaluation questions. The authors illustrate the advantages of HLM with an example of the evaluation of a technology-based reading program for elementary students. HLM enabled evaluators to have a deeper understanding of the relationship between program implementation and program outcomes by (a) providing a better and more proper estimate of the average program outcome, (b) providing an estimate of variation in student outcomes within classrooms, between classrooms, and between schools, and (c) most importantly, by providing a framework for probing what factors are related to variations in student outcomes. Through illustrating the potentials of HLM, the authors aim to advance and expand evaluative tools for establishing empirical evidence regarding program effectiveness in K-12 settings. The authors also hope to bring balance and additional insight into the current methodological discussions about the relationship between program implementation and program impact, rather than merely demonstrating the average effect of a program or policy intervention.

KW - Hierarchical linear modeling

KW - K-12 program evaluation

KW - Quasi-experiment

KW - What works

UR - http://www.scopus.com/inward/record.url?scp=77952815506&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=77952815506&partnerID=8YFLogxK

U2 - 10.1177/1098214010363022

DO - 10.1177/1098214010363022

M3 - Article

AN - SCOPUS:77952815506

VL - 31

SP - 162

EP - 179

JO - American Journal of Evaluation

JF - American Journal of Evaluation

SN - 1098-2140

IS - 2

ER -