Estimating Causal Effects of Education Interventions Using a Two-Rating Regression Discontinuity Design: Lessons From a Simulation Study and an Application

Kristin E. Porter, Sean F. Reardon, Fatih Unlu, Howard S. Bloom, Joseph R. Cimpian

Research output: Contribution to journalArticle

Abstract

A valuable extension of the single-rating regression discontinuity design (RDD) is a multiple-rating RDD (MRRDD). To date, four main methods have been used to estimate average treatment effects at the multiple treatment frontiers of an MRRDD: the “surface” method, the “frontier” method, the “binding-score” method, and the “fuzzy instrumental variables” method. This article uses a series of simulations to evaluate the relative performance of each of these four methods under a variety of different data-generating models. Focusing on a two-rating RDD (2RRDD), we compare the methods in terms of their bias, precision, and mean squared error when implemented as they most likely would be in practice—using optimal bandwidth selection. We also apply the lessons learned from the simulations to a real-world example that uses data from a study of an English learner reclassification policy. Overall, this article makes valuable contributions to the literature on MRRDDs in that it makes concrete recommendations for choosing among MRRDD estimation methods, for implementing any chosen method using local linear regression, and for providing accurate statistical inferences.

Original languageEnglish (US)
Pages (from-to)138-167
Number of pages30
JournalJournal of Research on Educational Effectiveness
Volume10
Issue number1
DOIs
StatePublished - Jan 2 2017

Fingerprint

rating
regression
simulation
education
trend
performance

Keywords

  • multiple ratings
  • RDD
  • regression discontinuity design

ASJC Scopus subject areas

  • Education

Cite this

Estimating Causal Effects of Education Interventions Using a Two-Rating Regression Discontinuity Design : Lessons From a Simulation Study and an Application. / Porter, Kristin E.; Reardon, Sean F.; Unlu, Fatih; Bloom, Howard S.; Cimpian, Joseph R.

In: Journal of Research on Educational Effectiveness, Vol. 10, No. 1, 02.01.2017, p. 138-167.

Research output: Contribution to journalArticle

@article{ddcac6927d2b41f38a10253d179c3950,
title = "Estimating Causal Effects of Education Interventions Using a Two-Rating Regression Discontinuity Design: Lessons From a Simulation Study and an Application",
abstract = "A valuable extension of the single-rating regression discontinuity design (RDD) is a multiple-rating RDD (MRRDD). To date, four main methods have been used to estimate average treatment effects at the multiple treatment frontiers of an MRRDD: the “surface” method, the “frontier” method, the “binding-score” method, and the “fuzzy instrumental variables” method. This article uses a series of simulations to evaluate the relative performance of each of these four methods under a variety of different data-generating models. Focusing on a two-rating RDD (2RRDD), we compare the methods in terms of their bias, precision, and mean squared error when implemented as they most likely would be in practice—using optimal bandwidth selection. We also apply the lessons learned from the simulations to a real-world example that uses data from a study of an English learner reclassification policy. Overall, this article makes valuable contributions to the literature on MRRDDs in that it makes concrete recommendations for choosing among MRRDD estimation methods, for implementing any chosen method using local linear regression, and for providing accurate statistical inferences.",
keywords = "multiple ratings, RDD, regression discontinuity design",
author = "Porter, {Kristin E.} and Reardon, {Sean F.} and Fatih Unlu and Bloom, {Howard S.} and Cimpian, {Joseph R.}",
year = "2017",
month = "1",
day = "2",
doi = "10.1080/19345747.2016.1219436",
language = "English (US)",
volume = "10",
pages = "138--167",
journal = "Journal of Research on Educational Effectiveness",
issn = "1934-5747",
publisher = "Routledge",
number = "1",

}

TY - JOUR

T1 - Estimating Causal Effects of Education Interventions Using a Two-Rating Regression Discontinuity Design

T2 - Lessons From a Simulation Study and an Application

AU - Porter, Kristin E.

AU - Reardon, Sean F.

AU - Unlu, Fatih

AU - Bloom, Howard S.

AU - Cimpian, Joseph R.

PY - 2017/1/2

Y1 - 2017/1/2

N2 - A valuable extension of the single-rating regression discontinuity design (RDD) is a multiple-rating RDD (MRRDD). To date, four main methods have been used to estimate average treatment effects at the multiple treatment frontiers of an MRRDD: the “surface” method, the “frontier” method, the “binding-score” method, and the “fuzzy instrumental variables” method. This article uses a series of simulations to evaluate the relative performance of each of these four methods under a variety of different data-generating models. Focusing on a two-rating RDD (2RRDD), we compare the methods in terms of their bias, precision, and mean squared error when implemented as they most likely would be in practice—using optimal bandwidth selection. We also apply the lessons learned from the simulations to a real-world example that uses data from a study of an English learner reclassification policy. Overall, this article makes valuable contributions to the literature on MRRDDs in that it makes concrete recommendations for choosing among MRRDD estimation methods, for implementing any chosen method using local linear regression, and for providing accurate statistical inferences.

AB - A valuable extension of the single-rating regression discontinuity design (RDD) is a multiple-rating RDD (MRRDD). To date, four main methods have been used to estimate average treatment effects at the multiple treatment frontiers of an MRRDD: the “surface” method, the “frontier” method, the “binding-score” method, and the “fuzzy instrumental variables” method. This article uses a series of simulations to evaluate the relative performance of each of these four methods under a variety of different data-generating models. Focusing on a two-rating RDD (2RRDD), we compare the methods in terms of their bias, precision, and mean squared error when implemented as they most likely would be in practice—using optimal bandwidth selection. We also apply the lessons learned from the simulations to a real-world example that uses data from a study of an English learner reclassification policy. Overall, this article makes valuable contributions to the literature on MRRDDs in that it makes concrete recommendations for choosing among MRRDD estimation methods, for implementing any chosen method using local linear regression, and for providing accurate statistical inferences.

KW - multiple ratings

KW - RDD

KW - regression discontinuity design

UR - http://www.scopus.com/inward/record.url?scp=84992109511&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84992109511&partnerID=8YFLogxK

U2 - 10.1080/19345747.2016.1219436

DO - 10.1080/19345747.2016.1219436

M3 - Article

VL - 10

SP - 138

EP - 167

JO - Journal of Research on Educational Effectiveness

JF - Journal of Research on Educational Effectiveness

SN - 1934-5747

IS - 1

ER -