Examining the Internal Validity and Statistical Precision of the Comparative Interrupted Time Series Design by Comparison With a Randomized Experiment

Travis St.Clair, Thomas D. Cook, Kelly Hallberg

Research output: Contribution to journalArticle

Abstract

Although evaluators often use an interrupted time series (ITS) design to test hypotheses about program effects, there are few empirical tests of the design's validity. We take a randomized experiment on an educational topic and compare its effects to those from a comparative ITS (CITS) design that uses the same treatment group as the experiment but a nonequivalent comparison group that is assessed at six time points before treatment. We estimate program effects with and without matching of the comparison schools, and we also systematically vary the number of pretest time points in the analysis. CITS designs produce impact estimates that are extremely close to the experimental benchmarks and, as implemented here, do so equally well with and without matching. Adding time points provides an advantage so long as the pretest trend differences in the treatment and comparison groups are correctly modeled. Otherwise, more time points can increase bias.

Original languageEnglish (US)
Pages (from-to)311-327
Number of pages17
JournalAmerican Journal of Evaluation
Volume35
Issue number3
DOIs
StatePublished - 2014

Fingerprint

time series
experiment
Benchmarking
Group
trend
time
Interrupted Time Series Analysis
Internal validity
Randomized experiments
school
Pre-test

Keywords

  • educational evaluation
  • interrupted time series
  • randomized clinical trial
  • within-study comparison

ASJC Scopus subject areas

  • Business and International Management
  • Strategy and Management
  • Social Psychology
  • Education
  • Health(social science)
  • Sociology and Political Science

Cite this

@article{c429eac176e24f1282997d347c09f13a,
title = "Examining the Internal Validity and Statistical Precision of the Comparative Interrupted Time Series Design by Comparison With a Randomized Experiment",
abstract = "Although evaluators often use an interrupted time series (ITS) design to test hypotheses about program effects, there are few empirical tests of the design's validity. We take a randomized experiment on an educational topic and compare its effects to those from a comparative ITS (CITS) design that uses the same treatment group as the experiment but a nonequivalent comparison group that is assessed at six time points before treatment. We estimate program effects with and without matching of the comparison schools, and we also systematically vary the number of pretest time points in the analysis. CITS designs produce impact estimates that are extremely close to the experimental benchmarks and, as implemented here, do so equally well with and without matching. Adding time points provides an advantage so long as the pretest trend differences in the treatment and comparison groups are correctly modeled. Otherwise, more time points can increase bias.",
keywords = "educational evaluation, interrupted time series, randomized clinical trial, within-study comparison",
author = "Travis St.Clair and Cook, {Thomas D.} and Kelly Hallberg",
year = "2014",
doi = "10.1177/1098214014527337",
language = "English (US)",
volume = "35",
pages = "311--327",
journal = "American Journal of Evaluation",
issn = "1098-2140",
publisher = "SAGE Publications Inc.",
number = "3",

}

TY - JOUR

T1 - Examining the Internal Validity and Statistical Precision of the Comparative Interrupted Time Series Design by Comparison With a Randomized Experiment

AU - St.Clair, Travis

AU - Cook, Thomas D.

AU - Hallberg, Kelly

PY - 2014

Y1 - 2014

N2 - Although evaluators often use an interrupted time series (ITS) design to test hypotheses about program effects, there are few empirical tests of the design's validity. We take a randomized experiment on an educational topic and compare its effects to those from a comparative ITS (CITS) design that uses the same treatment group as the experiment but a nonequivalent comparison group that is assessed at six time points before treatment. We estimate program effects with and without matching of the comparison schools, and we also systematically vary the number of pretest time points in the analysis. CITS designs produce impact estimates that are extremely close to the experimental benchmarks and, as implemented here, do so equally well with and without matching. Adding time points provides an advantage so long as the pretest trend differences in the treatment and comparison groups are correctly modeled. Otherwise, more time points can increase bias.

AB - Although evaluators often use an interrupted time series (ITS) design to test hypotheses about program effects, there are few empirical tests of the design's validity. We take a randomized experiment on an educational topic and compare its effects to those from a comparative ITS (CITS) design that uses the same treatment group as the experiment but a nonequivalent comparison group that is assessed at six time points before treatment. We estimate program effects with and without matching of the comparison schools, and we also systematically vary the number of pretest time points in the analysis. CITS designs produce impact estimates that are extremely close to the experimental benchmarks and, as implemented here, do so equally well with and without matching. Adding time points provides an advantage so long as the pretest trend differences in the treatment and comparison groups are correctly modeled. Otherwise, more time points can increase bias.

KW - educational evaluation

KW - interrupted time series

KW - randomized clinical trial

KW - within-study comparison

UR - http://www.scopus.com/inward/record.url?scp=84905181563&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84905181563&partnerID=8YFLogxK

U2 - 10.1177/1098214014527337

DO - 10.1177/1098214014527337

M3 - Article

VL - 35

SP - 311

EP - 327

JO - American Journal of Evaluation

JF - American Journal of Evaluation

SN - 1098-2140

IS - 3

ER -