Finding the impact in a messy intervention

Using an integrated design to evaluate a comprehensive citywide health initiative

Beth C. Weitzman, Tod Mijanovich, Diana Silver, Charles Brecher

Research output: Contribution to journalArticle

Abstract

This article uses the evaluation of the Robert Wood Johnson Foundation's (RWJF) Urban Health Initiative (UHI), a 10-year effort to improve health and safety outcomes in distressed cities, to demonstrate the strength of an evaluation design that integrates theory of change and quasi-experimental approaches, including the use of comparison cities. This paper focuses on the later stages of implementation and, especially, our methods for estimating program impacts. While the theory of change was used to make preliminary identification of intended outcomes, we used the sites' plans and early implementation to refine this list and revisit our strategy for estimating impacts. Using our integrated design, differences between program and comparison cities are considered impacts only if they were predicted by program theory, local plans for action, and early implementation. We find small, measurable changes in areas of greatest programmatic effort. We discuss the importance of the integrated design in identifying impacts.

Original languageEnglish (US)
Pages (from-to)495-514
Number of pages20
JournalAmerican Journal of Evaluation
Volume30
Issue number4
DOIs
StatePublished - Dec 2009

Fingerprint

Health
health
Urban Health
evaluation
Safety
Integrated design
Theory of change
Urban health
Program theory
Health and safety
Evaluation
Evaluation design

Keywords

  • Child health
  • Comprehensive community initiatives
  • Quasi-experimental designs
  • Theory of change
  • Urban health

ASJC Scopus subject areas

  • Business and International Management
  • Strategy and Management
  • Social Psychology
  • Education
  • Health(social science)
  • Sociology and Political Science

Cite this

Finding the impact in a messy intervention : Using an integrated design to evaluate a comprehensive citywide health initiative. / Weitzman, Beth C.; Mijanovich, Tod; Silver, Diana; Brecher, Charles.

In: American Journal of Evaluation, Vol. 30, No. 4, 12.2009, p. 495-514.

Research output: Contribution to journalArticle

@article{a80e88f676e740bdb4b7e1e52f145acf,
title = "Finding the impact in a messy intervention: Using an integrated design to evaluate a comprehensive citywide health initiative",
abstract = "This article uses the evaluation of the Robert Wood Johnson Foundation's (RWJF) Urban Health Initiative (UHI), a 10-year effort to improve health and safety outcomes in distressed cities, to demonstrate the strength of an evaluation design that integrates theory of change and quasi-experimental approaches, including the use of comparison cities. This paper focuses on the later stages of implementation and, especially, our methods for estimating program impacts. While the theory of change was used to make preliminary identification of intended outcomes, we used the sites' plans and early implementation to refine this list and revisit our strategy for estimating impacts. Using our integrated design, differences between program and comparison cities are considered impacts only if they were predicted by program theory, local plans for action, and early implementation. We find small, measurable changes in areas of greatest programmatic effort. We discuss the importance of the integrated design in identifying impacts.",
keywords = "Child health, Comprehensive community initiatives, Quasi-experimental designs, Theory of change, Urban health",
author = "Weitzman, {Beth C.} and Tod Mijanovich and Diana Silver and Charles Brecher",
year = "2009",
month = "12",
doi = "10.1177/1098214009347555",
language = "English (US)",
volume = "30",
pages = "495--514",
journal = "American Journal of Evaluation",
issn = "1098-2140",
publisher = "SAGE Publications Inc.",
number = "4",

}

TY - JOUR

T1 - Finding the impact in a messy intervention

T2 - Using an integrated design to evaluate a comprehensive citywide health initiative

AU - Weitzman, Beth C.

AU - Mijanovich, Tod

AU - Silver, Diana

AU - Brecher, Charles

PY - 2009/12

Y1 - 2009/12

N2 - This article uses the evaluation of the Robert Wood Johnson Foundation's (RWJF) Urban Health Initiative (UHI), a 10-year effort to improve health and safety outcomes in distressed cities, to demonstrate the strength of an evaluation design that integrates theory of change and quasi-experimental approaches, including the use of comparison cities. This paper focuses on the later stages of implementation and, especially, our methods for estimating program impacts. While the theory of change was used to make preliminary identification of intended outcomes, we used the sites' plans and early implementation to refine this list and revisit our strategy for estimating impacts. Using our integrated design, differences between program and comparison cities are considered impacts only if they were predicted by program theory, local plans for action, and early implementation. We find small, measurable changes in areas of greatest programmatic effort. We discuss the importance of the integrated design in identifying impacts.

AB - This article uses the evaluation of the Robert Wood Johnson Foundation's (RWJF) Urban Health Initiative (UHI), a 10-year effort to improve health and safety outcomes in distressed cities, to demonstrate the strength of an evaluation design that integrates theory of change and quasi-experimental approaches, including the use of comparison cities. This paper focuses on the later stages of implementation and, especially, our methods for estimating program impacts. While the theory of change was used to make preliminary identification of intended outcomes, we used the sites' plans and early implementation to refine this list and revisit our strategy for estimating impacts. Using our integrated design, differences between program and comparison cities are considered impacts only if they were predicted by program theory, local plans for action, and early implementation. We find small, measurable changes in areas of greatest programmatic effort. We discuss the importance of the integrated design in identifying impacts.

KW - Child health

KW - Comprehensive community initiatives

KW - Quasi-experimental designs

KW - Theory of change

KW - Urban health

UR - http://www.scopus.com/inward/record.url?scp=70449581003&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=70449581003&partnerID=8YFLogxK

U2 - 10.1177/1098214009347555

DO - 10.1177/1098214009347555

M3 - Article

VL - 30

SP - 495

EP - 514

JO - American Journal of Evaluation

JF - American Journal of Evaluation

SN - 1098-2140

IS - 4

ER -