Integrating a comparison group design into a theory of change evaluation: The case of the urban health initiative

Beth C. Weitzman, Diana Silver, Keri Nicole Dillman

Research output: Contribution to journalArticle

Abstract

This paper describes how we strengthened the theory of change approach to evaluating a complex social initiative by integrating it with a quasi-experimental, comparison group design. We also demonstrate the plausibility of selecting a credible comparison group through the use of cluster analysis, and describe our work in validating that analysis with additional measures. The integrated evaluation design relies on two points of comparison: (1) program theory to program experience; and (2) program cities to comparison cities. We describe how we are using this integrated design to evaluate the Robert Wood Johnson Foundation's Urban Health Initiative, an effort that aims to improve health and safety outcomes for children and youth in five distressed urban areas through a process of citywide, multi-sector planning and changed public and private systems. We also discuss how the use of two research frameworks and multiple methods can enrich our ability to test underlying assumptions and evaluate overall program effects. Using this integrated approach has provided evidence that the earliest phases of this initiative are unfolding as the theory would predict, and that the comparison cities are not undergoing a similar experience to those in UHI. Despite many remaining limitations, this integrated evaluation can provide greater confidence in assessing whether future changes in health and safety outcomes may have resulted from the Urban Health Initiative (UHI).

Original languageEnglish (US)
Pages (from-to)371-385
Number of pages15
JournalAmerican Journal of Evaluation
Volume23
Issue number4
DOIs
StatePublished - Dec 2002

Fingerprint

Urban Health
health
evaluation
Safety
Group
Health
Cluster Analysis
cluster analysis
urban area
Research
experience
confidence
Theory of change
Urban health
Evaluation
planning
ability
evidence

ASJC Scopus subject areas

  • Business and International Management
  • Strategy and Management
  • Social Psychology
  • Education
  • Health(social science)
  • Sociology and Political Science

Cite this

Integrating a comparison group design into a theory of change evaluation : The case of the urban health initiative. / Weitzman, Beth C.; Silver, Diana; Dillman, Keri Nicole.

In: American Journal of Evaluation, Vol. 23, No. 4, 12.2002, p. 371-385.

Research output: Contribution to journalArticle

@article{6540585288af4196af36b3a5b7f60e25,
title = "Integrating a comparison group design into a theory of change evaluation: The case of the urban health initiative",
abstract = "This paper describes how we strengthened the theory of change approach to evaluating a complex social initiative by integrating it with a quasi-experimental, comparison group design. We also demonstrate the plausibility of selecting a credible comparison group through the use of cluster analysis, and describe our work in validating that analysis with additional measures. The integrated evaluation design relies on two points of comparison: (1) program theory to program experience; and (2) program cities to comparison cities. We describe how we are using this integrated design to evaluate the Robert Wood Johnson Foundation's Urban Health Initiative, an effort that aims to improve health and safety outcomes for children and youth in five distressed urban areas through a process of citywide, multi-sector planning and changed public and private systems. We also discuss how the use of two research frameworks and multiple methods can enrich our ability to test underlying assumptions and evaluate overall program effects. Using this integrated approach has provided evidence that the earliest phases of this initiative are unfolding as the theory would predict, and that the comparison cities are not undergoing a similar experience to those in UHI. Despite many remaining limitations, this integrated evaluation can provide greater confidence in assessing whether future changes in health and safety outcomes may have resulted from the Urban Health Initiative (UHI).",
author = "Weitzman, {Beth C.} and Diana Silver and Dillman, {Keri Nicole}",
year = "2002",
month = "12",
doi = "10.1016/S1098-2140(02)00236-9",
language = "English (US)",
volume = "23",
pages = "371--385",
journal = "American Journal of Evaluation",
issn = "1098-2140",
publisher = "SAGE Publications Inc.",
number = "4",

}

TY - JOUR

T1 - Integrating a comparison group design into a theory of change evaluation

T2 - The case of the urban health initiative

AU - Weitzman, Beth C.

AU - Silver, Diana

AU - Dillman, Keri Nicole

PY - 2002/12

Y1 - 2002/12

N2 - This paper describes how we strengthened the theory of change approach to evaluating a complex social initiative by integrating it with a quasi-experimental, comparison group design. We also demonstrate the plausibility of selecting a credible comparison group through the use of cluster analysis, and describe our work in validating that analysis with additional measures. The integrated evaluation design relies on two points of comparison: (1) program theory to program experience; and (2) program cities to comparison cities. We describe how we are using this integrated design to evaluate the Robert Wood Johnson Foundation's Urban Health Initiative, an effort that aims to improve health and safety outcomes for children and youth in five distressed urban areas through a process of citywide, multi-sector planning and changed public and private systems. We also discuss how the use of two research frameworks and multiple methods can enrich our ability to test underlying assumptions and evaluate overall program effects. Using this integrated approach has provided evidence that the earliest phases of this initiative are unfolding as the theory would predict, and that the comparison cities are not undergoing a similar experience to those in UHI. Despite many remaining limitations, this integrated evaluation can provide greater confidence in assessing whether future changes in health and safety outcomes may have resulted from the Urban Health Initiative (UHI).

AB - This paper describes how we strengthened the theory of change approach to evaluating a complex social initiative by integrating it with a quasi-experimental, comparison group design. We also demonstrate the plausibility of selecting a credible comparison group through the use of cluster analysis, and describe our work in validating that analysis with additional measures. The integrated evaluation design relies on two points of comparison: (1) program theory to program experience; and (2) program cities to comparison cities. We describe how we are using this integrated design to evaluate the Robert Wood Johnson Foundation's Urban Health Initiative, an effort that aims to improve health and safety outcomes for children and youth in five distressed urban areas through a process of citywide, multi-sector planning and changed public and private systems. We also discuss how the use of two research frameworks and multiple methods can enrich our ability to test underlying assumptions and evaluate overall program effects. Using this integrated approach has provided evidence that the earliest phases of this initiative are unfolding as the theory would predict, and that the comparison cities are not undergoing a similar experience to those in UHI. Despite many remaining limitations, this integrated evaluation can provide greater confidence in assessing whether future changes in health and safety outcomes may have resulted from the Urban Health Initiative (UHI).

UR - http://www.scopus.com/inward/record.url?scp=0036891638&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=0036891638&partnerID=8YFLogxK

U2 - 10.1016/S1098-2140(02)00236-9

DO - 10.1016/S1098-2140(02)00236-9

M3 - Article

AN - SCOPUS:0036891638

VL - 23

SP - 371

EP - 385

JO - American Journal of Evaluation

JF - American Journal of Evaluation

SN - 1098-2140

IS - 4

ER -