Module 3: False Counterfactuals
Overview
This module reviews the weaknesses of the two false counterfactual identification strategies: pre vs. post designs and participant vs. non-participant comparisons. We review these issues through a case study of the Millennium Villages Project.
This module contains three reading assignments (one of which includes a paper and two subsequent comments), 29 minutes of video lecture, and an empirical exercise.
Readings
3.1 Impact Evaluation in Practice: Chapter 3
Review Questions
- How can we describe the causal impact of a program in terms of potential outcomes with and without treatment?
- What is a counterfactual?
- In what ways (three) must a valid comparison group be similar to the treatment group?
- What is an intent-to-treat estimate of a program's impact, and how does it differ from a treatment-on-the-treated estimate?
- What are the two false (or "counterfeit") counterfactuals, and what is the source of bias associated with each one?
3.2 The Millennium Villages evaluation and subsequent commentary
Note: the link should take you to the original published version; the version available on the Lancet website is corrected. You’ll know you are reading the original version if the last sentence of the abstract section titled Findings is: “The average annual rate of reduction of mortality in children younger than 5 years of age was three-times faster in Millennium Village sites than in the most recent 10-year national rural trends (7.8% vs 2.5%). The original version is also posted on glow (Williams students only).
Concerns about the Millennium Villages project report
Errors in a paper on the Millennium Villages project
Review Questions
- What is the Millennium Villages project? When and where was it implemented? What were the anticipated impacts?
- What is the primary outcome used in the study?
- What strategies do the authors use to characterize counterfactual outcomes in the absence of the project?
- What are the main "impacts" of the Millennium Villages project reported in the study?
- Describe the three critiques discussed in the letter to the editor by Jesse Bump, Michael Clemens, Gabriel Demombynes, and Lawrence Haddad.
- How do the authors of the original piece respond to these criticisms of their analysis?
3.3 When Does Rigorous Impact Evaluation Make a Difference? The Case of the Millennium Villages
Review Questions
- How would you characterize the effects of previous model village interventions (prior to the Millennium Villages project)?
- The authors compare trends (in outcomes of interest) in Millennium Villages to trends in what types of comparison areas?
- How do trends (in outcomes of interest) in Millennium Villages compare with trends in comparable rural areas?
Video Lectures
Lecture 3.1 False Counterfactuals (plus some extra math for fun) (28:58)
Review Questions
- What are the two false counterfactuals? What is the likely source of bias in each one?
- What are the two ways that you can use Stata to test whether the mean of a single variable differs between two groups?
- When you regress an outcome variable on (nothing but) a single dummy variable and a constant, how do the estimated regression coefficients relate to the means of the groups defined by the dummy variable?
Empirical Exercise
In this exercise, we will use the same data set as in Exercise 1, E1-CohenEtAl-data.dta, a subset of the data used in the paper Price Subsidies, Diagnostic Tests, and Targeting of Malaria Treatment: Evidence from a Randomized Controlled Trial by Jessica Cohen, Pascaline Dupas, and Simone Schaner, published in the American Economic Review in 2015.
In this exercise, we’ll look at the way beliefs about how malaria is transmitted differ across households. We’ll review Stata’s generate and egen commands, and we’ll also practice applying the concepts covered in lecture and the readings, testing the equality of means across two groups using several different methods.
You can download the activity as a do file or a pdf.