Portal de Periódicos da CAPES

Concerns about the Millennium Villages project report

2012; Elsevier BV; Volume: 379; Issue: 9830 Linguagem: Inglês

10.1016/s0140-6736(12)60848-4

1474-547X

Jesse B. Bump, Michael A. Clemens, Gabriel Demombynes, Lawrence Haddad,

Poverty, Education, and Child Welfare

We have three concerns about the research methods of Paul Pronyk and colleagues (published online May 8).1Pronyk PM Muniz M Nemser B et al.for the Millennium Villages Study GroupThe effect of an integrated multisector model for achieving the Millennium Development Goals and improving child survival in rural sub-Saharan Africa: a non-randomised controlled assessment.Lancet. 2012; (published online May 8.)https://doi.org/10.1016/S0140-6736(12)60207-4Summary Full Text Full Text PDF Scopus (57) Google Scholar Each of these concerns separately threatens the basis of one of the study's main findings—that child mortality fell at an annual rate "three-times faster in Millennium Village sites than in the most recent 10-year national rural trends (7·8% vs 2·6%)". First, Pronyk and colleagues do not mention that this headline finding is statistically insignificant, nor do they display anywhere the relevant 95% CI around the estimated on-site decline of 7·8%. Information in their table 2 allows estimation of that interval as about 1·4–15·3% (calculation available on request), which overlaps with Pronyk and colleagues' reported 2·6% background rate of decline. Second, we believe the reported 7·8% annual decline in child mortality is calculated erroneously. It is annualised from a 21·7% cumulative decline in mortality from baseline to endline, ostensibly over a period of 3 years. This is incorrect because the baseline period includes 5 years of births before baseline, whereas the endline period includes only 3 years of births before endline. This means that the births in the baseline datapoint occurred on average 2·5 years before baseline year, whereas the births in the endline datapoint occurred on average 1·5 years after the baseline year. In expectation, the average elapsed time between a baseline birth and an endline birth is therefore 4 years, not 3 years. A 21·7% decline in mortality between births separated on average by 4 years means that the mortality rate declined by 5·9% per year. This problem arises because the study did not use baseline and endline data collection periods of identical duration, as is standard in measuring mortality changes over time. The above analysis would require refinement if there was a structural break in the time series of instantaneous birth rates or death rates. But Pronyk and colleagues do not use such a model; they use a simpler model that includes an incorrect calculation. Furthermore, there is no evidence of such a structural break in instantaneous rates, nor could the study's two datapoints per site provide such evidence. Third, the study greatly underestimates the background nationwide rate of change in mortality. The paper states "Analysis of demographic and health surveys for countries included in the assessment shows that the average annual rate of reduction for mortality in rural areas for the period 1990–2010 was 1·6%—with the 1991–2000 annual reduction at 0·5% increasing to 2·6% from 2001–10 (appendix p 9)." That appendix consists of a single figure with a best-fit line for the period 1990–2010 and does not make fully clear the basis for the 2001–10 calculation. The figure does not include points for three of the most recent Demographic and Health Surveys (DHS; Rwanda, 2010, Senegal, 2010, Uganda, 2011). The most recent DHS numbers show that the average nationwide decline across the nine countries in question is 6·4% per year (table).2Demombynes G Trommlerová SK What has driven the decline of infant mortality in Kenya? Policy Research Working Paper 6057. World Bank, Washington, DC2012Crossref Google ScholarTableChild mortality (5-year rate) per 1000 livebirthsBeforeAfterAnnual % declineGhana111 (2003)80 (2008)6·3%Kenya115 (2003)74 (2008)8·4%Malawi133 (2004)112 (2010)2·8%Mali229 (2001)191 (2006)3·6%Nigeria201 (2003)157 (2008)4·8%Rwanda103 (2007)76 (2010)9·6%Senegal121 (2005)72 (2010)9·9%Tanzania112 (2004)81 (2010)5·3%Uganda128 (2006)90 (2011)6·8%Average....6·4%Data are from Demographic and Health Surveys with a survey after 2005. "Before" indicates the second-to-most-recent full DHS survey currently available; "after" indicates the most recent full DHS survey currently available. Open table in a new tab Data are from Demographic and Health Surveys with a survey after 2005. "Before" indicates the second-to-most-recent full DHS survey currently available; "after" indicates the most recent full DHS survey currently available. These numbers overlap more closely with the project period (post-2006) than the points shown in the figure in Pronyk and colleagues' appendix. These numbers are nationwide changes rather than rural-only changes, but since rural-only declines in these countries are typically as fast or faster than nationwide declines (appendix), the nationwide average decline above is reasonably considered a lower bound on the corresponding rural-only decline. The above observations imply that a key finding of the paper—that child mortality fell at the treatment sites at triple the nationwide rural background rate—is incorrect. Child mortality fell at 5·9% per year at the sites versus 6·4% per year on average across all areas of the countries in question (probably more in rural areas alone) according to the available data that most closely match the project period. This difference is not significant. We have additional concerns about the validity of the comparison villages used elsewhere in the paper, and the representativeness of the treatment villages in question. Those concerns, and directly relevant quantitative analysis, appear in the literature3Clemens MA Demombynes G When does rigorous impact evaluation make a difference? The case of the Millennium Villages.J Dev Effectiveness. 2011; 3: 305-339Crossref Scopus (30) Google Scholar but go unmentioned by Pronyk and colleagues. LH's institution, but not LH, is a partner in the impact evaluation of the Millennium Villages Project in Ghana, for the UK Department for International Development. The other authors declare that they have no conflicts of interest. Download .pdf (.1 MB) Help with pdf files Supplementary appendix