Guest post by Chris Barnett, Director of Technical Excellence, Itad
Back in the mid-2000s, one project stood out as a bold attempt to achieve the Millennium Development Goals (MDGs) at a local level: The Millennium Villages Project (MVP). Spearheaded by Professor Jeffrey Sachs, the project sought to demonstrate how the MDGs could be achieved through an integrated approach to rural development. For the past six years, I’ve been managing an evaluation of the MVP in northern Ghana, funded by DFID, to provide rigorous evidence about whether this approach to development really works.
Intuitively, the MVP made sense in many ways: do everything at once (health, education, agriculture, etc) so the effect will be far greater than a piecemeal approach of single, small-scale interventions. Not only that, but we do actually know what works in lots of areas: applying artificial fertiliser to maize rapidly increases yields, bed nets protect people from malaria-carrying mosquitos; vaccinations prevent disease.
As the former UN Secretary-General, Ban Ki-Moon was quoted as saying said back in 2010: “Millennium Villages throughout Africa are showcasing… how effective an integrated strategy for health care, education, agriculture, and small business can be. We are seeing how to make the most of new technologies. And we are seeing how empowering women can empower whole societies…”.
So, lots of optimism… but, it’s been controversial.
Controversy seemed to peak around 2011 to 2013, which just happened to be when we started out in northern Ghana. Putting aside the more heated ‘war of words’, Michael Clemens and others were writing papers and blogging on the need for rigorous evidence (such as this Guardian blog, or putting the case forward that ‘rigorous impact evidence is not a luxury…’).
So, here we are – more than five years later – bringing together evidence of the impact of the Millennium Villages Project (for northern Ghana at least):
Did the project have an impact? Well, yes and no.
No, in terms of achieving the MDGs at a local level (see MVP Evaluation Briefing 1). Certainly, for the key indicators for poverty (MDG 1) there does not seem to have been a reduction. There were income improvements that could be attributed to the Millennium Villages project, but there was no accompanying increase in consumption, and the improvements in income were insufficient to break the poverty trap. The population remains poor.
But yes, there were impacts. We found statistically significant impact against 7 of the 28 MDG indicators. For example, primary school attendance improved by 7.7%, intermediate health indicators improved for births attended by skilled professionals, children sleeping under bed nets, and so on.
Of course, assessing the project just against the MDGs is somewhat ridiculous. Rarely can a project achieve such high-level goals in a short period (although that is what the project set out to do), and the MDGs themselves are far from a definitive list of indicators. Nor indeed do they reflect the realities of people living in the area. So, there is lots of exploratory analysis that we cover elsewhere…
Does the project offer value for money? This is THE controversial one.
Caution needs to be applied when interpreting the analysis, as no direct comparison with similar projects was possible (see MVP Evaluation Briefing 3). Our analysis shows that the returns to investment in education appear to be highest, although both the returns in education and health could be achieved at a much lower cost.
Given that this was always intended as a demonstration project, we also undertook a sensitivity analysis to see if transferring such a project to local ownership could improve the value for money. Even with a 50% reduction in overheads however, there is questionable cost-effectiveness.
So, what can we conclude?
Certainly, achieving lasting change is difficult. The project’s over-ambitious aims (and often claims) made it the target for much disagreement. There were some good ideas, and things that worked well (for example, the tractor hire services, vaccinations, investments in health centres and school infrastructure). But it was starting from a low base of investment – as anyone who’s travelled to the very northern areas of Ghana knows – and it takes time.
Finally, a few things stand out for me about the Millennium Villages approach in northern Ghana:
Firstly, although many ‘quick wins’ had been scientifically tested (with rigorous evidence), the project seemed to underestimate the need for considerable experimentation in each context – working with the grain of local government and communities, as well as working with how people respond differently to interventions and adapt solutions. For example, people were not always using bed nets, finding hard at certain times of the year to reconcile the use of bed nets with the night heat. (As an aside, it is also surprising how many creative uses for bed nets there were, whether as screens for toilets or bathing areas, to protect young fruit trees, to cover store rooms, as pillows or bedding to lie on, as fishing nets, door curtains and sun blinds).
And secondly, sustainability is a huge challenge (see MVP Evaluation Briefing 9): Approaches like top-up allowances for education and health workers, new ‘super’ health facilities, and free ambulance services, etc., all had a measure of success, but were beyond what could be sustained locally. Spending by the project tended to focus on building infrastructure, providing supplies and extra staffing (‘things’) with insufficient attention to the behaviour change needed to create long-lasting impact. For example, the statistical analysis shows a sharp increase in the access to toilets (part of MDG 7), with toilets rapidly built during the last two years of the project – yet, the qualitative research highlights how people still preferred to go ‘free range’, under a ‘nice baobab tree’ or among rocks away from the village.
I am sure that for many readers of ‘From Poverty to Power’ this evidence only serves to reconfirm their perspective of development – the need to understand power, work with complexity, disrupt the system. But, I leave you with one challenge: would your chosen project or policy have fared any better if subject to robust evaluation? And, how do we know?