How do you measure the difficult stuff (empowerment, resilience) and whether any change is attributable to your role?

In one of his grumpier moments, Owen Barder recently branded me as ‘anti-data’, which (if you think about it for a minute) would be a bit weird forGandhi v logframe cartoonanyone working in the development sector. The real issue is of course, what kind of data tell you useful things about different kinds of programme, and how you collect them. If people equate ‘data’ solely with ‘numbers’, then I think we have a problem.

To try and make sense of this, I’ve been talking to a few of our measurement wallahs, expecially the ones who are working on a bright, shiny new acronym – take a bow HTMB: Hard to Measure Benefits. That’s women’s empowerment, policy influence or resilience rather than bed nets or vaccinations. It fits nicely with the discussion on complex systems (Owen’s pronouncement came after we helped launch Ben Ramalingam’s book Aid on the Edge of Chaos).

A few random thoughts, (largely recycled from Claire Hutchings and Jennie Richmond):

Necessary v Sufficient: there was an exchange recently during the debates around the Open Government summit. IDS researcher Duncan Edwards pointed out that it simply cannot be proven that to “provide access to data/information –> some magic occurs –> we see positive change.”  To which a transparency activist replied ‘it’s necessary, even if not sufficient’.

But the aid business if full of ‘necessary-but-not-sufficient’ factors – good laws and policies (if they aren’t implemented); information and data; corporate codes of conduct; aid itself. And that changes the nature of results and measurement. If what you are doing is necessary but not sufficient, then there is no point waiting for the end result and working backwards, because nothing observable has happened yet (unless you’re happy to wait for several decades, and then try and prove attribution).

Example: funding progressive movements that are taking on a repressive government. Can you only assess impact after/if the revolution happens? No, you have to find ways of measuring progress, which are likely to revolve around systematically gathering other people’s opinions of what you have been doing.

Even if the revolution does come, you need to establish whether any of it can be attributed to your contributions – how much of the Arab Spring was down to your little grant or workshop?! Process tracing is interesting here – you not only investigate whether the evidence supports your hypothesis that your intervention contributed, but also consider alternative narratives about other possible causes. You then make a judgement call about the relative plausibility of different versions of events.

Counterfactuals when N=1:  Counterfactuals are wonderful things – find a comparable sample where the intervention has not happened, and if your selection is sufficiently rigorous, any difference between the two samples should be down to your intervention. But what about when N=1? We recently had a campaign on Coca Cola which was rapidly followed by the company shifting its policy on land grabs. What’s the counterfactual there – Pepsi?! Far more convincing to talk to Coke management, or industry watchers, and get their opinion on the reasons for the policy change, and our role within it.

All of these often-derided ‘qual’ methods can of course be used rigorously, or sloppily (just like quantitative ones).

Learning v accountabilityAccountability v Learning: What are we doing all this measurement for? In practice M&E (monitoring and evaluation) often take precedence over L (Learning), because the main driver of the whole exercise is accountability to donors and/or political paymasters. That means ‘proving’ that aid works takes precedence over learning how to get better at it. Nowhere is this clearer than in attitudes to failure – a goldmine in terms of learning, but a nightmare in terms of accountability. Or in timelines – if you want to learn, then do your monitoring from the start, so that you can adapt and improve. If you want to convince donors, just wait til the end.

People also feel that M&E frameworks are rigid and unchangeable once they’ve been agreed with donors, and they are often being reported against by people who were not involved in the original design conversations. That can mean that the measures don’t mean that much to them – they are simply reporting requirements, rather than potentially useful pieces of information.  But when you actually talk to flesh and blood donors, they are increasingly willing to adjust indicators in light of the progress of a piece of work – sorry, no excuse for inaction there.

Horses for Courses: Claire has been playing around with the different categories of  ‘hard to measure’ benefits that Oxfam is grappling with, and reckons most fall into one (or more) of the following three categories

  1. The issue is complex and therefore hard to understand, define and/or quantify (e.g. women’s empowerment – see diagram, resilience). Here we’re developing ‘complex theoretical constructs’ – mash up indices of several different components that try and capture the concept
  2. The intervention is trying to influence a complex system (and is often itself complex) eg advocacy, or governance reform.  For both, we’re investing in real time information to inform short- and medium term analysis and decisions on ‘course corrections’.   In terms of final asessments on effectiveness or impact, policy advcocay is where we think process tracing can be most useful.  The units are too small to estimate the counterfactual and so it’s about playing detective to try and identify the catalysts and enablers of change, and see if your intervention is among them.  We’re still learning how best to approach evaluation of governance programming.  Some will lend themselves to more traditional statistical approaches  (eg the We Can campaign on Violence Against Women), but others have too few units to be classified as ‘large n’ and too many to be considered truly ‘small n’.  We trialled outcome harvesting + process tracing last year in Tanzania.
  3. The intervention is taking place in a complex, shifting system.  This might be more traditional WASH programming, or attempting to influence the complex systems of governance work in fragile states. The evaluation designs may be similar to categories one and two above, but the sustainability of any positive changes will be much less certain, and like 2, we are most interested in realtime learning informing adaptation, which requires a less cumbersome, more agile approach.

Got all that? My head hurts. Time for a cup of tea.

Subscribe to our Newsletter

You can unsubscribe at any time by clicking the link in the footer of our emails. For information about our privacy practices, please see our .

We use MailChimp as our marketing platform. By subscribing, you acknowledge that your information will be transferred to MailChimp for processing. Learn more about MailChimp's privacy practices here.


9 Responses to “How do you measure the difficult stuff (empowerment, resilience) and whether any change is attributable to your role?”
  1. Good stuff Duncan and I am sure most of the M&E practitioners will agree. In a recent international meeting a colleague shared the concerns that how much money is being diverted to measurement and not on things which could benefit the target groups. Please keep raising these issues.

  2. Great read Duncan.

    I’m currently looking at the impact of a social accountability/empowerment programme in Ghana, and after struggling for a while with trying to come up with a meaningful statistical methodology, I gave up and decided to go in the qual direction. Hopefully when it comes to analysis in three months time I won’t be tearing my hair out, but this gives me a little more confidence that I’m going in the right direction!

  3. Thanks Duncan. Excellent as always.

    I’m sorry if I sounded uncharacteristically grumpy. I was pushing back against your characterisation of two opposing tribes: a “measurement & results” tribe and a “complexity” tribe. (If I remember rightly you had us standing on hills shouting at each other, and wondered if we would descend into the valley for hand to hand combat).

    My point was that this is an entirely false dichotomy. For many interventions in complex adaptive systems, the most successful approaches will be based on measurement and data. We cannot predict in detail whether or how interventions will work, but we can (and should) measure effects of what we do, and we should adjust the interventions accordingly. Indeed, my view is that this approach to complexity is not only possible, but the only approach which is likely to be successful.

    As your blog post shows, there is plenty you can do to use measurement, data and numbers in complex systems.

    So should I read this blog post as meaning that you now agree that as we understand more about complexity this should make us more, not less, interested in data and measurement?


  4. Hi Duncan,

    The timing of this blog post is ideal for us here at Publish What You Fund as this is something we want to start looking at in the new year. We’ve spent a lot of time pushing donors to publish to the International Aid Transparency Initiative (IATI) and we want to make sure that the data is useful for both accountability but also learning.

    One of the findings from our 2013 Aid Transparency Index is that only a handful of donors are publishing results data (either in IATI or elsewhere) so we want to find out more about how different groups might use results data; how results information could be published to make it more useful; and how this can help balance out the see saw, i.e. how results can contribute to learning. We want to start discussing this in early 2014 so interested to get your thoughts.

    Best wishes,

  5. Duncan

    Hi Owen, it all depends. As long as the complexity and the value for money tribes are both clear that we need to measure what actually matters (and helps us learn) not just what is easy to count, then there is no need for hand to hand combat. But if the results tribe dismiss qual and insist that anything short of an RCT is just anecdote, then we got a problem.

  6. Jake Allen

    I think its fair to raise the point that in the same way complexitists and measurites are screaming bloody threats at one another across a divide, neither are (or perhaps neither should) hard to measurers amd donors be so at odds.

    In the interests of disclosure, I have HTMB all over my hands and bit on my shirt too, but even if I didn’t the pont would be the same.

    DFID, to take our closest example, knows their systems and approach are a bit clunky. Check out Pete Vowles’ blog for example. Now being a big, P/political organisation, it’s not likely to move fast, but the intention is certainly there, not least in their testing (and measuring! ) of complxity approaches in practice.

    But look, im not a donor apologist. A fair argument coukd be made that this is actually more if a process of getting back to where ee should have been, but hindsight is a lovely thing.

    I think the summary of HTMB and where next is to crack the ‘no progess without evidence/no evidence without progress’ catch 22 by everyone bringing their respective toys to the party and sharing, which usually means donor cash and NGO programmes. The will is there on both sides, so let’s all keep melting that iceberg!

  7. What if you knew that an identifiable and significant subset of your beneficiaries were 10 times more likely to be denied access to education and 5 times more likely to have serious health issues – and your M&E systems were not detecting this?

    This post came on the 3rd Dec – International day of persons with disabilities, a topic which was originally concieved as being part of this particular debate within PPA learning groups… Measuring the extent to which develoment involves or reaches disabled people isn’t as hard as some imagine – the Washington Group on disability statistics has come up with a simple set of questions for national census and 23 countries have started using this. It was very refreshing today to see PLAN and the London School of Hygine and Tropical Medicine release an analysis of their global database of 1.4 million children – identifying that disabled girls and boys were 10 times more likely to not be at school. Yes 10 times that is not a misprint. Food for thought for any mainstream agency involved in education? The chances of disabled children having serious health issues were about 5 times higher. These are shocking – but unsuprising – results. Disabled people are too many in number and the level of disadantage too great for this to be regarded as a minor issue. It is time for everyone to raise their game – and please do so with the involvement of disabled people….just as gender sensitive programming must include women. The post 2015 report said that no development intervention should be considered successful if disabled people were left out….’leave no-body behind’ the rallying call. Was anyone listening?

  8. Colleen Duggan

    Nice, thought-provoking post, as always, Duncan. For me there’s a particular category that sticks out like a sore thumb and I’m not sure where to put it. I’ll call it “complex unmentionables”. Examples might include preventive diplomacy interventions (for obvious reasons,Track Three NGO peace actors and diplomats of any stripe have clear reasons for not wanting to crow about their success); military and civil society organisations who work behind the scenes to convince non-state armed actors to stop recruiting 10 years olds. These results are not only “hard-to-measure”. There are good reasons why they are hard to talk about in either quantitave or qualitative terms.