How do you Measure the Impact of Influencing (and ask some v useful questions en route)?

MEL (monitoring, evaluation and learning) is a lot more interesting than it sounds. Done badly, it can amount to little more than bean-counting to satisfy the donor, of little value to the actual programme or people who are supposed to benefit. But done well, it raises all sorts of really important questions about how the programme/project is designed, early enough to make a difference, then feeds back what is learned (I absolutely refuse to say ‘the learnings’) to improve later iterations of the work. More ME+L then ME+L.

I thought about this when I saw the great new online guide to the MEL of Influencing, from Oxfam. It’s a lot harder to measure the impact of a campaign or a behind-closed-doors bit of advocacy, than more traditional aid activities like vaccines or handing out seeds and tools, which made it even more enticing.

Let’s look at the ‘influencing outcomes’ section, because that’s what interests me most. It starts off with a nice graphic showing potential impacts on people’s lives, from tangible to intangible.

It then moves on to test your grasp of some pretty arcane language (e.g. the difference between outputs and outcomes), and makes some very sensible remarks about why you should be doing all this:

The design is funky (although it’s 100% online – I do wish I could just print it out and read it – sigh). Drop down menus allow you to choose between following it through as a course (v linear, tsk), looking for particular ‘tools’ (e.g. depending on whether you want help with planning, monitoring, evaluating or learning) or want to ‘select a journey’ from ‘building a theory of change’ to ‘influencing outcomes’.

‘Outcomes are usually very important for an MEL System, because they provide early information on whether a project or programme is on course or whether any desired changes are beginning to happen.

If we waited to measure the ultimate impact of the project or programme without bothering to look at the outcomes, it might be to late to realize that we were not in the right track. Any MEL system or process designed to feed into management decision-making needs to assess outcomes on a regular basis.

Avoid M&E system that purely look at what is being delivered on the assumption that if products or services are delivered properly, they will automatically translate into change.

PRIORITIZE OUTCOMES THAT…

You directly influence (rather than indirectly support)

Are important to your goal & mission

Are not too costly to measure

Will produce credible data’

It then moves on to the different variables that can be used to classify or categorize outcomes:

Depending on the Type of Outcome (WHAT is changing?)

Depending on the Actor-stakeholder (WHO is changing?) – some examples for campaigns to change policies

Depending on the Level of the desired change (TO WHAT EXTENT is anything changing?), eg this example on a digital campaign

And that is just section 2.2 – Here’s the full contents list

1 INTRODUCTION

1.1 INFLUENCING

1.2 MEL AT OXFAM

1.3 MEL OF INFLUENCING

2 WHAT IS IT YOU WILL MEASURE?

2.1 THEORY OF CHANGE

2.2 OUTPUTS, OUTCOMES AND IMPACT

2.3 INDICATORS

3 HOW WILL YOU MEASURE?

3.1 SETTING UP A MEL PLAN

3.2 MONITORING; WHEN AND HOW

3.3 EVALUATION: WHEN AND HOW

3.4 LEARNING: WHEN AND HOW

3.5 ROLES AND RESPONSIBILITIES

4 TOOLS AND METHODS

5.A ETHICS OF EVALUATION

5.B GUIDANCE AND TEMPLATES FOR COMISSIONING EXTERNAL EVALUATION 

5.C SAFEGUARDING AND MEL

5.D ORIENTATIONS TO DO MONITORING AND EVALUATION OF INFLUENCING PROGRAMS WHERE SCALING UP OR INDIRECT REACH IS RELEVANT

All in all, a real treasure trove for anyone trying to design an influencing exercise, whether an insider/lobby exercise, or a more public protest or media campaign. And as you’d expect from Oxfam, a lot of attention to tackling gender inequalities throughout.

Excellent stuff.

5 ANNEXES

Subscribe to our Newsletter

You can unsubscribe at any time by clicking the link in the footer of our emails. For information about our privacy practices, please see our .

We use MailChimp as our marketing platform. By subscribing, you acknowledge that your information will be transferred to MailChimp for processing. Learn more about MailChimp's privacy practices here.

Comments

16 Responses to “How do you Measure the Impact of Influencing (and ask some v useful questions en route)?”
  1. When it comes to measurement we typically have 3 options:
    1. measure the thing directly (test everyone to determine the rates of covid in the community);
    2. measure the consequences of the thing (test for the amount of covid in waste water); or
    3. ask stakeholders what they think (what do health workers think about rates of covid in the community).

    Option number 1 is the most accurate, time consuming and expensive while number 3 is quicker, cheaper and the least accurate.

  2. This looks interesting. I wonder if it addresses the challenge that a policy win tends to be a one-off, so often the results of factors beyond our control. For one thing we know little about the daily slog of policy makers (see https://evaluationandcommunicationinpractice.net/how-much-do-we-know-about-policymakers-daily-slog/ ). We see policy impact as something to aim for, yet seeking to measure success remains elusive. We see it mainly as an art, where timing, luck, readiness and a lot of practical wisdom are needed (see: https://evaluationandcommunicationinpractice.net/evaluating-policy-impact-an-arena-for-practical-wisdom/ ). Looking forward to having access to the publication.

  3. Thanks for this Duncan, a really useful resource. I haven’t looked through in detail yet (I’m a slow reader!), but I wonder to what extent the approach and tools are similar to WaterAid’s ‘common approach to PMER of advocacy’ and our recommended tools to go with that (PMER = Planning, Monitoring, Evaluation & Reporting). https://washmatters.wateraid.org/publications/wateraids-common-approach-to-pmer-of-advocacy
    I will have a read and see what I think, but welcome others views too (obv Oxfam have gone for a swankier digital design thingy!).
    We’re also in the process of putting together an ‘Advocacy Handbook’ at WaterAid – so not just on MEL, but on advocacy more broadly. We’ve got an ancient 2007 ‘Advocacy sourcebook’ that is just begging for a refresh. https://www.joinforwater.ngo/sites/default/files/library_assets/W_PRA_E4_advocacy_sourcebook.pdf
    Do others have great examples of ‘handbooks’ that they have found most useful and practical? Do share links if you can.
    Cheers,
    Dan (WaterAid Global Head of Campaigns)

Leave a Reply

Your email address will not be published. Required fields are marked *