Are Academics really that bad at achieving/measuring Impact? Summary of last week’s punch-up

Last week’s post about academics struggling to design their research for impact certainly got a reaction. Maybe not a twitter storm, but twitter stormat least a bit of a squall. So it’s time to summarize the debate and reflect a bit.

The post annoyed some people in the ‘research for impact’ community, because it was basically saying nothing much has changed. ‘The world has moved on’ said Pauline Rose of Cambridge University, ‘It is no longer academics versus NGOs/practitioners but very much working together, identifying commonalities and comparative advantages, and collectively wanting to ensure rigorous research that will achieve change; and in doing so, recognising the nuances of what impact is and how it can be achieved – including the need to start before the research even begins’. IDS’ James Georgalakis was more pithy: ‘Time to change the record’. Both reminded us that there is a lot going on and (‘fessing up here) I’m not across a lot of it. For example, the Rethinking Research Partnerships network and the Impact Initiative (for whom I’ve even written a piece on academic-NGO collaboration).

But their protestations reminded me of a recent conversation with an Oxfam evaluation guru who says one of the problems she faces is that in the aid business, 90% of the attention and conversation surrounds 10% of the work, often the best, most innovative bits. The other 90% is much more same old, same old. The trouble is that those of us looking at and/or contributing to the 10% conclude, like Pauline, that everything has moved on, when a lot of it hasn’t. It was the shock of being exposed to some bog standard stuff that led to the post – a small, doubtless unrepresentative sample, but perhaps more representative than the view from inside the 10% bubble. For the record, I get the same sense of frustration as James and Pauline when I hear people slagging off NGOs for being unreconstructed fools, knaves or often both, and it’s probably at least partly for similar 90/10 reasons (although it may just be that the work is rubbish).

are we making an impactSeveral commenters sounded warning bells about language. According to ODI’s Josephine Tsui ‘Whether you use the term advocacy, influence, or engagement, different groups will have an adverse reaction depending on their political situation. However the political process is the same, you have something to say and you need to target who you’re talking to.’

So much for the blog. The twitter traffic was both more pro (lots of people recognizing the depressing portrait painted in the post) and more deeply critical.

Chicago University’s Chris Blattman weighed in with perhaps the deepest critique. In a series of tweets, he said:

‘This is the wrong way to think about research impact. It’s hardly this direct. The best research changes the intellectual conversation. It changes how textbooks are written. It changes what young scholars do next. It changes how undergraduates and MAs learn the discipline. The UK Government obsession with measuring policy impact of research will only help their universities fall behind. Academics will manage what is measured and turn into think tanks rather than make long term investments and take risks. Sad!’

Which reminded me of a painfully brief meeting with DFID’s head of research some years ago: ‘I’ve come to talk about Research for Advocacy’ I declared. ‘That’s an oxymoron’ he replied, ‘you either have research, or you have advocacy, but you can’t do both without contaminating the research’. Someone obviously forgot to tell the REF.

So, should academics deliberately seek to influence policy and beliefs, and try and ‘count what counts’, including the hard to measure stuff, or push back against the whole effort to oblige them to both achieve and measure impact? Just to prove that I haven’t been entirely seduced by academic preferences for nuance and shades of grey, it’s time for a poll

[poll id=”50″]

But in the interests of nuance (ahem…), you can vote for more than one

Subscribe to our Newsletter

You can unsubscribe at any time by clicking the link in the footer of our emails. For information about our privacy practices, please see our .

We use MailChimp as our marketing platform. By subscribing, you acknowledge that your information will be transferred to MailChimp for processing. Learn more about MailChimp's privacy practices here.


5 Responses to “Are Academics really that bad at achieving/measuring Impact? Summary of last week’s punch-up”
  1. Hi Duncan, thanks for following up on the issue – it’s definitely a debate worth continuing. For me (as a former NGO campaigner turned academic comms/impact type) there are a couple of issues worth teasing out:

    1. I don’t think lumping everything under the term ‘impact’ is particularly helpful – and when you do it has a tendency to favour incremental policy change impacts. This should be celebrated where it happens, but shouldn’t be considered the only show in town. As Chris Blattman suggests, academics are in a position to make a much more transformative intervention to influence broader development thinking. Perhaps thinking more explicit about how academics create longer term ‘change’ is a useful term?

    2. Leading on from this, I think it’s worth highlighting in my experience academics operate quite differently when they work on short term donor funded research projects, in comparison to their on-going field of study. When DFID commission a research project, there’s often a clear policy rationale – and often a fairly open door to influence DFID itself on an issue. Resources (money, time, comms/impact staff) are also increasingly factored into the budget – along with the reporting requirements (or self-justifications) that go with it. On-going, personal research agendas often don’t provide the same short term influencing opportunity, and universities are horrendously understaffed/resourced to support academics to get their work ‘out there’. However, this is where the potentially paradigm shifting stuff often comes from.

    I’ve recently written a blog post reflecting in more details on how academics best create change:

    To answer your poll questions, my even more nuanced (I’ve obviously gone native) response would be that academics should use their research to achieve positive development outcomes – which may well be beyond policy change. Of course there are particularly tactics and approaches that academics can learn from NGOs, but we shouldn’t assume a cut and paste, ‘best practice’ approach to policy influencing is sufficient.

    Cheers! Chris

  2. Martin

    Is there a comparison to be made with the role of academics in the environment and conservation worlds, including in associated advocacy? In my experience, it’s much simpler there, perhaps because we are working with ‘hard’ sciences (and hard scientists)?

  3. Hi Duncan,
    Just catching up with this debate – thanks for the shout out to RRP 😉
    I’m slightly ambivalent in my response. Beyond the Development Studies community there has been quite significant work around not just improving impact but also challenging the nature of impact and promoting collaborative, ‘engaged’ and ‘useful’ research practices (e.g. RURU’s work on the intricacies of knowledge-to-action and the NCCPE’s initiatives on promoting institutional change in universities and research funders as well as supporting individual academics) In the Arts and Humanities sector, the massive and highly influential Connected Communities programme (focused on community-development and cultural heritage in the UK) has contributed to quite a radical shift in research funding to involve community-practitioners in agenda-setting/design/implementation of research and the recent book edited by Keri Facer and Kate Pahl ‘Valuing Interdisciplinary and Collaborative Research: Beyond Impact’ is a great read.
    I do agree, however, that the mainstream approaches to impact (as in the REF) are extremely powerful, creating perverse incentives and need to be addressed. I also agree that the issue of attribution is a large part of the problem. In academia impact is always attributed to an individual and/or project – many of which are short-term, tied opportunistically to the specific funding call and often autonomous so not part of a broader programme. This serves to undermine any genuinely collaborative and sustained, long-term research strategy – which as we know is the most effective route into impact.
    So in response to your poll, for me its not simply a matter of academia or NGOs learning from the other about ‘how to do impact’ but more about grappling with more fundamental cultures/practices within academia (and probably NGOs too – that’s a separate issue) that frame how impact is conceptualised and operationalised – e.g. the individual academic-as-expert/brand/cash-cow and the short-term project-based approach to research funding.

  4. Kate

    I think NGO experience in promoting impact on research is pretty varied. Oxfam obviously has lots of experience in disseminating research/promoting impact. Some other NGOs I’ve spoken to (esp. at country office level or Southern NGOs) have said either that we often don’t manage dissemination because we lack resources (seeing good dissemination as a workshop), or we need to work with academics for dissemination because a journal article is more powerful (esp. in medical/health sphere and where advocacy targets are international organisations). Some NGOs in the same context were working closely with policy makers throughout research to support impact – but so were some academics. It’s a very mixed picture, depending partly on skills/experience/attitude of people leading the work.

  5. Jes

    Hi Duncan,

    Thanks for this!

    I think you have been a bit too diplomatic on your voting options 😉

    Seriously, yes, everybody can and should learn from each other. BUT: When asking the question how to make sure that significant research funding is targeted to support change, giving the “both can learn from each other” option is a too easy way out for all of us. Looking at the research work on sustainability, I still see a lot of research that understands interaction with decision-takers from civil society and Governments as publishing a policy brief. In my experience the “make it or brake it” point of attempting transformational research is letting non-scientific stakeholders influence what is being researched and how it is researched. While the language has changed towards that, the practice really has not.

    While Chris Blattmann has a point that research can achieve what he depicts, poor guidance and limited career prospects in science often lead to super narrow research questions that add to very specific academic debates that are often out of touch with what matters for supporting change. So, I am very much with your 90/10 analysis.

    Being late to the debate, there is little I can add apart from encouraging you, Duncan, to please keep this topic up. There is a lot to be gained with development work and science being strongly interlinked and we are only starting to geht there – acknowledging, though, that UK development research is often more advanced in this regard than what is happening in most of continental Europe.