What is the evidence for evidence-based policy making? Pretty thin, actually.

conference report sets outevidence its theory of change in a handy diagram – the major conference sessions are indicated in boxes. evidence conference ToC Conclusion? ‘There is a shortage of evidence on policy makers’ actual capacity to use research evidence and there is even less evidence on effective strategies to build policy makers’ capacity. Furthermore, many presentations highlighted the insidious effect of corruption on use of evidence in policy making processes.’ i.e. you can have all the arguments you like on the nature of evidence – disciplinary and political bias, what constitutes knowledge etc etc (as this blog recently did), but policy makers are often either unable or unwilling to use it anyway – supply doesn’t guarantee demand. The aid agencies and research councils that fund research are very keen to promote this shift from worrying about supply to wondering how to boost demand (although the researchers are often less keen – they just want to be left alone to churn out papers and develop their careers). What was nice about this conference was the amount of on-the-ground grassroots research on how decision makers actually use (or more often ignore) research in places like Nigeria (‘political manipulation and ambition seem to be among the strongest determinants of factors influencing policy development processes’) and Indonesia (‘Even if technocratic or political – it doesn’t matter – it’s 90% personality’). One thing I learned is that agonising over per diems is not confined to the aid business: ‘One particularly heated debate concerned the frequent requests from policy makers for ‘sitting fees’ in order to attend training or seminars which could inform them about research issues. Participants agreed that this practice is widespread in most of the African countries represented; however, opinions on how to respond to this differed. Some suggested that those who aim to inform policy makers about research need to just accept that paying these fees is necessary and should therefore include them in their budgets. However others felt that continuing to pay such fees just propagates the problem and that those funding research communication and uptake work should take a ‘zero-tolerance’ approach.’ On the demand side, the report considers both capacity and incentives. On capacity ‘most people don’t know what they don’t know!’ will resonate with researchers in NGOs trying to convince their colleagues to look harder at the evidence. There’s a mountain to climb: a survey of Zambian parliamentary researchers and librarians (and these had positively agreed that they needed to use research) found that ‘only one in three believed there was consensus that the CIA did not invent HIV’. ‘Research-evidence is often used opportunistically to back up pre-existing political decisions/opinions (confirmation bias)’. That preference for policy-based evidence-making is alive and well in the big aid donors and NGOs too, of course…….. And unfortunately, research from Ghana, Sierra Leone, Uganda, and Zambia concluded that ‘a lack of capacity to understand research was perceived as beneficial to policy makers since it ‘allowed’ them to ignore evidence and instead follow their own agenda. Thus, there is not only a lack of capacity but also a disincentive to build capacity.’ Oh dear. How to build the incentives to use research, assuming these political obstacles are not insuperable? On HIV policy in Pakistan, DFID ‘built the capacity of civil society organisations representing marginalised groups to demand policy change’. evidence based change placardOther useful tips:

  • including policy makers in the design phase of research projects (get them on the advisory board, guys, don’t just see them as seminar fodder once you’ve finished the research)
  • networks and linkages between researchers and policy makers are necessary but definitely not sufficient
  • researchers need to change the (often dire) way they communicate their work – in one case study from Ghana ‘photographs of real people suffering from mental illness is far more powerful in influencing opinions than any policy brief could be.’ (Well duh)
  • target the ‘policy entrepreneurs’ with influence over decision-makers (the Minister’s old university professor etc)
  • ‘There is a tendency for researchers and research intermediaries to focus their communication efforts on elected representatives and appointed officials but to ignore the crucial role that technocratic staff play.’
All good stuff, but the report reminded me of the governance debates of a few years ago, in that even though it recognized the problem is incentives and politics, kept drifting back to the comfort zone of supply issues (if they don’t want research, we just have to get better at communicating or building their capacity), rather than thinking harder about the demand side. For example:
  • Anyone involved in advocacy knows that the openness of policy makers to new ideas is episodic, and linked to things like changes of administration, scandals, crises and failures. So how does research need to be redesigned to capitalise on such brief windows of opportunity?
  • Opposition parties are often much less well resourced, and much more malleable in their thinking as they cast around for clever ideas that will help them win power – to what extent should researchers concentrate on those without power, rather than those currently in office?
  • Young minds are (generally) more open to new ideas than old ones: should researchers target future leaders (who are pretty easy to identify by faculty and university) rather than waste their time on the current generation?
The evidence debate, you won’t be surprised to hear, continues……]]>

Subscribe to our Newsletter

You can unsubscribe at any time by clicking the link in the footer of our emails. For information about our privacy practices, please see our .

We use MailChimp as our marketing platform. By subscribing, you acknowledge that your information will be transferred to MailChimp for processing. Learn more about MailChimp's privacy practices here.


15 Responses to “What is the evidence for evidence-based policy making? Pretty thin, actually.”
  1. This sounds like a ‘researchers gathering’. As policy making and practice are politically driven processes, more efforts are required to see where is the space, opportunity and possibility. Sometime I feel that production and supply dominates research. At some point, I am happy to share how we are thinking about the nexus of policy, research and practice.

  2. Andrew Dorward

    I was asked recently to come up with examples of research that had influenced agricultural development policy. I found it hard to come up with many specific examples – but did find some (there were problems in deciding which policy makers, what soft of policies (eg what to do? how to do it?), what is research (reviews, policy briefs, seminal studies?)). However it made me realise that a cumulative effect and the development of ‘conventional wisdom’ may often be more important than particular single research pieces.
    It also made me think about research that (in my view at least) should not have been influential. One then gets a two by two matrix (sorry!) with influence Yes/No on one dimension and validity Yes/No on the other and four cells:
    1. Research that should have been influential and was (the jackpot!)
    2. Research that should have been influential and was NOT (failed influence?)
    3. Research that should NOT have been influential and was (failed screening?)
    4. Bad research that no one paid any attention to (failed resarch but …..?).
    The different cells raise lots of demand and supply questions.

  3. Ellie Chowns

    Hmm… You ask ‘How to build the incentives to use research?’ after reporting the heated debates on ‘sitting allowances’. Perhaps we should think more laterally: provide financial incentives for policymakers that can show they have acted on research (and what the results have been), rather than for those that have simply attended a presentation? Cash-on-delivery for research-into-policy?

  4. The photo book on mental health example is from this Mwananchi project in Ghana, funded from the DFID Governance and Transparency Fund which aimed to influence the Mental Health Bill. It is a good example of taking a hidden, local issue to the national level: http://www.mwananchi-africa.org/ghana/2012/6/12/basineeds-uses-photobook-to-influence-policy-making-on-menta.html
    I actually visited this project towards the end of last year and developed a story of change, starting from when they started, cataloguing all the major events associated with the bill, and dynamics around key actors at different times and what they did until when the Bill was passed in parliament. What we learnt from this process was that the photo-book research-based evidence was a critical piece in the dynamic that contributed to the “tipping point” towards actual passing of the bill with other contextual factors being equally important for identifying what works under what circumstances. They are part of the story, and these stories of change often involve other actors that are usually not acknowledged in project reports and impact evaluations.
    We have also learnt from implementing the Governance and Transparency Fund project across six African countries over the last five years that another important distinction is about how you engage with policy-makers at different levels of government. We’ve found evidence can be very effective at pushing local officials and policy-makers into taking action in low accountability contexts. This blog talks a bit more about getting the right combination of evidence and relationships: http://www.mwananchi-africa.org/news/2012/11/12/accountability-is-based-on-relationships-but-data-helps-too.html
    I’m currently working on a major synthesis report which will pull together lessons from supporting accountability, with research-based evidence – hope this will be a contribution to the debate from a long term observation of grassroots politics and change.

  5. Ian

    Really interesting stuff. I’m not sure that aid workers (other than researchers) make as much use of research as we hope/expect either. Here’s something I wrote a while ago on why.
    I think the issue of incentives is key. Why should policy makers make use of our knowledge – what’s in it for them, or how do we make it worthwhile? I think we also need to recognize that policy makers do make use of other forms of “knowledge” in making their decisions such as practical experience and political realities – it might well be that an approach recommended by theory or research findings is not politically or technically feasible but the policy maker not the researcher is the one who knows this.
    Getting researchers and policy makers to talk to each other and work together more is an important step here.

  6. kieran

    1Politicians are often driven by factors other than reseach based evidence.
    2 Most research cannot help but be based on some key Value based assumptions.
    Do we all pack up and go home or just accept that we are operating in an unequal and unfair world of values and ideas. Perhaps the best we can is try and be upfront about our own values and bias and then seek to persuade and influence policy knowing that those in power will accept or reject for reasons that may well not get you a pass rate on an MA in development studies.

  7. Muthoni Muriu

    The moral of the story is that influencing national policy makers with objective, honest policy research and analysis is truly labour intensive. To Tembo’s point, you need to engage different policy makers, on different aspects of the same policy, sometimes in different geographies, to create the sort of critical mass that will drive conversation and hopefully decisions in the desired direction. One or two ‘validation’ workshops or conference won’t do it.
    Our experience with the now defunct (but still relevant) ‘Make Trade Fair’ campaign, was that we needed to speak with technocrats in the Minstries of Agriculture, Trade, Planning and Foreign affairs, relevant embassy trade advisers (and ambassadors) in Brussels and Geneva; trusted policy institutions; random academics working for CIDA/SIDa/DIFD etc who had connections with said ministries; equally random World Bank/IMF/EU commission folks in-country; friendly journalists etc etc….to get the Minister of Trade to take a position on one policy recommendation!
    If the diagram shown above were in 3D, we would be able to show the other layers that must be represented if advocates are to truly understand how policy makers decide (or don’t) on a position. Our big ‘AHA’ moment was working with legislative committees in parliament – they are great at opening doors and you can shamelessly play to egos by packaging policy recommendations in sound bites –
    ‘stop stealing our gold’?
    So how do you boost demand? One corridor conversation at a time …with the increasingly educated technocrats now found in the corridors of developing country governments. Advocates should take heart, it’s not all uphill anymore!

  8. Enrique Mendizabal

    Thanks Duncan. Very useful synthesis. I particularly liked your first quote about not knowing if there is a deficit in the use of evidence. Or maybe that is how I read it.
    When I joined ODI in 2004 I remember thinking that if we were saying that ‘not enough policy was based on evidence’ and hence the need for us to help researchers communicate it better we should at least know how much was enough. I am an economist at heart, you see.
    I asked this question a few times more but never really too loudly as it sort of challenged the industry we were creating around this very simple idea that policy need MORE evidence.
    Ironically we have developed an entire industry promoting the use of evidence based on a hunch… Anyway.
    Anyway, I am not quite sure that policymakers do not use evidence. I think the problem is that it is often not ‘our’ evidence. And in a world in which donors keep asking researchers to demonstrate their impact this is simply unacceptable. How dare they? Not use OUR evidence?
    Well…. Maybe there is such a thing as a hierarchy of evidence, or the perception of credibility, or, well, power.
    (3 posts on how there is no gap between research and policy:
    The theory of change diagram.. Oh how simple it all looks… reminds me of something else. The use of the word policymakers is very telling of our industry/sector. Years ago (I recalled this annecdote in a book on think tanks and political parties in Latin America, by the way, published by International Idea) I heard a discussion between an Argentinian and an Indian panelist at one of these bridging research and policy events: the Argentinean keep talking of politicians and the Indian kept correcting him saying ‘ I think you mean policymakers’. The Argentinean thought about it for a moment and replied: ‘well, it’s all the same’.
    You see, in Spanish the word for politics and policies is the same. And for most of the developing world the idea that a policymakers is a neutral civil servant (a technocrat) is simply preposterous. Laughable. Still, we insist with this ‘politics matters’ narrative but then develop a theory of change that does not even have the decency to mention politicians and politics… ‘Incentives’ being a technical code word for needing to hire a consultant to help with some framework to put ‘audiences’ in a table or something like that. Political economy analysis, they call it now…
    I insist: Politics is politics and one has to live its consequences to really get it. Until then we are never really going to grasp how politics (and all that other stuff) affect the way that people make choices (public or private).
    The solution, as I said in a comment to the evidence based policy debate, is not to train researchers on how to communicate (although this is useful) but to make a serious commitment to invest in education and the capacity to make informed choices: including to dismiss evidence that does not fit our ideologies. Just like we do in the rich world…
    It will take time, it will mean lots of ‘intermediaries’ and consultants will lose their jobs, but it will deliver societies capable of making their own mistakes, learn from them and get on with it.
    …. Not surprising then that you conclude that: Well, duh… I usually do too.

  9. Hi Duncan, thank you for this nice synthesis and the stimulating questions. It made me think about the interview I did with Prof. Leonor Briones, Professor Emeritus in Public Administration at the University of the Philippines, a couple of years ago. When I asked here how can academics like her influence policy makers, her reply was that the best way it to train policy makers once they are elected, for example, as MPs. The main universities here in the Philippines, University of the Philippines or Ateneo, have all set up executive courses for the new (and often younger) MPs. These are MPs that may have had little experience in politics and policy making and the courses prepare them for their new role. She said that when the courses started few years ago, the new MPs were reluctant to participate or did not want the media to know as if participating in the courses would be an admission of ignorance. With time, however, these course have become almost fashionable with new MPs eagerly advertising where they got their certificate (and certificates are really important here int he Philippines). Prof. Briones concluded the interview saying that there are also other ways for researchers and academics to achieve influence. To be elected is one. She has Presidential Adviser for Social Development with Cabinet Rank at the Office of the President and National Treasurer of the Philippines. However, she said that the courses are very important is creating lasting networks and linkages that are sustainable and very valuable for bridging research with policy and politics.

  10. Thanks for the review of the paper Duncan. When the EIPM programme at INASP decided to organise this conference it was because we noticed a lot of the conversation around evidence informed policy was really about how my evidence can inform policy i.e. policy influence of a particular piece of research. Any failure to have research uptake was blamed on poor research communication. We also saw capacity building programmes designed without an understanding of exactly what it was that needed building. We deliberately aimed to hear from researchers and practitioners working in the global south about barriers to the use of evidence beyond anecdote. The conference has certainly informed our approaches in the programme and we are keen to see more academic and practice based research on the challenges on the demand side. I think you say it well when you compare it to the governance debate and the risk of ‘…drifting back to the comfort zone of supply issues (if they don’t want research, we just have to get better at communicating or building their capacity), rather than thinking harder about the demand side’