Guest post from Derek Thorne
Back in 2015, Duncan Green published a piece on FP2P asking whether a TripAdvisor-style feedback system could work in development. If you follow the link, you’ll see it generated a lot of feedback!
The idea was – and is – that TripAdvisor, and systems like it, have put significant power in the hands of consumers, allowing them to publicly post feedback on hotels and restaurants without waiting to be asked, thus creating pressure for services to improve. Could a tool or approach like this achieve something similar in development and social change work?
Six years on, I’d like to have a go at answering that question. That’s because my organisation, Integrity Action, has been working with partners for some years now to set up feedback systems that have some things in common with the TripAdvisor model.
Let’s be clear about one thing first though: transferring the TripAdvisor model directly into development or humanitarian programming wouldn’t work. And why on earth should it? It is a tool built for a commercial context that doesn’t have a great deal in common with the range of contexts in which development work is carried out. But it is interesting to think about the “essence” of TripAdvisor – the principles that underpin it. Here I will pull out two of them and look at how they have been applied, both in our work and that of other organisations.
Open feedback – but open for whom?
The concept of openly published feedback seems attractive. While submitting feedback in private may feel like tossing it into a black hole, posting it in public means it can’t be hidden and can’t be ignored.
Not so fast. With TripAdvisor, hotels and restaurants exist in an open market, and consumers – who have easy access to that platform – can choose where to take their custom based on the ratings they see. That provides a strong incentive for the “supply side” (hotels, restaurants etc.) to act on the feedback of the “demand side”.
In the development and social change context, where citizens rarely get to choose between different project or service providers, the incentives need to work differently. The obvious route is via donors, who could think twice about funding an organisation if they see poor feedback. It might work – but (1) I’m not convinced donors would consistently care about this, or see it as reliable, (2) uniformly positive feedback is itself problematic, as we shall see, and (3) isn’t it rather disempowering for citizens to rely on donors to exert the pressure that’s needed?
However, it turns out the incentives can work more directly than this. It’s just a matter of making sure open feedback is visible to the right people.
Let’s take a pleasingly offline example. As part of Integrity Action’s SHINE project, secondary school students in Nepal monitored their own schools, “reviewing” things like teacher attendance and sanitation. Instead of using a mobile app or digital tool to report what they found (which, in any case, they wouldn’t be allowed to use in school), they used a simple poster (see pic). It appears as a grid, with a traffic-light rating for each of 12 service delivery areas recorded on a monthly basis.
By being displayed on the wall at school, and shared with local officials, this feedback was made open to all the right people. We saw it being used by student “Integrity Clubs” to figure out what issues to bring to regular meetings with school management; it also allows parents, local officials, or other interested parties to know what’s happening and potentially apply pressure.
There are also some digital tools making open feedback work in the development space. Kujakuja displays simple aggregated feedback from community surveys. Loop is a new tool, capturing feedback from multiple channels (including SMS) and applying safeguards before displaying comments online. mySociety has a set of open source tools, including FixMyStreet, which has been applied in Uruguay, Uganda and elsewhere. And Integrity Action has developed a tool called DevelopmentCheck, which citizens have used to monitor over $1 billion worth of development spending. However in all cases, there is an acknowledgement that simply sharing feedback online isn’t going to produce the right incentives in places with low internet penetration, so other means may have to be used to disseminate it.
Can the principle of open feedback be applied more generally in how organisations implement development projects? This was one of the questions asked within a recent “feedback pilot”, in which Integrity Action worked with four UK Aid Direct grantees to enhance their projects’ feedback approaches. We learned some useful lessons from this, including the importance of thinking through who is going to use the data and how, and that it’s easier to make quantitative feedback open, rather than qualitative comments. Read more in our learning paper.
Independent feedback – but what about power dynamics?
Another important feature of TripAdvisor is that – unlike a development agency’s feedback system – it is independent of the services being reviewed. This is meant to make it more trusted. It’s also meant to add “strength” to the feedback, because no matter how bad people’s reviews are, the service in question can’t just throw away the feedback or turn off the system altogether.
Many of the tools and approaches mentioned above take this approach – including the poster. However, at least from Integrity Action’s standpoint, the primary reason is not to make the feedback more trusted by people who would look at it. It’s to build trust with the people who give feedback in the first place. That’s because development has a problem which TripAdvisor doesn’t: a steep difference in power between those providing the services and those making use of them.
This gives rise to what is sometimes referred to as “courtesy bias”, where people are more likely to give positive feedback because they worry that negative feedback may offend the service provider and might even result in the service being withdrawn. It prevents serious concerns and safeguarding issues being raised too.
What do we want? Negative feedback!
This raises what might be the most important difference between TripAdvisor and any equivalent approach in development. Above all else, hotels and restaurants on TripAdvisor want five stars and raving comments. That’s what helps elevate their business on that platform.
Development projects, on the other hand, don’t want that – or they shouldn’t. They need to know where they are going wrong, how things could be improved, how people’s conditions are changing, and much else besides. Feedback is primarily for learning, not for recommendation.
This explains why an independent feedback platform and open publishing go well together. If the service provider is gathering feedback and publishing it openly, it provides a disincentive to look for the negative feedback that will really add value to the project and drive improvements. An independent platform doesn’t have this issue – though it could face external pressure to massage the results it displays, and “fake feedback” could become a problem too. (Don’t forget Campbell’s Law: the more important an indicator is, “the more subject it will be to corruption pressures”.)
This tension, between sharing feedback and encouraging negative feedback, can be addressed in another way: by using feedback metrics that don’t exactly measure quality, but rather responsiveness. A few years ago Integrity Action started using a “Fix Rate” metric, which tracks problems with a project (as identified by citizens) and solutions to those problems. The Fix Rate is the percentage of problems that have been solved to the satisfaction of the citizens concerned. So, to get a high Fix Rate, it pays to get feedback that identifies problems – and then to solve them.
A final point: I accept it’s pretty Northern-centric to take a tool that is primarily used in high-income contexts, and ask “well, how can we apply it in low-income contexts? Surely this will provide benefits!” What I’ve tried to show here is that it’s not as simple as that – but that it does provide an interesting prompt to explore some interesting features of feedback. But there are now so many interesting feedback and accountability approaches originating in the South that it might even be time to ask: what can other parts of the world learn from an Eyes and Ears, a HealthTools, or a Checkmyschool? Now there’s an article I would love to read.