As we design and deliver climate change adaptation projects with our partner organisations in Nepal, we have to monitor how successful our work is. Working out what to measure is not as simple as it seems. And in figuring it out, we must always keep in mind that what we choose to measure has an significant impact on what the project does.
Success is judged in different ways, by different people. TGT makes its own judgements; but given that we also receive funding from other grant giving bodies, their expectations about success need to be taken into account too. The degree to which funding bodies decide what is important and what success looks like effects project design, delivery and ultimately outcomes.
Our approach has always been to keep our thoughts on what matters in the background and we are fortunate to work with funders who largely feel the same way. What matters to us is that the projects are having a positive impact on what matters to the people we work with in the villages in Nepal. In short, we keep our egos in check and allow project design to be led bottom-up from the people to us via our local partner NGOs in Nepal.
Not all small NGOs are as fortunate, many are effectively forced to design projects that have easily measurable performance indicators that tick the boxes of large foundations and grant giving bodies. However, as adaptation comes ever more sharply into focus, this problem may become more acute for us and others as funders and delivery organisations are scrutinised more widely for their adaptation work.
With more attention, comes more anxiety as we all want to be able to prove that we are being successful; a salve to this would be the emergence of a universal way to judge whether an adaptation project has been a success. That way we can do what is globally accepted as the ‘right thing’ on adaptation and report on our successes (in comparrison to others). Partly driven by this and partly driven by a habit of doing things the way we’ve always done things, a quest is underway to establish universal ways to judge adaptation success. It might be a grave error.
Lisa Dilling, University of Colorado, and colleagues from around the globe (including Farid Ahmad from ICIMOD, Nepal) recently posed an important question: ‘Is adaptation success a flawed concept?’ They explored this important question in a two page comment article in Nature Climate Change (Vol 9, 570-574). According to the authors, a universal way of measuring adaptation success remains elusive. Seeking it at all might be unwise.
The desire to find universally applicable measures is also driven by the dominant role the UNFCCC (UN Climate Change) continues to play and the habits we have got into in our technocratic, managerial times. Understandably, much emphasis to date at the UNFCCC has been on driving and measuring mitigation efforts; it is only since the realization that we can’t now prevent dangerous climate change and the recognition of this in the Paris Agreement in 2015, that adaptation has truly come into focus. But what works for mitigation measurement, may not work for adaptation.
Member parties have grown used to comparing themselves against each other on mitigation measures. They, for example, can measure the rate at which they are reducing carbon emissions and report on how well they are doing compared to other countries of comparable size and economic fortune (something the UK Government has become very fond of doing in recent years). This works when data is relatively easy to collect and report, which it is for emissions reduction efforts. There is an appetite to be able to do the same for adaptation over the coming decades so that politicians can boast of the progress they have made on adaptation compared to other countries (or opposition parties), in same way as they currently do for mitigation and scores of other issues - education, health, crime, social care, etc, etc.
Trying to apply this approach to climate change adaptation - attempting to define and measure universally applicable measures of adaptation - is proving difficult; it is a square pegs, round holes scenario. As Dilling et. al., point out ‘most adaptation projects are implemented at the local level and start from wildly differing baseline conditions.’ Furthermore, there is great variation in the nature of adaptations needed, their scale and the time it takes for adaptation measures to take effect.
Seeking universal success measures for adaptation to fit the political paradigm of tangible, measurable and reportable outcomes, isn’t easy because adaptation is such a difficult thing to quantify and compare across different contexts. And yet we try, which leads to us having to simplify (likely oversimplify), what success looks like. We therefore risk falling into the age old trap of designing projects that can be measured, rather than projects that are truly effective.
If we are to stick with this pursuit of finding universal measures of adaptation success (and it is questionable whether we should) an alternative approach is needed. Dilling and her colleagues propose one in their comment piece. They suggest we measure the ‘capabilities of households and governments to pursue a range of adaptation futures’. This fits with an argument that says it is more important to measure the adaptive capacity of a person (or community, or nation) than it is to measure whether they have adapted to the climate change effects they are currently experiencing. More on this later.
To highlight what is meant by ‘capabilities’ it is worth quote Dilling and her colleagues at length:
Capabilities critical to effectively respond to climate change may include access to increased education, which is necessary to build diversified livelihoods; access to healthcare, necessary to respond to new health risks; access to technology, increasingly necessary for communication and information access; strengthened social support, necessary to prepare for and respond to shocks; and good governance, necessary to ensure services are delivered. Capabilities can directly support adaptation: for example, according to Striessnig et al., female education may be the single most important socio-economic factor associated with reduced vulnerability to disasters. Some processes and international agreements have already outlined metrics of success that may support the building of these capabilities (for example the UN Sustainable Development Goals, the Sendai Framework and the Convention on Biological Diversity), which in turn lay the foundation needed to support adaptation efforts as climate change unfolds.
We have met many farmers who, thanks to the income they have been able to generate from the sale of agricultural produce, have been able to better support the education of their children and grandchildren. These children are thus developing the capabilities they need to adapt effectively to climate change as it unfolds in whichever context they find themselves in the future, be that on the family farm, or in any other environment they might find themselves living in in the future.
So, whilst TGT’s project might be about enabling farmers to grow crops that are more resilient to the changes being felt as a result of climate breakdown, the outcomes farmers are interested in and most value relate to impact income from tomato sales is having on their children’s education.
When we were writing the script to our short film ‘Coffee. Climate. Community.’ we included a line, to emphasize how that as well as the direct environmental and ecological benefits of agro-forestry for Deusa in Solukhumbu, there is a direct benefit that relates to this idea of developing ‘capabilities’:
NARRATOR: “There is an obvious benefit too. Increased wealth, even when it is modest, makes other things possible. The income generated [from selling coffee] can be invested in other climate change adaptation projects like water harvesting, insect control, polytunnels and landslide prevention.”
Dilling et.al., sum up their piece with a conclusion and recommendation for what we should be focusing our efforts on when it comes to project evaluation:
Efforts of the scientific and international communities would be better spent understanding how to build, support and measure capabilities of communities, and how those capabilities in turn enable adaptive capacity for climate change, rather than trying to develop a universal definition of adaptation success.
For TGT, in seeking to measure how successful our projects are, we must try not to get too hung up on counting beans. Adaptation is an ongoing process in the same way that climate change is ongoing. We are never going to run a project where at the end of it we can say to ourselves and our funders: ‘right; that village has now adapted to climate change’; but what we can hope to do is say ‘we have enabled the people in that village to develop capabilities that improve their adaptive capacity.’
Importantly, Dilling et. al., emphasize the agency that should be given to households and governments, so that they play a central role in determining the capabilities that are to be developed by an initiative. Not only is this important from a moral and power balance perspective, it is also important to ensuring projecs actually work. By empowering local people to create metrics that matter to them, local aspirations are built into project design, which should then have a positive effect on how engaged and invested people are in the initiative they are involved in.
To do this effectively, it is up to us, our NGO partners and ultimately our funders, to feel comfortable inviting people to define which ‘capabilities’ they want to enhance. Once we understand this, we can grasp better what the project needs to do. By introducing agro-forestry to a remote mountain village, we not only make agriculture in that village more resilient to the climate change impacts already being felt, we also enable farmers to raise income that makes them more capable of dealing with climate change long term. We must keep an eye on both these things, measuring the success of the agro-forestry work in the here and now, but also keep a close eye on measurements relating to the ‘capabilities’ that matter in the community.
Locally then, in any context, we are players in a broader effort to improve access to education, infrastructure, healthcare, social welfare and so on. These are the metrics that matter in the end, because without basic services people won’t have the foundations they need to continue to adapt to climate change as it evolves and worsens.