‘It’s all about results’
International cooperation is complex and subject to constant change. That’s why, for every single project, GIZ has to examine the context and decide which approaches are likely to prove effective and which not. Evaluations provide the essential basis for this process. They measure and assess, and enable us to learn lessons that we can apply in the future. Below you can read a double interview given by Martha Gutiérrez (GIZ) and Jörg Faust (DEval), who describe exactly how evaluation works and what it can contribute to the current debate on international cooperation.
International cooperation has recently come in for criticism. Why are evaluations more important than ever in times like these?
Martha Gutiérrez (MG) The public have a justified interest in knowing where their tax money goes. Evaluations fulfil our duty of accountability to the public. They show us what international cooperation is achieving, what works and what doesn’t. Whatever functions well needs to be expanded and replicated. But sometimes the evaluations tell us that our attempts at resolving certain problems in our partner countries are on the wrong track. Then we may need to change course on the ground. We ensure that the reports based on those evaluations are transparent and publicly available – even if projects get a poor rating. In addition, the reports are translated into other languages to fulfil our duty of accountability to partner governments.
Jörg Faust (JF) In Germany, there is probably no other policy area that undergoes such a structured and systematic form of scrutiny as international cooperation. The traditional importance of evaluation stems from the fact that taxpayers’ money is often spent in challenging contexts.
MG One third of the money spent by GIZ on behalf of the German Federal Ministry for Economic Cooperation and Development (BMZ) is independently evaluated against international standards. We look at an average of 80 projects around the world every year. In addition, there are assessments of individual sectors or countries, plus specific analyses of corporate policy issues and evaluations for other commissioning parties. The number of evaluations puts us in second place worldwide, behind the Korea International Cooperation Agency. This means that we check thoroughly whether we achieve the greatest possible impact with the funds we actually spend.
DEval’s Opinion Monitor regularly researches the attitudes of ordinary people to development policy. Mr Faust, what is the most important result from the latest survey? And what does that mean in terms of evaluation?
JF Our Opinion Monitor reflects the critical views expressed in the current debate. Public support for a consistent or increasing development budget has fallen by more than 20 percentage points between 2022 an 2024. One major explanatory factor is that people rate their own financial position, and that of the state, as worse. Against this background, independent evaluation can improve the effectiveness and credibility of this policy area. The information it provides must show a degree of self-awareness by including errors as well as successes and by emphasising institutionalised learning.
‘Evaluation at GIZ takes a threefold approach: recording results, communicating results and learning from results. We have defined it as a core process because the three aspects form a single unit.’
Despite this, the system is repeatedly accused of being an enclosed bubble in which closely linked units assess each other …
MG In actual practice, the picture is quite different. The evaluations are prepared by independent consultants. Every three years we organise a Europe-wide tender procedure, resulting in a pool of around 100 international evaluators. They work together in teams and also cooperate with local evaluators. Furthermore, our unit is independent; it reports directly to the Management Board and is not integrated into GIZ’s operational business. So we only coordinate the evaluations, we don’t influence them. All of these factors ensure that we end up with methodologically sound and independent assessments.
JF Evaluations in Germany follow the recognised OECD criteria of sustainability, relevance, coherence, effectiveness, impact and efficiency. The Bundesrechnungshof (Germany’s supreme audit institution) reviewed the evaluation system for international cooperation in 2021; there was no criticism regarding independence. The field where evaluations can improve is communication, which is frequently too technical and sometimes contains too much jargon – and that applies to the entire policy area.
Alongside accountability, learning from evaluations is another important consideration. Could you give some examples to illustrate how that works?
MG There are lots of examples. We’ve presented a selection in this report. Take GIZ’s support for the Egyptian-German Committee for the Promotion of Renewable Energy, Energy Efficiency and Environmental Protection. One of the evaluation findings concerned the potential for improving the committee’s own knowledge management system to make it less reliant on external support. In response, with help from GIZ, the agency made some changes. It introduced train-the-trainer programmes, wrote manuals and improved the documentation of processes and knowledge. As a result, it became more effective. That’s a good example of the threefold approach to evaluation at GIZ: recording results, communicating results and learning from results. We have defined it as a core process because the three aspects form a single unit.
Two organisations, one objective: evaluations at GIZ and at DEval
GIZ’s Evaluation Unit examines the company’s own programmes and projects, mainly those commissioned by BMZ. But GIZ also evaluates projects and programmes commissioned by other organisations, such as the German Federal Foreign Office. In performing this role, the unit is not integrated into GIZ’s operational business, and projects are assessed by independent evaluators. This creates the distance necessary for constructive criticism. By contrast, DEval’s role is to evaluate German development cooperation. It has a mandate from the German Federal Government to carry out independent evaluation work. Moreover, it regularly canvasses public opinion on foreign and security policy. DEval and GIZ maintain regular contact so that they benefit and learn from each other. Their activities complement each other.
Mr Faust, how important is learning through evaluation?
JF Evaluation has three functions: generating findings with practical relevance, fulfilling accountability obligations and providing opportunities to learn. Often the learning takes place during the evaluation and not just at the end. That’s usually the case when the people being evaluated see the whole process more as an opportunity and less as a risk. The oversight function of evaluations can also act as a driver for learning.
Ms Gutiérrez, how does that work at GIZ? When are projects evaluated?
MG We take a sample containing roughly 40 per cent of our BMZ commissions from among projects which are scheduled for completion within the next 18 months. We choose this period so that we can assess whether anything has been achieved and, if so, what that is. Until 2017, projects steered their evaluations themselves. Then GIZ changed that and centralised the evaluations to make them more independent, as we were saying earlier. Today, the projects themselves work with good monitoring systems to see whether their indicators are being achieved or not. If they notice that something is going wrong, they generally make adjustments. This means that projects implemented by GIZ achieve good ratings overall. Only a small proportion are assessed as ‘unsuccessful’ – despite the fact that in 2022 we added a further quality benchmark. Now, if a project scores 4, 5 or 6 on any one of the OECD criteria of effectiveness, impact or sustainability, it is deemed to be ‘unsuccessful’ overall. That’s why we call these three criteria ‘knock-out’ or ‘KO’ criteria.
Projects in the energy sector achieve especially good ratings in comparison with other sectors. Why is that?
MG That’s correct. The average score in this area was 1.97, whereas the average for all evaluated projects in the current reporting period was 2.50. One of the main reasons for this is that we’re reacting directly to a high level of need in our partner countries and their economies. They need energy to continue their development, and ideally that should be climate- and resource-friendly. In some cases, these forms of energy are now cheaper, and the approach also makes them less dependent on energy imports. Our support can be effective here. Relevance, effectiveness and sustainability score relatively well in energy projects because partner countries and companies invest in and maintain the projects. That’s one of the reasons for the good rating.
‘No other policy area undergoes such a structured and systematic form of scrutiny as international cooperation.’
Given these findings, shouldn’t there be much more investment in climate and energy?
JF These topics will remain very important aspects of international cooperation in the foreseeable future. When it comes to access to energy in rural Africa, there are not only successes but also challenges, for instance in relation to access by disadvantaged groups via decentralised power grids, or our recommendation to provide additional funding for climate-friendly forms of energy for cooking that do not pose a risk to human health.
MG In my view, the results show above all that international cooperation is having an impact, precisely in the energy sector. This benefits our partner countries, but ultimately our own economy as well. Every second euro is earned in Germany through exports.
JF Ms Gutiérrez is referring to the important question of foreign trade. Since the 2010s, a number of studies have shown that international cooperation boosts exports from donor countries. Economists from the University of Göttingen reconfirmed this link very recently.
MG It’s important to make the point about the value of international development to the German and European economies because they also benefit from greater action on climate and environmental protection worldwide. But this is not solely about economic cooperation. We’re interested in results on the ground. Better conditions for the local economy and for local people are key aspects of international cooperation. GIZ wishes to contribute to a future that is worth living around the globe. Our evaluations tell us that we really are contributing to that goal, but in every case they also tell us where there is room for improvement. We never stop learning.
