Monitoring and evaluation

Monitoring is a continuing process in which data on specified indicators is systematically collected in order to provide insights into the progress of certain developments. In the M&E process, this monitoring is generally used to understand how the funds are being applied and to obtain indications of the progress of the projects, e.g. how many stakeholders have been reached [1]. Evaluation is a process in which the relevance and degree of success visible in this information is assessed [1]. Monitoring and evaluation are synergistic: monitoring provides the quantitative and structured context, evaluation provides a balanced interpretation [2]. Remote-sensing in itself is a useful tool to monitor the impact of certain programmes aimed at agricultural development [3], but is not used for this purpose in G4AW. M&E is an important component of the G4AW programme: projects provide progress information annually as part of their reporting requirements, which is subsequently interpreted by the G4AW project advisor and M&E staff. At G4AW level, continuous progress is monitored in the Akvo Really Simple Reporting (RSR) platform [4] and published publicly in the IATI format. M&E is carried out at project and programme level. The G4AW M&E framework focuses on four components: 'output', 'use of service', 'outcome' and 'impact' [5]. A strong focus is placed in digital inclusion (reaching women and youth).

Reaching impact
A challenge within all digital solutions, such as G4AW services, is monitoring and evaluation (M&E) in relation to real on-the-ground impact for indicators for which the innovation program was set up. A distinction is made between high-level outcome indicators at one side, which are of a more abstract nature and cannot be measured directly. An example of such an outcome indicator is increased food security. This indicator is not measured directly, but rather is analysed through the aggregation of multiple more concrete ‘output’ indicators, for instance measured increases in land productivity. Specifically for the G4AW programme, these two categories are supplemented with a third category: use-of-service. This separate category allows further insight in the channels through which the G4AW services contribute to increased food security. The interrelationship between these categories of indicators, from concrete to more abstract, can be schematized in a so-called ‘logic framework’ or ‘Theory of Change’. These also outline how the G4AW activities ultimately contribute to the policy objectives of the Netherlands Ministry of Foreign Affairs in themes as food and nutrition security and economic development.

Business-oriented approach and BI
G4AW projects have a business-oriented approach. This implies that more business-related indicators such as customer satisfaction and business analytics. This provides additional information in the uptake of services and potential bottlenecks, and thus provides insights in how services can be improved further. This can result in improved product-market fits, which is expected to benefit the financial sustainability of the services post-project phase. Within G4AW, the more ODA-related M&E approach and business analytics are to a high degree two sides of the same coin.

Lessons learned in operationalizing the M&E framework
Monitoring & Evaluation frameworks require operational guidelines for the design, implementation and harmonization of monitoring and evaluation systems [7]. A particular challenge related to the M&E of digital agricultural advisory services is that the nature of the G4AW services and related delivery methods significantly differ between projects. How can you compare the impact of a nationally broadcast weather forecast to plot-level agronomic advice by app when it comes to improving food security? The nature of the services and the related multiplier effects [8] are too different. Design of a system that can adequately monitor and evaluate these different services in a way that the outputs can be compared is not straightforward. One-size-fits-all approaches do not do justice to the degree of diversity between the G4AW projects. In practice, the G4AW project advisor and M&E staff will discuss with the project consortia and agree to a semi-tailormade approach. At one side, the consistency of similar indicators across the projects is safeguarded, facilitating the aggregation at G4AW programme level. At the same time, projects can emphasize those indicators from a pre-determined set that fit best with their service and intervention logic. G4AW is actively analyzing experiences in the application of the M&E framework, and works to translate lessons learned in stepwise improvements in M&E methods, while ensuring interannual comparability of M&E data.

Policy objectives
Besides the monitoring of the individual projects under the G4AW programme, it is also important for policy-makers in the project countries to evaluate the impact of the created digital technologies holistically. There is always a public sector organization involved in the project country, and ideally they will assist in evaluating the position and potential of the services in the national framework. This requires a “whole of government” approach [9], in order to find the most suitable digital pathway to ensure maximum impact. This should consider the overall status of the information and communication technologies the capacity to build and invest in infrastructures and human resources that would foster technology innovations and transformation of the agriculture sector [9].

Recommended readings:

1. UNDP programming manual on monitoring, reporting and evaluation
2. WORLD BANK GROUP: What is Monitoring and Evaluation?
3. PWC report: Challenges and solutions in monitoring & evaluating international development cooperation
4. AKVO Really Simple Reporting
5. M&E framework G4AW
6. G4AW Project Reporting call 2017-2018
7. FAO report: Operational guidelines for the design, implementation and harmonization of monitoring and evaluation systems for climate-smart agriculture
8. RVO report: Digital Farming in Kenya
9. Bytes to Sustain Our Bites: Leveraging Digital Agriculture for Sustainable Development Goals (SDGs). Thirty-fifth Session of the FAO Conference for Asia and the Pacific
10. GSMA: Adapting monitoring and evaluation tools to design impactful digital services