Challenges in Evaluating Systemic Change

As the process of initiating a systemic change needs a new approach, so too does the evaluation of its success. While we can learn from traditional best practices within the sector, there are too many pitfalls in simply foisting the whole process on a systemic change scenario.

As the contextual nature of each attempt at systemic change means the approach to measuring it will be different there are some key challenges to be aware of in designing a measurement process.

1.       The definition of change. All things will be linked to this so it is crucial it is backed up by a thorough understanding of the system, paths to change, levers and potential barriers, and external changes.

2.       The appetite of funders. Systemic change takes time and therefore the impact of it will not be seen for years following the initial investments. How to balance the need for impact metrics for funders and stakeholders and the ongoing nature of the work, will lead us to develop a graduated system, providing shorter-term outcomes, based on work completed and indications of change, alongside longer-time larger changes that will alter the system being targeted. Funders need to be on board with the timescales and the associated costs of their results.

3.       Piloting an evaluation system. Systemic change is about layering blocks as opposed to the linear running of a programme, in the majority of cases, removing the opportunity to pilot an evaluation system.

4.       Collecting a baseline. When the intended impact is a system wide change, deciding on and collecting baseline data can be difficult. How to collect or access the relevant data will depend on the context, as well as the capacity and resource of the organisation, so in many cases using a system map as a baseline may be a more accessible approach to this.

5.       Attribution versus contribution. With any social change programme defining attribution against contribution is difficult, however the role of any one actor within a defined system change is nigh on impossible to pinpoint.

6.       Embedding evaluation. All evaluation, but particularly systemic evaluation should see measurement as part of the process not a separate, parallel component. Constant learning, adaptation and iteration based on evidence presents the best chance of generating systemic impact.

7.       Need for common metrics across funders portfolios. As traditional impact evaluation systems become more sophisticated and systematised, the creation of common metrics across portfolios is becoming more and more common. This create challenges for systemic change as it is based on very individualised contexts, and the systems to measure it must reflect this.

8.       Reliability of long-term data collection. As systemic change can be years in the making, often outlasting the original funded activity, ensuring the reliability of data collection over years, with multiple different people being involved, can be difficult.

None of these challenges and tensions are insurmountable but they need to be addressed at the outset with clear communication and contingency plans. A framework needs to be able to incorporate the nuances within systemic change while also being appropriate for the capacity of the organisation, resource available and is accessible to all the stakeholders.