In this time of crisis, speed has been of the essence to reach vulnerable communities. We previously explored how non-profits are rapidly responding and adapting - from mobilising ground-up efforts to distribute masks and food, to moving programmes online via live Facebook sessions.
In the move to respond quickly, longer term priorities such as evaluation and impact management may have taken a back seat. The suspension of planned programmes has put routine monitoring and evaluation on hold. Some funders have begun to loosen or eliminate restrictions on current grants and measurement indicators as programmes evolve to fit the situation. This move towards flexibility and support for core operations has been pivotal for non-profits to be more responsive to evolving needs.
Instead of abandoning evaluation, however, now is the time to re-imagine it. It is more important than ever to reflect on and learn about what is working and what is not during COVID-19, in order to improve both existing programmes as well as new crisis response programmes. It may also be necessary to demonstrate and measure the effectiveness of these programmes for continued funding support. Evaluative thinking can help non-profits prioritise resources, adjust programmes and evaluate them for accountability and learning.
Developmental evaluation - an approach specifically designed for uncertainty
As non-profits adapt to the ‘new normal’, many are asking questions: How can we effectively respond and meet the immediate needs that are emerging now? Are remote and online programmes (e.g. teleconsultations with social workers) effective and should they be continued post-COVID-19? In the longer term, how can we learn from what we are doing to emerge from this crisis stronger?
Developmental evaluation is an approach specifically designed to answer such questions and support decision-making in uncertain times like these. It focuses on rapid and evidence-based adaptation and encourages innovation in a constantly evolving environment. Some key ideas are:
1. Collect real-time data to adjust strategies as they are being implemented
Collecting data in real-time throughout a programme, instead of retrospectively at the end, enables feedback and iterations for improving programme design and strategies. Gathering data on what's happening, what's emerging, how needs are changing, and what are our future options can guide us through how to effectively adjust our priorities, our resources, and decide how to implement interventions.
For example, many organisations are embracing rapid online and phone surveys to regularly collect data in light of physical distancing restrictions. The World Bank is conducting high frequency monitoring (every 4-6 weeks over a year) of the impact of COVID-19 impact on households via mobile phone surveys. These surveys, which can be deployed rapidly, target over 100 countries across all developing regions. The results provide policymakers with timely and relevant information on the impacts of the crisis as well as the effectiveness of their policy measures to save lives and support livelihoods.
2. 'Good enough’ data may be more useful for decision-making
A hallmark of the pandemic response has been the urgency with which decisions have to be made. In this situation, ‘good enough’ data can be more useful than ‘perfect’ data to support timely decision-making. While methodological rigour is important, now may not be the right time to focus on it - rigorous data collected too late may no longer be relevant. Rather than developing and administering a full survey, obtaining qualitative data though smaller, purposeful samples of interviews with clients could be more impactful.
In Singapore, ballooning COVID-19 cases amongst migrant workers prompted urgent action from all sectors. Massey University in New Zealand and HOME Singapore jointly presented recommendations in a White Paper to address workers’ immediate concerns (e.g. wages, mental health), their living conditions, and longer term structural change. The findings were based on data collected remotely through a rapid qualitative survey conducted as phone interviews with 100 migrant workers. The methodology relied on observational data and descriptive accounts that could be collected quickly from a small snowball sample.
3. Continuously learn and adapt through evaluation
As we have previously discussed, COVID-19 is a golden opportunity for learning. In the absence of a playbook, non-profits have been improvising brand new processes, partnerships, and ways of working. The next step is to learn from this new experience through reflection and internalisation - which can be done with continuous evaluation.
Within non-profits, this could mean setting up the processes and culture for data collection and continuous learning. This doesn’t have to be complicated or time-consuming - a lot can be learned from simple, regular check-ins with staff and clients about their observations and experiences. The key is to institutionalise it. An After Action Review, through which a team regularly gathers to openly and honestly reflect on a piece of work, could be a simple way to process success and failure in order to improve going forward.
Non-profits can also learn from peers in addition to themselves. Non-profits must break out of the habit of working in silos and mindset of competition and collaborate with other non-profits to share information, pool resources, and take joint action. Continuous peer-to-peer learning can be enabled by earmarking time, space and resources to gather and share - from learning circles or exchanges to communities of practice. Some organisations are sharing cloud-hosted documents with colleagues so they can access data analysis in real-time as it is being conducted.
4. Be flexible and plan for change
The saying that “change is the only constant” has never been more true. In order to emerge stronger and more resilient, non-profits will need to continue to call on their ability to adapt as the crisis moves from one phase to another. Programme goals, target populations, outcomes and indicators will continue to be subject to change - even those that have been unaffected up until now. This will demand flexibility in implementation, evaluation design, data collection, and reporting criteria. Non-profits must plan for change, facilitate, document and share it to understand how new ways of working are effective (or not) and which ones are worth keeping post COVID-19.
A good starting point could be to review existing data to see what is still relevant and useful in the current situation. This helps us understand how to better collect and use data to adjust and influence decisions that shape an organisation’s services. The Inspiring Impact worksheet offers a step-by-step guide to know what information you should continue collecting, what to stop collecting, and what to start collecting.
Just Cause is committed to support funders and non-profits to truly learn and improve from the data that they collect. How is your organisation reflecting on what is working during these uncertain times? We would love to hear from you, please get in touch to share your journey.