Impact is the ultimate outcome for any social change initiative yet it’s also the most elusive and poorly defined of our evaluation metrics in practice. How do we change this? In this first post in a series, I bring us into the world of understanding impact.
Non-profits are in the business of making a positive difference in spaces that are ignored or neglected by other institutions society. Whether it’s a lack of governmental support, business investment, or shared infrastructure, non-profits work to fill those gaps through channeling social enterprise and innovation, charity, community mobilization and action, and volunteerism. There are many models of non-profit work, but every one of them seeks to generate positive impact.
They do this through intervention — which is a way of saying messing with the systems that influence people’s lives. By messing, I’m referring to the act of reconfiguring connections, relationships, and structures between things in a system. This also means doing new things, for the sake of brevity, I’ll refer to all of those seeking to create change in this space as social innovators, using the broadest use of that term.
Non-profit organization seek to intervene in systems by disrupting the harmful effects of policies, practices, and environmental conditions and amplifying positive actions (knowledge, skills, motivation, contributions, and designs) to produce benefits for society. Most of the time, these harms and benefits are centred on particular populations, situations, and settings. To illustrate, a community health centre might seek to disrupt the effects of a lack of access to healthcare and amplify health promotion activities within a community. Or a educational outreach program for young people might seek to disrupt the effects of poverty by introducing and amplifying strategies for learning that transcend traditional school programming.
But what does it mean to have impact?
Defining Impact
Impact is described as:
“Positive and negative, primary and secondary long-term effects produced by a development intervention, directly or indirectly, intended or unintended.”
(OECD-DAC, 2010)
As part of the OECD’s efforts to measure impact, they add the following note to clarify their definition:
Note: Impact addresses the ultimate significance and potentially transformative effects of the intervention. It seeks to identify the social, environmental and economic effects of the intervention that are longer term or broader in scope than those already captured under the effectiveness criterion. Beyond the immediate results, this criterion seeks to capture the indirect, secondary and potential consequences of the intervention. It does so by examining the holistic and enduring changes in systems or norms, and potential effects on people’s wellbeing, human rights, gender equality, and the environment.
OECD DAC Network on Development Evaluation (EvalNet)
John Gargani and Robert McLean have spent a lot of time thinking about what it means to create impact and measure it successfully. Their book, Scaling Impact, recounts their experiences working in international development evaluation and the challenges they faced in capturing the true effects of what programs were doing and how to take positive innovations and scale them. They describe impact this way:
Impact is one or more consequences, intended or unintended, of an action or actions. Innovators strive to create impacts that are meaningful to people, but not all impacts are, and people may find different meanings in them. Some impacts may be considered detrimental.
McLean, R., Gargani, J., & Chambers, R. (2019). Scaling impact: innovation for the public good. Routledge.
What’s important to note about this description is that McLean and Gargani intentionally and explicitly note that impact is both 1) context and audience dependent, and 2) not always positive. Thus, when organizations and change-makers seek impact, what they are really saying is that they want to create positive benefits with few harms.
Beyond Outcomes to System Influence
What distinguishes impact from other aspects of evaluation of social innovation is that it, by the definitions above, are systems-oriented. We might seek to generate new things, which are outputs. We might also design products, services, programs and policies to achieve certain things, which are outcomes. Impact reflects whether these outcomes achieve something beyond the direct effects.
To use the examples above, impact of a community health centre’s work might be to reduce the overall burden of illness and disease in a population over time. For that education program, the impact might be achieved through a cadre of successful graduates who contribute to their community through having well-paying, fulfilling jobs and career pathways that match or exceed those of students who didn’t experience poverty growing up – breaking cycles perpetuated by a lack of educational achievement.
Impact is like ripples in a pond, where the intervention is the stone that hits the water and the effects are what is touched by the small waves it creates. How wide, far, and to what scale those ripples go is what our evaluations seek to capture. We also need a sensitivity to our evaluations to detect our intervention’s influence in ways that reflect unexpected effects, not just those we can anticipate.
To understand and assess impact requires means and measures of change that are sensitive to the conditions within systems and able to connect the actions with the effects that take place within them. This means understanding how systems work, creating viable theories of change that reflect those systems, and establishing evaluation strategies that work with both.
What those are and how they operate is what I’ll explore in the next post in this series.
Image credits: Photo by Jordan McDonald on Unsplash