Tag: theory of change

evaluation

Meaning and metrics for innovation

miguel-a-amutio-426624-unsplash.jpg

Metrics are at the heart of evaluation of impact and value in products and services although they are rarely straightforward. What makes a good metric requires some thinking about what the meaning of a metric is, first. 

I recently read a story on what makes a good metric from Chris Moran, Editor of Strategic Projects at The Guardian. Chris’s work is about building, engaging, and retaining audiences online so he spends a lot of time thinking about metrics and what they mean.

Chris – with support from many others — outlines the five characteristics of a good metric as being:

  1. Relevant
  2. Measurable
  3. Actionable
  4. Reliable
  5. Readable (less likely to be misunderstood)

(What I liked was that he also pointed to additional criteria that didn’t quite make the cut but, as he suggests, could).

This list was developed in the context of communications initiatives, which is exactly the point we need to consider: context matters when it comes to metrics. Context also is holistic, thus we need to consider these five (plus the others?) criteria as a whole if we’re to develop, deploy, and interpret data from these metrics.

As John Hagel puts it: we are moving from the industrial age where standardized metrics and scale dominated to the contextual age.

Sensemaking and metrics

Innovation is entirely context-dependent. A new iPhone might not mean much to someone who has had one but could be transformative to someone who’s never had that computing power in their hand. Home visits by a doctor or healer were once the only way people were treated for sickness (and is still the case in some parts of the world) and now home visits are novel and represent an innovation in many areas of Western healthcare.

Demographic characteristics are one area where sensemaking is critical when it comes to metrics and measures. Sensemaking is a process of literally making sense of something within a specific context. It’s used when there are no standard or obvious means to understand the meaning of something at the outset, rather meaning is made through investigation, reflection, and other data. It is a process that involves asking questions about value — and value is at the core of innovation.

For example, identity questions on race, sexual orientation, gender, and place of origin all require intense sensemaking before, during, and after use. Asking these questions gets us to consider: what value is it to know any of this?

How is a metric useful without an understanding of the value in which it is meant to reflect?

What we’ve seen from population research is that failure to ask these questions has left many at the margins without a voice — their experience isn’t captured in the data used to make policy decisions. We’ve seen the opposite when we do ask these questions — unwisely — such as strange claims made on associations, over-generalizations, and stereotypes formed from data that somehow ‘links’ certain characteristics to behaviours without critical thought: we create policies that exclude because we have data.

The lesson we learn from behavioural science is that, if you have enough data, you can pretty much connect anything to anything. Therefore, we need to be very careful about what we collect data on and what metrics we use.

The role of theory of change and theory of stage

One reason for these strange associations (or absence) is the lack of a theory of change to explain why any of these variables ought to play a role in explaining what happens. A good, proper theory of change provides a rationale for why something should lead to something else and what might come from it all. It is anchored in data, evidence, theory, and design (which ties it together).

Metrics are the means by which we can assess the fit of a theory of change. What often gets missed is that fit is also context-based by time. Some metrics have a better fit at different times during an innovation’s development.

For example, a particular metric might be more useful in later-stage research where there is an established base of knowledge (e.g., when an innovation is mature) versus when we are looking at the early formation of an idea. The proof-of-concept stage (i.e., ‘can this idea work?’) is very different than if something is in the ‘can this scale’? stage. To that end, metrics need to be fit with something akin to a theory of stage. This would help explain how an innovation might develop at the early stage versus later ones.

Metrics are useful. Blindly using metrics — or using the wrong ones — can be harmful in ways that might be unmeasurable without the proper thinking about what they do, what they represent, and which ones to use.

Choose wisely.

Photo by Miguel A. Amutio on Unsplash

behaviour changecomplexityemergenceevaluationsocial systems

International Women’s Day, Every Day

IMG_4693

Today is International Women’s Day when the attention of one half of the world’s population is brought to the entire world, suggesting that maybe this day is best honoured the other 364 days as well. Time to consider how this might look. 

People worldwide will be celebrating and honouring women as part of International Women’s Day (#IWD2107) and it’s hard to conceive of any issue that is more worthy of such recognition. The theme for this year’s day is Be Bold for Change (#BeBoldForChange) with a variety of resources and promotional campaigns set up to raise awareness of women’s issues worldwide; support women and men in advocating for positive, healthy change around sex and gender-based discrimination; and creating a climate of positive human development for everyone, worldwide.

Depending on your perspective, this celebration of women worldwide on International Women’s Day is either something to be cherished or viewed with discouraged puzzlement — and both reflect the enormity of the issues that women face.

Women make up more than half of the globe’s population, are most often charged with raising children, represent the highest percentage of caregivers in most societies, and yet are systematically excluded (at worst) or badly included (at best) from many of the levers of power to enable them to sit on par with men on many issues that matter to women. A simple and depressing Google image search of Fortune 500 CEOs will find a white male wall of images that would almost suggest that a woman’s presence is there by mistake. If this is the starting place, the end is surely worse.

The puzzlement comes not from celebrating women, rather from the fact that we still need a place to do it because it’s not part of the fabric of everyday life for far too many, despite it being 2017 (two years past 2015 as Canadian Prime Minister Justin Trudeau remarked on his gender-balanced cabinet appointments). Just as Black Lives Matter is a necessary statement (and movement) because, for many, the lives of black people are treated as if they don’t matter, we need to celebrate women because they are too often treated as if they are the furthest thing from celebration-worthy.

It’s not elsewhere

A look through many of the various campaigns and promotional material looking at advancing gender equity will find a very visible presence of images and foci on the developing world. While it most certainly the case that women in these regions face considerable gender-based disadvantages the emphasis on the ‘other’ parts of the world can take our attention away from what is happening closer to home. In Canada, the wage gap between men and women has actually increased in recent years, with women earning 72% of what men do.

Not only do women earn less, but they make far less gains than men and this is made even more so if you belong to a racial minority. According to the Institute for Women’s Policy Research in the United States based on the current rate of change, a white women will need to wait until 2059 to achieve pay equity. For Black women, this stretches to the absurd 2124 and for Hispanic women it’s at the incomprehensible 2248. Yes. some women in the United States will need to wait for 230+ years to see their pay equal than of a man at the current rate of change.

And this is just on matters of pay. The issues women face are far deeper and beyond comment in one simple article such as this.

The point is that women are systematic disadvantaged everywhere and the solution space needs to take a systems perspective if there is any hope of making meaningful progress and making the rate of change something better than expecting something different in 10 generations from now. One of the best ways to ensure that women succeed is to engage the other half of the system: men.

Areas of action: the role of men

It has been heartening to see an outpouring of support for #IWD2017 from men, something that is notably different from past years. Too often celebrations of diversity, resistance, or change involve the group most disadvantaged, but not enough from those whose power is challenged, yet whose involvement is necessary for systemic change to happen. Men need to play a big role in the change. This is not a ‘what about me’ kind of statement from men, but a realistic assertion that systems change cannot take place without engagement of the different parts in the system and that means involving both sexes in the change process.

The matter of violence against women is one of the areas where men’s involvement is critical and starting to attract greater engagement from men. One of the attractors used to draw men to this issue is through sport. Breakaway is a soccer (football) themed online game designed for boys (and girls) aged 8-15 to educate and illustrate issues of gender-based violence. Using sport and the things that boys are interested in (like video games, playing with friends) is a clever means of upending the usual approach of simply telling people about the harms associated with gender-based violence and hoping something changes.

In Canada, many of the Canadian Football League teams have programs aimed at their fans to raise awareness of and prevent violence against women. This is providing a more constructive counter to the horrible displays of gender-based violence from football players in the National Football League in the United States in recent years. Games like Breakaway and the integration of sport leaders into the conversation starts to change the dialogue around who commits violence, what the norms are around violence, and provide positive examples for young men to follow in living a life violence-free.

Changing the narrative: A systems perspective

The matter of women’s rights, freedoms, and opportunities is not simply solved due to the conflation of social, economic, geographic, and historical factors that have shaped the institutions and norms that surround sex and gender-based discrimination. That knotting up of issues is the hallmark of a complex system and thus, if we are to make substantive progress for women (and humanity, at large) on these issues the matter is better served by taking a systems approach. A great place to start is recognizing the complexity of the matter.

Attractors are forces that draw in (or repel) energy — attention, information, enthusiasm, focus, commitment, and more — and finding those that will attract both men and women (whether together or apart) to women’s issues is key. The use of sport and games as a means of attracting men is one example. Many men and boys engage in sport for creativity, recreation, social connection, and skill development and channeling those positive qualities toward inclusion of, respect for, and support of women and their rights is one way to scaffold from one issue to the next.

Engagement of thought leaders, opinion leaders and micro-influencers can also be a tool by shifting the norms, content and tenor of the discussion. These individuals are those that are on the pulse of trends, reflecting social aspirations, or simply provide direct means to cut through the clutter of the mediasphere to deliver a message. This is not just about celebrities, but those who are listened to. This amplifies a positive attractor within the system and draws more men (and women) into constructive conversations and actions.

An attractor-based approach to systems change also requires engagement of diversity within that system. This is another reason to consider the micro-influencer: someone who is a big deal in a small(ish) social space. These might be people on Instagram or within a community of practice or a local champion that has a committed, devoted following or engaged audience. These influencers speak to niche populations, issues, contexts and media forms that resonate with small segments of the population, deeply. That deeper engagement is what will propel people to make substantive changes in their behaviour, speak out, and further push change forward rather than a wide, thin engagement strategy. This last point speaks to the role of evaluation in all of this.

Evaluating the revolution

Social change is only thus because something happened that was different than was before. The only way to tell if the present is different than the past is to evaluate (compare) and potentially to attribute what happened to something that was done. But evaluation is more than social accounting, it’s also about gathering and using information to make things better and more impactful as things unfold. We don’t want to wait until 2059 to see if whatever efforts were put in place today will lead some women to pay equity. We might (and hopefully do) want to see things amplified so that this target date is brought closer to us.

The way to do this is to develop an evaluation strategy that clearly describes what is happening, what efforts are being developed and employed to support change, articulate a theory of change, and then create a series of strategic data collection measures (*that might not all be quantitative) that can be deployed at a system level and various smaller levels within the system to monitor and evaluate what kind of change we are producing. This allows us to ensure that whatever positive attractors we have are amplified and reinforced and those that are negative are disposed of or dampened. This can only be done if we have the feedback mechanisms in place and that is what evaluation delivers.

As we recognize the strengths and wonders that women bring to this world every day and the struggles they face, let’s consider how we can build on this energy and create attractors that can last beyond a day, a month or season to being something that is part of the fabric of life every day. That would be truly something to celebrate.

 

 

 

education & learningresearchsystems thinking

The urban legends of learning (and other inconvenient truths)

Learning simulacrum, simulation or something else?

Learning simulacrum, simulation or something else?

Learning styles, technology-driven teaching, and self-direction are all concepts that anyone interested in education should be familiar with, yet the foundations for their adoption into the classroom, lab or boardroom are more suspect than you might think. Today we look at the three urban legends of learning and what that might mean for education, innovation and beyond. 

What kind of learner are you? Are you a visual learner perhaps, where you need information presented in a particular visual style to make sense of it? Maybe you need to problem-solve to learn because that’s the way you’ve been told is best for your education.

Perhaps you are a self-directed learner who is one that, when given the right encouragement and tools, will find your way through the muck to the answers and that others just need to get out of the way. With tools like the web and social media, you have the world’s knowledge at your disposal and have little need to be ‘taught’ that stuff, because its online.

And if you’re a digital native (PDF), this is all second nature to you because you’re able to use multiple technologies simultaneously to solve multiple problems together with ease if given the ability to do so. After all, you’ve had these tools your entire life.

A recent article by Paul Kirschner and Jeroen van Merriënboer published in the peer-reviewed journal Educational Psychologist challenges these ‘truths’ and many more, calling them urban legends:

An urban legend, urban myth, urban tale, or contemporary legend, is a form of modern folklore consisting of stories that may or may not have been believed by their tellers to be true.

The authors are quick to point out that there are differences in the way people approach material and prefer to learn, but they also illustrate that there is relatively little evidence to support much of the thinking that surrounds these practices, confusing learning preferences for learning outcomes. I’ve commented on this before, noting that too often learning is conflated with interest and enjoyment when they are different things and if we were really serious about it we might change the way we do a great deal many things in life.

In the paper, the authors debunk — or at least question — the evidence that supports the ‘legends’ of digital natives as a type of learner, the presence of specific learning styles and the need to customize learning to suit such styles of learning, and that of the lone self-educator. In each case, the authors present much evidence to challenge these ideas so as not to take them as truths, but hypotheses that have little support for them in practice.

Science and its inconvenient truths about learning

Science has a funny way of revealing truths that we may find uncomfortable or at least challenge our current orthodoxy.

This reminds me of a terrific quote from the movie Men in Black that illustrates the fragility of ideas in the presence and absence of evidence after one of the characters (played by Will Smith) uncovers that aliens were living on earth (in the film) and is consoled by his partner (played by Tommy Lee Jones) about what is known and unknown in the world:

Fifteen hundred years ago everybody knew the Earth was the center of the universe. Five hundred years ago, everybody knew the Earth was flat, and fifteen minutes ago, you knew that humans were alone on this planet. Imagine what you’ll know tomorrow.

One of the problems with learning is that there is a lot to learn and not all of it is the same in content, format and situational utility. Knowledge is not a ‘thing’ in the way that potatoes, shoes, patio furniture, orange juice, and pencils are things where you can have more or less of it and measure the increase, decrease and change in it over time. But we often treat it that way. Further, knowledge is also highly contextualized and combines elements that are stable, emergent, and transformative in new, complex arrangements simultaneously over time. It is a complex adaptive system.

Learning (in practice) resists simple truths.

It’s why we can be taught something over and again and not get it, while other things get picked up quickly within the same person even if the two ‘things’ seem alike. The conditions in which a person might learn are cultural (e.g., exposure to teaching styles at school, classroom designs, educational systems, availability and exposure to technology, life experiences, emphasis on reflective living/practice within society, time to reflect etc..) and psycho-social/biological (e.g., attention, intelligence, social proximity, literacy, cognitive capacity for information processing, ability to engage with others) so to reduce this complex phenomena to a series of statements about technology, preference and perception is highly problematic.

Science doesn’t have all the answers — far from it — but at least it can test out what is consistent and observable over time and build on that. In doing so, it exposes the responsibility we have as educators and learners.

With great power comes great responsibility…?

Underpinning the urban legends discussed by Kirschner and van Merriënboer and not discussed is the tendency for these legends to create a hands-off learning systems where workplaces, schools, and social systems are freed from the responsibility of shaping learning experiences and opportunities. It effectively reduces institutional knowledge, wisdom and experience to mere variables in a panoply of info-bites treated as all the same.

It also assumes that design doesn’t matter, which undermines the ability to create spaces and places that optimize learning options for people from diverse circumstances.

This mindset frees organizations from having to give time to learning, provide direction (i.e., do their own homework and set the conditions for effective learning and knowledge integration at the outset). It also frees us up from having to choose, to commit to certain ideas and theories, which means some form of discernment, priority setting, and strategy. That requires work up front and leadership and hard, critical, and time-consuming conversations about what is important, what we value in our work, and what we want to see.

When we assume everyone will just find their way we abdicate that responsibility.

Divesting resources and increasing distraction

In my home country of Canada, governments have been doing this with social investment for years where the federal government divests interest to the provinces who divest it to cities and towns who divest it to the public (and private) sector, which means our taxes never go up even if the demands on services do and we find that individual citizens are responsible for more of the process of generating collective benefit without the advantage of any scaled system to support resource allocation and deployment throughout society (which is why we have governments in the first place). It also means our services and supports — mostly — get smaller, lesser in quality, more spread thinly, and lose their impact because there isn’t the scaled allocation of resources to support them.

Learning is the same way. We divest our interests in it and before you know it, we learn less and do less with it because we haven’t the cultural capital, traditions or infrastructure to handle it. Universities turn campus life to an online experience. Secondary schools stop or reduce teaching physical education that involves actual physical activity.  Scholarly research is reduced to a Google search. Books are given up as learning vehicles because they take too long to read. It goes on.

It’s not that there are no advantages to some of these ideas in some bites, but that we are transforming the entire enterprise with next to no sense of the systems they are operating in, the mission they are to accomplish, a theory of change that is backed up by evidence, or the will to generate the evidence needed to advise and the resources to engage in the sensemaking needed to evaluate that evidence.

Science, systems and learning

It is time to start some serious conversations about systems, science and learning. It would help if we started getting serious about what we mean when we speak of learning, what theories we use to underpin that language and what evidence we have (or need) to understand what those theories mean in practice and for policy. This starts by asking better questions — and lots of them — about learning and its role in our lives and work.

Design thinking and systems thinking are two thinking tools that can help us find and frame these issues. Mindfulness and its ethics associated with non-judgement, open-mindedness, compassion and curiosity are also key tools. The less we judge, the more open we are to asking good questions about what we are seeing that can lead us to getting better answers rather than getting trapped by urban legends.

Doing this within a systems thinking frame also allows us to see how what we learn and where and how we learn is interconnected to better spot areas of leverage and problems in our assumptions.

This might allow us to make many of our urban legends obsolete instead of allowing them to grow like the alligators that live in the sewers of New York City. 

 

 

innovationsocial innovation

The Finger Pointing to the Moon

SuperLuna

SuperLuna

In social innovation we are at risk of confusing our stories of success for real, genuine impact. Without theories, implementation science or evaluation we risk aspiring to travel to the moon, yet leaving our rockets stuck on the launchpad.  

There is a Buddhist expression that goes like this:

Be careful not to confuse the finger pointing to the moon for the moon itself. *

It’s a wonderful phrase that is playful and yet rich in many meanings. Among the most poignant of these meanings is related to the confusion between representation and reality, something we are starting to see exemplified in the world of social innovation and its related fields like design and systems thinking.

On July 13, 2014 the earth experienced a “supermoon” (captured in the above photograph), named because of its close passage to earth. While it may have seemed also close enough to touch, it was still a distance unfathomable to nearly everyone except a handful on this planet. There was a lot of fingers pointed to the moon that night.

While the moon has held fascination for humans for millennia, it’s also worth drawing our attention to the pointing fingers, too.

Pointing fingers

How often do you hear “we are doing amazing stuff“when hearing about leaders describe their social innovations in the community, universities, government, business or partnerships between them? Thankfully, it’s probably a lot more than ever because the world needs good, quality innovative thinking and action. Indeed, judging from the rhetoric at conferences and events and published literature in the academic literature and popular press it seems we are becoming more innovative all the time.

We are changing the world.

…Except, that is a largely useless statement on its own, even if well meaning.

Without documentation of what this “amazing stuff” looks like, a theory or logic explaining how those activities are connected to an outcome and an observed link between it all (i.e., evaluation) there really is no evidence that the world is changed – or at least changed in a manner that is better than had we done something else or nothing at all. That is the tricky part about working with complex systems, particularly large ones. How the world is changed is subtitle of the the book by Brenda Zimmerman, Frances Westley and Michael Quinn Patton on complexity and evaluation in social change, Getting to Maybe. It is because change requires theory, strategic implementation and evaluation that these three leaders in such topics came together to discuss what can be called social innovation. They introduce theory, strategy and evaluation ideas in the book and — while the book has remained a popular text — I rarely see them referred to in serious conversations about social innovation.

Unfortunately, concrete discussion of these three areas — theory, strategic implementation, and evaluation — is largely absent from the dialogue on social innovation. No more was this evident than in the social innovation week events held across Canada in May and June of this year as part of a series of gatherings between practitioners, researchers and policy makers from all kinds of different sectors and disciplines. The events brought together some of the leading thinkers, funders, institutes and social labs from around the world and was as close to the “social innovation olympics” as one could get. The stories told were inspirational, the diversity in the programming was wide, and the ideas shared were creative and interesting.

And yet, many of those I spoke to (including myself) were left with the question: What do I do with any of this? Without something specific to anchor to that question remained unanswered.

Lots of love, not enough (research) power

As often happens, these gatherings serve more as a rallying cry for those working in a sector — something that is quite important on its own as a critical support mechanism — but less about challenging ourselves. As Geoff Mulgan from Nesta noted in the closing keynote to the Social Frontiers event in Vancouver (and riffing off Adam Kahane’s notion of power and love as a vehicle for social transformation), the week featured a lot of love and not so much expression of power (as in critique).

Reflecting on the social innovation events I’ve attended, the books and articles I’ve read, and the conversations I’ve had in the first six months of 2014 it seems evident that the love is being felt by many, but that it is woefully under-powered (pun intended). The social innovation week events just clustered a lot of this conversation in one week, but it’s a sign of a larger trend that emphasizes storytelling independent of the kind of details that one might find at an academic event. Stories can inspire (love), but they rarely guide (power). Adam Kahane is right: we need both to be successful.

The good news is that we are doing love very well and that’s a great start. However, we need to start thinking about the power part of that equation.

There is a dearth of quality research in the field of social innovation and relatively little in the way of concrete theory or documented practice to guide anyone new to this area of work. Yes, there are many stories, but these offer little beyond inspiration to follow. It’s time to add some guidance and a space for critique to the larger narrative in which these stories are told.

Repeating patterns

What often comes from the Q & A sessions following a presentation of a social innovation initiative are the same answers as ‘lessons learned’:

  • Partnerships and trust are key
  • This is very hard work and its all very complex
  • Relationships are important
  • Get buy-in from stakeholders and bring people together to discuss the issues
  • It always takes longer than you think to do things
  • It’s hard to get and maintain resources

I can’t think of a single presentation over the past six months where these weren’t presented as  ‘take-home messages’.

Yet, none of these answers explain what was done in tangible terms, how well it was done, what alternatives exist (if any), what was the rationale for the program and any research/evidence/theory that underpins that logic, and what unintended consequences have emerged from these initiatives and what evaluated outcomes they had besides numbers of participants/events/dollars moved.

We cannot move forward beyond love if we don’t find some way to power-up our work.

Theories of change: The fingers and the moons

Perhaps the best place to start to remedy this problem of detail is developing a theory of change for social innovation**.

Indeed, the emergence of discourse on theory of change in worlds of social enterprise, innovation and services in recent years has been refreshing. A theory of change is pretty much what it sounds like: a set of interconnected propositions that link ideas to outcomes and the processes that exist between them all. A theory of change answers the question: Why should this idea/program/policy produce (specific) changes?

The strengths of the theory of change movement (as one might call it) is that it is inspiring social innovators to think critically about the logic in their programs at a human scale. More flexible than a program logic model and more detailed than a simple hypothesis, a theory of change can guide strategy and evaluation simultaneously and works well with other social innovation-friendly concepts like developmental evaluation and design.

The weaknesses in the movement is that many theories of change fail to consider what has already been developed. There is an enormous amount of conceptual and empirical work done on behaviour change theories at the individual, organization, community and systems level that can inform a theory of change. Disciplines such as psychology, sociology, political theory, geography and planning, business and organizational behaviour, evolutionary biology and others all have well-researched and developed theories to explain changes in activity. Too often, I see theories developed without knowledge or consideration of such established theories. This is not to say that one must rely on past work (particularly in the innovation space where examples might be few in number), but if a theory is solid and has evidence behind it then it is worth considering. Not all theories are created equal.

It is time for social innovation to start raising the bar for itself and the world it seeks to change. It is time to start advancing theories, strategic implementation and evaluation practice and research so that the social innovation events of the future foster real power for change and not just inspiration and love.

 

* one of the more cited translated versions of this phrase has been attributed to Thich Nhat Hanh who suggests the Buddha remarked: “just as a finger pointing at the moon is not the moon itself. A thinking person makes use of the finger to see the moon. A person who only looks at the finger and mistakes it for the moon will never see the real moon.”

** This actually means many theories of change. A theory of change is program-specific and might be identical to another program and built upon the same foundations as others, but just as a program logic model is unique to each program, so too is a theory of change.

Photo credit: SuperLuna with different filters by Paolo Francolini used under Creative Commons License via Flickr