Category: social innovation

complexityevaluationsocial innovation

Developmental Evaluation’s Traps

IMG_0868.jpg

Developmental evaluation holds promise for product and service designers looking to understand the process, outcomes, and strategies of innovation and link them to effects. It’s the great promise of DE that is also the reason to be most wary of it and beware the traps that are set for those unaware.  

Developmental evaluation (DE), when used to support innovation, is about weaving design with data and strategy. It’s about taking a systematic, structured approach to paying attention to what you’re doing, what is being produced (and how), and anchoring it to why you’re doing it by using monitoring and evaluation data. DE helps to identify potentially promising practices or products and guide the strategic decision-making process that comes with innovation. When embedded within a design process, DE provides evidence to support the innovation process from ideation through to business model execution and product delivery.

This evidence might include the kind of information that helps an organization know when to scale up effort, change direction (“pivot”), or abandon a strategy altogether.

Powerful stuff.

Except, it can also be a trap.

It’s a Trap!

Star Wars fans will recognize the phrase “It’s a Trap!” as one of special — and much parodied — significance. Much like the Rebel fleet’s jeopardized quest to destroy the Death Star in Return of the Jedi, embarking on a DE is no easy or simple task.

DE was developed by Michael Quinn Patton and others working in the social innovation sector in response to the needs of programs operating in areas of high volatility, uncertainty, complexity, and ambiguity in helping them function better within this environment through evaluation. This meant providing the kind of useful data that recognized the context, allowed for strategic decision making with rigorous evaluation and not using tools that are ill-suited for complexity to simply do the ‘wrong thing righter‘.

The following are some of ‘traps’ that I’ve seen organizations fall into when approaching DE. A parallel set of posts exploring the practicalities of these traps are going up on the Cense site along with tips and tools to use to avoid and navigate them.

A trap is something that is usually camouflaged and employs some type of lure to draw people into it. It is, by its nature, deceptive and intended to ensnare those that come into it. By knowing what the traps are and what to look for, you might just avoid falling into them.

A different approach, same resourcing

A major trap is going into a DE is thinking that it is just another type of evaluation and thus requires the same resources as one might put toward a standard evaluation. Wrong.

DE most often requires more resources to design and manage than a standard program evaluation for many reasons. One the most important is that DE is about evaluation + strategy + design (the emphasis is on the ‘+’s). In a DE budget, one needs to account for the fact that three activities that were normally treated separately are now coming together. It may not mean that the costs are necessarily more (they often are), but that the work required will span multiple budget lines.

This also means that operationally one cannot simply have an evaluator, a strategist, and a program designer work separately. There must be some collaboration and time spent interacting for DE to be useful. That requires coordination costs.

Another big issue is that DE data can be ‘fuzzy’ or ambiguous — even if collected with a strong design and method — because the innovation activity usually has to be contextualized. Further complicating things is that the DE datastream is bidirectional. DE data comes from the program products and process as well as the strategic decision-making and design choices. This mutually influencing process generates more data, but also requires sensemaking to sort through and understand what the data means in the context of its use.

The biggest resource that gets missed? Time. This means not giving enough time to have the conversations about the data to make sense of its meaning. Setting aside regular time at intervals appropriate to the problem context is a must and too often organizations don’t budget this in.

The second? Focus. While a DE approach can capture an enormous wealth of data about the process, outcomes, strategic choices, and design innovations there is a need to temper the amount collected. More is not always better. More can be a sign of a lack of focus and lead organizations to collect data for data’s sake, not for a strategic purpose. If you don’t have a strategic intent, more data isn’t going to help.

The pivot problem

The term pivot comes from the Lean Startup approach and is found in Agile and other product development systems that rely on short-burst, iterative cycles with accompanying feedback. A pivot is a change of direction based on feedback. Collect the data, see the results, and if the results don’t yield what you want, make a change and adapt. Sounds good, right?

It is, except when the results aren’t well-grounded in data. DE has given cover to organizations for making arbitrary decisions based on the idea of pivoting when they really haven’t executed well or given things enough time to determine if a change of direction is warranted. I once heard the explanation given by an educator about how his team was so good at pivoting their strategy for how they were training their clients and students. They were taking a developmental approach to the course (because it was on complexity and social innovation). Yet, I knew that the team — a group of highly skilled educators — hadn’t spent nearly enough time coordinating and planning the course.

There are times when a presenter is putting things last minute into a presentation to capitalize on something that emerged from the situation to add to the quality of the presentation and then there is someone who has not put the time and thought into what they are doing and rushing at the last minute. One is about a pivot to contribute to excellence, the other is not executing properly. The trap is confusing the two.

Fearing success

“If you can’t get over your fear of the stuff that’s working, then I think you need to give up and do something else” – Seth Godin

A truly successful innovation changes things — mindsets, workflows, systems, and outcomes. Innovation affects the things it touches in ways that might not be foreseen. It also means recognizing that things will have to change in order to accommodate the success of whatever innovation you develop. But change can be hard to adjust to even when it is what you wanted.

It’s a strange truth that many non-profits are designed to put themselves out of business. If there were no more political injustices or human rights violations around the world there would be no Amnesty International. The World Wildlife Fund or Greenpeace wouldn’t exist if the natural world were deemed safe and protected. Conversely, there are no prominent NGO’s developed to eradicate polio anymore because pretty much have….or did we?

Self-sabotage exists for many reasons including a discomfort with change (staying the same is easier than changing), preservation of status, and a variety of inter-personal, relational reasons as psychologist Ellen Hendrikson explains.

Seth Godin suggests you need to find something else if you’re afraid of success and that might work. I’d prefer that organizations do the kind of innovation therapy with themselves, engage in organizational mindfulness, and do the emotional, strategic, and reflective work to ensure they are prepared for success — as well as failure, which is a big part of the innovation journey.

DE is a strong tool for capturing success (in whatever form that takes) within the complexity of a situation and the trap is when the focus is on too many parts or ones that aren’t providing useful information. It’s not always possible to know this at the start, but there are things that can be done to hone things over time. As the saying goes: when everything is in focus, nothing is in focus.

Keeping the parking brake on

And you may win this war that’s coming
But would you tolerate the peace? – “This War” by Sting

You can’t drive far or well with your parking brake on. However, if innovation is meant to change the systems. You can’t keep the same thinking and structures in place and still expect to move forward. Developmental evaluation is not just for understanding your product or service, it’s also meant to inform the ways in which that entire process influences your organization. They are symbiotic: one affects the other.

Just as we might fear success, we may also not prepare (or tolerate) it when it comes. Success with one goal means having to set new goals. It changes the goal posts. It also means that one needs to reframe what success means going ahead. Sports teams face this problem in reframing their mission after winning a championship. The same thing is true for organizations.

This is why building a culture of innovation is so important with DE embedded within that culture. Innovation can’t be considered a ‘one-off’, rather it needs to be part of the fabric of the organization. If you set yourself up for change, real change, as a developmental organization, you’re more likely to be ready for the peace after the war is over as the lyric above asks.

Sealing the trap door

Learning — which is at the heart of DE — fails in bad systems. Preventing the traps discussed above requires building a developmental mindset within an organization along with doing a DE. Without the mindset, its unlikely anyone will avoid falling through the traps described above. Change your mind, and you can change the world.

It’s a reminder of the needs to put in the work to make change real and that DE is not just plug-and-play. To quote Martin Luther King Jr:

“Change does not roll in on the wheels of inevitability, but comes through continuous struggle. And so we must straighten our backs and work for our freedom. A man can’t ride you unless your back is bent.”

 

For more on how Developmental Evaluation can help you to innovate, contact Cense Ltd and let them show you what’s possible.  

Image credit: Author

art & designcomplexityinnovationsocial innovationsystems science

Seeking simplicity among complexity? Go Dutch!

DutchCycle_Snapseed

In a world awash in content and the resulting complexity that comes when it all intersects the viable options for how to manage it remain few. The Dutch De Stijl art and design movement might offer some lessons on dealing with complexity that we can apply beyond products to creating beautiful, functional, and effective services, settings and policy options.

Are you informed about the world? Chances are the answer to that question is both no and yes. There’s no question that you’re informed, the question might be more on what you’re informed about, to what extent, whether that’s of your interest (and relevance and need) and whether it’s an accurate (and useful) depiction of the world around you. That’s a much more complicated set of questions with a troubling set of answers. But one group (the Dutch) may have found some solutions… but we’ll get to that in a moment.

First let’s look at what we’re up against: data streams of distraction.

Data streams of distraction

Consider the many information sources we’re presented with daily.

Consider mine in no particular order, starting with digital : Email (multiple accounts), two course management portals, Instagram, Twitter (two accounts), LinkedIn, Facebook, Facebook Messenger, Whats App, comments on my website or Facebook company page, about 2 dozen apps (on my iPhone and iPad), myriad websites I visit, text messages and, oh yes, occasionally the phone will ring. Next, there’s physical magazines, books, radio or music streams and television, too. Looking out my window I see cranes and buildings and billboards from my downtown loft apartment (and hear birds singing, above it all).

I also encounter real-life human beings, too and they have things to share and more information for me. Funny, that.

This is based on what I choose to look at (even if some choices are rather constrained, such as knowing there is only one way to reach someone and that means engaging with a particular media form I intensely dislike — I’m talking about you, Facebook). Travelling through my day, others will approach and engage, I’ll encounter new things that present themselves and will be handed, shown, flashed or spoken to plenty of other information. The volume of information keeps growing with every encounter.

Then there’s the information stored in memory, the remnants of all of those other days, experiences, and a lifetime of events and information.

This will all happen in real-time, refer to present situations, the past, many possible futures, contain truths, lies, myths and be incomplete in parts all over. It is, in short, a perfect representation of complexity. And it’s causing us a lot of problems.

Information overload

The term ‘information(al) overload’ has been coined to describe the exposure to too much information or data. Information overload and the design problems that information abundance provides has contributed to . Engineers, the builders of much of our critical infrastructure (including, ironically, information technology), know this firsthand and are growing in their concern over how they see that influencing their work. In 2012 the IEEE published a book (PDF) that looked deeply at the role of information overload where the authors note that information overload is not just when people seek new information, but when it information searches for them. The authors argue that:

Information overload “places knowledge workers and managers worldwide in a chronic state of mental overload. It exacts a massive toll on employee productivity and causes significant personal harm, while organizations ultimately pay the price with extensive financial loss”

Annual Reviews, an academic publisher of multidisciplinary research, was motivated to write a piece on information overload in their industry (PDF), noting the present problem is partly one of removing intermediaries:

“…the removal of the intermediary (typically the librarian, but sometimes the publisher) from the information seeking chain…means we are all librarians now, and have to behave like them—constantly reviewing and validating data.”

That takes a lot of work. Both of these works are from 2011-2012 and since then the continued expansion of broadband and mobile technologies, facilitated by cameras and cheaper access to technology, has only added to the amount of information available. The content generation capacity of the public has increased, the consequences are no different, and the solutions fewer.

Perversely, one of the strategies we use to battle overload is to throw more content at the problem as Tom Fishburne shows in this cartoon. We create greater complexity by adding more complexity.  This is the tension. We want to add more information to clarify, rather than strip it away, and end up doing the opposite.

Yet, there may be hope and it is rooted in pragmatism and a desire for beauty: the Dutch design movement, De Stijl.

Designing away complexity: going Dutch

5781402553_2f30fe623c_b

To the untrained eye (which, until a few weeks ago, was mine until I met Corrie van Walraven) the image above would suggest a modern styled home built in the last 20 or 30 years.  Rietveld Schröder House, pictured, was actually built in 1924 and reflects a Dutch design ethos that’s continued through to today of keeping things clean, organized, efficient, flexible, and beautiful.

By many standards the Netherlands has shown itself to be an expert in complexity. Holland is among the most densely populated countries in the world, manages to grow food, survive and thrive in a physical environment that shouldn’t even exist (it is, after all , situated mostly under water). They’ve become masters of adaptation, because they’ve had to be. Dutch design reflects much of this and De Stijl is a perfect example.

Though Dutch design has had many facets and movements De Stijl remains popular partly because of it’s ability to create simplicity amid complexity while creating beauty. Beauty in a designed artifact means it has an evident function, but also elicits a positive aesthetic experience. As Steven de Groot’s research has shown, beauty does not only have intrinsically attractive qualities, but its presence in organizations can lead to higher productivity, employee retention and satisfaction, and overall institutional effectiveness.

Beauty provides an experience of positivity, generally free from confusion, and often clarity. It is lack of clarity and the presence of confusion that is what complexity often brings. Anything that can increase the first and reduce the second while remaining adaptive to the realities of complexity (e.g., information seeking you out) and the data stream is something worth paying attention to; that’s where De Stijl and examples like the Rietveld Schröder House provide guidance.

The house, pictured above, was designed to create a fluid, adaptive space that could configure to a variety of situations and evolve over time. It deals with the amount of content — people, furniture — adaptively, within the boundaries of its walls, in ways that preserve form and function, yet do not get bound too tightly to any particular model. Another distinction is that it is designed to provide the least distinction between the indoor and outdoor spaces. Thus, the design feels somewhat less visible through its simplicity.

Coherence within boundaries

What the De Stijl movement does well is integrate complex ideas together, beautifully, and subscribing to a design philosophy that mirrors Dieter Rams’ belief that we should design as little as possible. De Stijl is about creating coherence – beneficial coherence in complexity terms — within boundaries. It’s work doesn’t seek to integrate the outside and inside (indeed, the criticism of the Rietveld Schröder House is that it doesn’t integrate well within the neighbourhood), but it does exceptionally well within the boundaries of its walls.

What we can take from this is the emphasis on internal coherence within our informational and organizational spaces, because those are the areas we can place boundaries. Systems thinking is all about boundary setting otherwise the focus becomes incoherent. This means being deliberate about where we set up our personal boundaries, professional boundaries and learning boundaries, but in keeping with De Stijl, keeping those flexible and adaptive and always moving, yet in a system that strives for coherence. One of the reasons information overload happens is because we have too much to create coherence with and because we’ve lost what our intention was with the information in the first place.

So a takeaway is this: be intentional about what you’re looking for and what you use. Be mindful of the things that give you coherence in your work and life and create a learning space where you can adapt. Strategy and purpose can help determine this — connect to this. Use the principles of Dutch design through De Stijl to design the conditions that support meaning making.

And if you want a great example in the personal realm, check out another creative thinker with Dutch lineage, Leisse Wilcox, on how self-love through better personal, environmental and social design (my word, not hers) can make you a happier person. That might be the best design you can create of them all.

Acknowledgements: A big thank you to Corrie van Walraven for sharing with me a piece on the De Stijl movement that inspired this post. Corrie’s a great representative of how wonderful the Dutch are and her generosity of spirit and great job as a host is greatly appreciated.

Image Credits: Author and Rietveld Schröder House by frm_tokyo used under Creative Commons License via Flickr.

social innovationsocial systemssystems thinking

Lost together

Lost and found

A post certainty world

Doing new things to create social value means going into the great unknown, yet our fear of being lost need not prevent us from innovating, wisely and sustainably. Instead of being lost alone, we can be lost together. 

I’ve heard it all so many times before

It’s all a dream to me now
A dream to me now
And if we’re lost
Then we are lost together

– Blue Rodeo (Lost Together)

Humans have real problems with uncertainty. Risk mitigation is an enormous field of work within business, government and politics and permeates decision making in our organizations. It’s partly this reason that our politicians too often speak so cryptically to the point of basically uttering nonsense – because they want to avoid the risk of saying something that will hurt them. The alternative perhaps is to spout so much untruth that it no longer matters what you say, because others will create messages about you.

Thankfully, we are still — and hopefully into the future — in a world where most of what organizations do is considered and evaluated with some care to the truth. ‘Truth’ or facts are much easier to deal with in those systems where we can generate the kind of evidence that enable us to make clear decisions based on replicable, verifiable and defensible research. Ordered systems where there is a cause-and-effect relationship that is (usually) clear, consistent and observable are those where we can design interventions to mitigate risk with confidence.

Risky options

There are four approaches to risk mitigation.

  1. Risk Acceptance involves awareness of what risks are present within the system and establishing strategy and an organizational culture where the nature, type and potential consequences of risks are (largely) known, accepted and lived with.
  2. Risk Avoidance takes the opposite approach and seeks to steer operations away from activities where risk is limited.
  3. Risk Limitation seeks to curtail and mitigate the effects of risk where possible and often involves contingency planning and balancing activities with higher levels of uncertainty with areas of greater confidence and certainty.
  4. Risk Transference involves finding ways to offload risk to a third party. An example can be found in many partnerships between organizations of different sizes or types where one is able to absorb certain risks while others cannot for various reasons and the activities allow for one partner to take lead on an activity that isn’t feasible for another to do so.

Within social innovation — those activities involving public engagement, new thinking and social benefit — there are few opportunities for #2, plenty for #1 and #3 and a growing number for #4.

Risk is a core part of innovation. To innovate requires risking time, energy, focus and other resources toward the attempt at something new to produce a valued alternative to the status quo. For many human service organizations and funders, these resources are so thinly spread and small in abundance that the idea of considering risk seems like a risk itself. Yet, the real problem comes in assuming that one can choose whether or not to engage risk. Unless you’re operating in a closed system that has relatively few changing elements to it, you’re exposed to risk by virtue of being in the system. To draw on one of my favourite quotes from the author Guiseppe di Lampedusa:

If we want things to stay as they are, things will have to change.

So even keeping things away from risk involves risk because if the world around you is changing the system changes with it and so, too does your position in it. If this makes you feel lost, you’re not alone. Many organizations (individuals, too) are struggling with figuring out where they fit in the world. If you want evidence of this consider the growing calls for skilled knowledge workers at the same time we are seeing a crisis among those with a PhD — those with the most knowledge (of certain sorts) — in the job market.

Community of flashlights

There is a parable of the drunkard who loses his keys on his way home at night and searches for them under the streetlight not because that’s where he lost them, but because it’s easier to see that spurred something called the Streetlight Effect. It’s about the tendency to draw on what we know and what we have at our disposal to frame problems and seek to solve them, even if they are the wrong tools — a form of observation bias in psychology. Streetlights are anchored, stable means of illuminating a street within a certain range – a risk zone, perhaps — but remain incomplete solutions.

Flashlights on the other hands have the same limitations (a beam of light), are less powerful, but are adaptive. You can port a flashlight or torch and aim it to where you want the light to shine. They are not as powerful as a streetlight in terms of luminosity, but are far more nimble. However, if you bring more than one flashlight together, all of a sudden the power of the light is extended. This is the principle behind many of the commercial LED systems that are in use. Small numbers of lights brought together, each using low energy, but collectively providing a powerful, adaptive lighting system

27242239234_91fb8d5b05_o

This same principle can apply to organizations seeking to make change. Like an LED flashlight, they need a housing to hold and focus the lights. This can be in the form of a backbone organization such as those advocated in collective impact strategies. It can also be a set of principles or simple rules that provide a set of guides for organizations operating independently to follow, which will stimulate a consistent pattern of activity when applied, allowing similar, focused action on the same target at a distance.

This latter approach differs from collective impact, which is a top-down and bottom-up approach simultaneously and is a good means of focusing on larger, macro issues such as poverty reduction, climate change and city-building. It is an approach that holds potential for working within these larger issues on smaller, more dynamic ones such as neighbourhood building, conservation actions within a specific region, and workplace health promotion. In both cases the light analogy can hold and they need not be done exclusive of one another.

Let there be light

A flashlight initiative requires a lot of things coming together. It can be led by individuals making connections between others, brokering relationships and building community. It requires a vision that others can buy into and an understanding of the system (it’s level of complexity, structure and history). This understanding is what serves as the foundation for the determination of the ‘rules’ of the system, those touch points, attractors, leverage points and areas of push and pull that engage energy within a system (stay tuned to a future post for more detailed examples).

Much of the open-source movement is based on this model. This is about creating that housing for ideas to build and form freely, but with constraints. It’s a model that can work when collective impact is at a scale too large for an organization (or individual) to adequately envision contribution, but an alternative to going alone or relying only on the streetlight to find your way.

You might be lost, but with a flashlight you’ll be lost together and may just find your way.

Image credits: Author (Cameron Norman)

evaluationsocial innovation

Flipping the Social Impact Finger

7150802251_e1c559d402_k

Look around and one will notice a lot of talk about social enterprise and social impact. Look closer and you’ll find a lot more of the former and far less of the latter. 

There’s a Buddhist-inspired phrase that I find myself reflecting on often when traveling in the social innovation/entrepreneurship/enterprise/impact sphere:

Do not confuse the finger pointing to the moon for the moon itself

As terms like social enterprise and entrepreneurship, social innovation, social laboratories and social impact (which I’ll lump together as [social] for expediency of writing) become better known and written about its easy to caught up in the excitement and proclaiming its success in changing the world. Indeed, we are seeing a real shift in not only what is being done, but a mental shift in what can be perceived to be done among communities that never saw opportunities to advance before.

However exciting this is, there is what I see as a growing tendency to lose the forest amid the trees by focusing on the growth of [social] and less on the impact part of that collection of terms. In other words, there’s a sense that lots of talk and activity in [social] is translating to social impact. Maybe, but how do we know?

Investment and ROI in change

As I’ve written before using the same guiding phrase cited above, there is a great tendency to confuse conversation about something with the very thing that is being talked about in social impact. For all of the attention paid to the amount of ventures and the amount of venture capital raised to support new initiatives across the social innovation spectrum in recent years, precious little change has been witnessed in the evaluations made available of these projects.

As one government official working in this sector recently told me:

We tend to run out of steam after (innovations) get launched and lose focus, forgetting to evaluate what kind of impact and both intended and unintended consequences come with that investment

As we celebrate the investment in new ventures, track the launch of new start-ups, and document the number of people working in the [social] sector we can mistake that for impact. To be sure, having people working in a sector is a sign of jobs, but the question of whether they are temporary, suitably paying, satisfactory, or sustainable are the kind of questions that evaluators might ask and remain largely unanswered.

The principal ROI of [social] is social benefit. That benefit comes in the form of improved products and services, better economic conditions for more people, and a healthier planet and wellbeing for the population of humans on it in different measures. These aren’t theoretical benefits, they need to be real ones and the only way we will know if we achieve anything approximating this is through evaluation.

Crashing, but not wrecking the party

Evaluation needs to crash the party, but it need not kill the mood. A latent fear among many in [social] is likely that, should we invest so much energy, enthusiasm, money and talent on [social] and find that it doesn’t yield the benefits we expect or need a fickle populace of investors, governments and the public will abandon the sector. While there will always be trend-hunters who will pursue the latest ‘flavour of the month’, [social] is not that. It is here to stay.

The focus on evaluation however will determine the speed, scope and shape of its development. Without showing real impact and learning from those initiatives that produce positive benefit (or do not) we will substantially limit [social] and the celebratory parties that we now have at the launch of a new initiative, a featured post on a mainstream site, or a new book will become fewer and farther between.

Photo credit: Moonrise by James Niland used under Creative Commons licence via Flickr. Thanks for sharing your art, James.

behaviour changecomplexitypublic healthsocial innovation

Confusing change-making with actual change

658beggar_KeepCoinsChange

Change-making is the process of transformation and not to be confused with the transformed outcome that results from such a process. We confuse the two at our peril.

“We are changing the world” is a rallying cry from many individuals and organizations working in social innovation and entrepreneurship which is both a truth and untruth at the same time. Saying you’re changing the world is far easier than actually doing it. One is dramatic — the kind that make for great reality TV as we’ll discuss — and the other is rather dull, plodding and incremental. But it may be the latter that really wins the day.

Organizations like Ashoka (and others) promote themselves as a change-maker organization authoring blogs titled “everything you need to know about change-making”. That kind of language, while attractive and potentially inspiring to diverse audiences, points to a mindset that views social change in relatively simple, linear terms. This line of thinking suggests change is about having the right knowledge and the right plan and the ability to pull it together and execute.

This is a mindset that highlights great people and great acts supported by great plans and processes. I’m not here to dismiss the work that groups like Ashoka do, but to ask questions about whether the recipe approach is all that’s needed. Is it really that simple?

Lies like: “It’s calories in, calories out”

Too often social change is viewed with the same flawed perspective that weight loss is. Just stop eating so much food (and the right stuff) and exercise and you’ll be fine — calories in and out as the quote suggests — and you’re fine. The reality is, it isn’t that simple.

A heartbreaking and enlightening piece in the New York Times profiled the lives and struggles of past winners of the reality show The Biggest Loser (in parallel with a new study released on this group of people (PDF)) that showed that all but one of the contestants regained weight after the show as illustrated below:

BiggestLoser 2016-05-03 09.17.10

The original study, published in the journal Obesity, considers the role of metabolic adaptation that takes place with the authors suggesting that a person’s metabolism makes a proportional response to compensate for the wide fluctuations in weight to return contestants to their original pre-show weight.

Consider that during the show these contestants were constantly monitored, given world-class nutritional and exercise supports, had tens of thousands of people cheering them on and also had a cash prize to vie for. This was as good as it was going to get for anyone wanting to lose weight shy of surgical options (which have their own problems).

Besides being disheartening to everyone who is struggling with obesity, the paper illuminates the inner workings of our body and reveals it to be a complex adaptive system rather than the simple one that we commonly envision when embarking on a new diet or fitness regime. Might social change be the same?

We can do more and we often do

I’m fond of saying that we often do less than we think and more than we know.

That means we tend to expect that our intentions and efforts to make change produce the results that we seek directly and because of our involvement. In short, we treat social change as a straightforward process. While that is sometimes true, rare is it that programs aiming at social change coming close to achieving their stated systems goals (“changing the world”) or anything close to it.

This is likely the case for a number of reasons:

  • Funders often require clear goals and targets for programs in advance and fund based on promises to achieve these results;
  • These kind of results are also the ones that are attractive to outside audiences such as donors, partners, academics, and the public at large (X problem solved! Y number of people served! Z thousand actions taken!), but may not fully articulate the depth and context to which such actions produce real change;
  • Promising results to stakeholders and funders suggests that a program is operating in a simple or complicated system, rather than a complex one (which is rarely, if ever the case with social change);
  • Because program teams know these promised outcomes don’t fit with their system they cherry-pick the simplest measures that might be achievable, but may also be the least meaningful in terms of social change.
  • Programs will often further choose to emphasize those areas within the complex system that have embedded ordered (or simple) systems in them to show effect, rather than look at the bigger aims.

The process of change that comes from healthy change-making can be transformative for the change-maker themselves, yet not yield much in the way of tangible outcomes related to the initial charge. The reasons likely have to do with the compensatory behaviours of the system — akin to social metabolic adaptation — subduing the efforts we make and the initial gains we might experience.

Yet, we do more at the same time. Danny Cahill, one of the contestants profiled in the story for the New York Times, spoke about how the lesson learned from his post-show weight gain was that the original weight gain wasn’t his fault in the first place

“That shame that was on my shoulders went off”

What he’s doing is adapting his plan, his goals and working differently to rethink what he can do, what’s possible and what is yet to be discovered. This is the approach that we take when we use developmental evaluation; we adapt, evolve and re-design based on the evidence while continually exploring ways to get to where we want to go.

A marathon, not a sprint, in a laboratory

The Biggest Loser is a sprint: all of the change work compressed into a short period of time. It’s a lab experiment, but as we know what happens in a laboratory doesn’t always translate directly into the world outside its walls because the constraints have changed. As the show’s attending physician, Dr. Robert Huizenga, told the New York Times:

“Unfortunately, many contestants are unable to find or afford adequate ongoing support with exercise doctors, psychologists, sleep specialists, and trainers — and that’s something we all need to work hard to change”

This quote illustrates the fallacy of real-world change initiatives and exposes some of the problems we see with many of the organizations who claim to have the knowledge about how to change the world. Have these organizations or funders gone back to see what they’ve done or what’s left after all the initial funding and resources were pulled? This is not just a public, private or non-profit problem: it’s everywhere.

I have a colleague who spent much time working with someone who “was hired to clean up the messes that [large, internationally recognized social change & design firm] left behind” because the original, press-grabbing solution actually failed in the long run. And the failure wasn’t in the lack of success, but the lack of learning because that firm and the funders were off to another project. Without building local capacity for change and a sustained, long-term marathon mindset (vs. the sprint) we are setting ourselves up for failure. Without that mindset, lack of success may truly be a failure because there is no capacity to learn and act based on that learning. Otherwise, the learning is just a part of an experimental approach consistent with an innovation laboratory. The latter is a positive, the former, not so much.

Part of the laboratory approach to change is that labs — real research labs — focus on radical, expansive, long-term and persistent incrementalism. Now that might sound dull and unsexy (which is why few seem to follow it in the social innovation lab space), but it’s how change — big change — happens. The key is not in thinking small, but thinking long-term by linking small changes together persistently. To illustrate, consider the weight gain conundrum as posed by obesity researcher Dr. Michael Rosenbaum in speaking to the Times:

“We eat about 900,000 to a million calories a year, and burn them all except those annoying 3,000 to 5,000 calories that result in an average annual weight gain of about one to two pounds,” he said. “These very small differences between intake and output average out to only about 10 to 20 calories per day — less than one Starburst candy — but the cumulative consequences over time can be devastating.”

Building a marathon laboratory

Marathoners are guided by a strange combination of urgency, persistence and patience. When you run 26 miles (42 km) there’s no sprinting if you want to finish the same day you started. The urgency is what pushes runners to give just a little more at specific times to improve their standing and win. Persistence is the repetition of a small number of key things (simple rules in a complex system) that keep the gains coming and the adaptations consistent. Patience is knowing that there are few radical changes that will positively impact the race, just a lot of modifications and hard work over time.

Real laboratories seek to learn a lot, simply and consistently and apply the lessons from one experiment to the next to extend knowledge, confirm findings, and explore new territory.

Marathons aren’t as fun to watch as the 100m sprint in competitive athletics and lab work is far less sexy than the mythical ‘eureka’ moments of ‘discovery’ that get promoted, but that’s what changes the world. The key is to build organizations that support this. It means recognizing learning and that it comes from poor outcomes as well as positive ones. It encourages asking questions, being persistent and not resting on laurels. It also means avoiding getting drawn into being ‘sexy’ and ‘newsworthy’ and instead focusing on the small, but important things that make the news possible in the first place.

Doing that might not be as sweet as a Starburst candy, but it might avoid us having to eat it.

 

 

 

behaviour changecomplexitypsychologysocial innovationsocial systems

Decoding the change genome

11722018066_c29148186b_k

Would we invest in something if we had little hard data to suggest what we could expect to gain from that investment? This is often the case with social programs, yet its a domain that has resisted the kind of data-driven approaches to investment that we’ve seen in other sectors and one theory is that we can approach change in the same way we code the genome, but: is that a good idea?

Jason Saul is a maverick in social impact work and dresses the part: he’s wearing a suit. That’s not typically the uniform of those working in the social sector railing against the system, but that’s one of the many things that gets people talking about what he and his colleagues at Mission Measurement are trying to do. That mission is clear: bring the same detailed analysis of the factors involved in contributing to real impact from the known evidence that we would do to nearly any other area of investment.

The way to achieving this mission is to take the thinking behind the Music Genome Project, the algorithms that power the music service Pandora, and apply it to social impact. This is a big task and done by coding the known literature on social impact from across the vast spectrum of research from different disciplines, methods, theories and modeling techniques. A short video from Mission Measurement on this approach nicely outlines the thinking behind this way of looking at evaluation, measurement, and social impact.

Saul presented his vision for measurement and evaluation to a rapt audience in Toronto at the MaRS Discovery District on April 11th as part of their Global Leaders series en route to the Skoll World Forum ; this is a synopsis of what came from that presentation and it’s implications for social impact measurement.

(Re) Producing change

Saul began his presentation by pointing to an uncomfortable truth in social impact: We spread money around with good intention and little insight into actual change. He claims (no reference provided) that 2000 studies are published per day on behaviour change, yet there remains an absence of common metrics and measures within evaluation to detect change. One of the reasons is that social scientists, program leaders, and community advocates resist standardization making the claim that context matters too much to allow aggregation.

Saul isn’t denying that there is truth to the importance of context, but argues that it’s often used as an unreasonable barrier to leading evaluations with evidence. To this end, he’s right. For example, the data from psychology alone shows a poor track record of reproducibility, and thus offers much less to social change initiatives than is needed. As a professional evaluator and social scientist, I’m not often keen to being told how to do what I do, (but sometimes I benefit from it). That can be a barrier, but also it points to a problem: if the data shows how poorly it is replicated, then is following it a good idea in the first place? 

Are we doing things righter than we think or wronger than we know?

To this end, Saul is advocating a meta-evaluative perspective: linking together the studies from across the field by breaking down its components into something akin to a genome. By looking at the combination of components (the thinking goes) like we do in genetics we can start to see certain expressions of particular behaviour and related outcomes. If we knew these things in advance, we could potentially invest our energy and funds into programs that were much more likely to succeed. We also could rapidly scale and replicate programs that are successful by understanding the features that contribute to their fundamental design for change.

The epigenetic nature of change

Genetics is a complex thing. Even on matters where there is reasonably strong data connecting certain genetic traits to biological expression, there are few examples of genes as ‘destiny’as they are too often portrayed. In other words, it almost always depends on a number of things. In recent years the concept of epigenetics has risen in prominence to provide explanations of how genes get expressed and it has as much to do with what environmental conditions are present as it is the gene combinations themselves . McGill scientist Moshe Szyf and his colleagues pioneered research into how genes are suppressed, expressed and transformed through engagement with the natural world and thus helped create the field of epigenetics. Where we once thought genes were prescriptions for certain outcomes, we now know that it’s not that simple.

By approaching change as a genome, there is a risk that the metaphor can lead to false conclusions about the complexity of change. This is not to dismiss the valid arguments being made around poor data standardization, sharing, and research replication, but it calls into question how far the genome model can go with respect to social programs without breaking down. For evaluators looking at social impact, the opportunity is that we can systematically look at the factors that consistently produce change if we have appropriate comparisons. (That is a big if.)

Saul outlined many of the challenges that beset evaluation of social impact research including the ‘file-drawer effect’ and related publication bias, differences in measurement tools, and lack of (documented) fidelity of programs. Speaking on the matter in response to Saul’s presentation, Cathy Taylor from the Ontario Non-Profit Network, raised the challenge that comes when much of what is known about a program is not documented, but embodied in program staff and shared through exchanges.  The matter of tacit knowledge  and practice-based evidence is one that bedevils efforts to compare programs and many social programs are rich in context — people, places, things, interactions — that remain un-captured in any systematic way and it is that kind of data capture that is needed if we wish to understand the epigenetic nature of change.

Unlike Moshe Szyf and his fellow scientists working in labs, we can’t isolate, observe and track everything our participants do in the world in the service of – or support to – their programs, because they aren’t rats in a cage.

Systems thinking about change

One of the other criticisms of the model that Saul and his colleagues have developed is that it is rather reductionist in its expression. While there is ample consideration of contextual factors in his presentation of the model, the social impact genome is fundamentally based on reductionist approaches to understanding change. A reductionist approach to explaining social change has been derided by many working in social innovation and environmental science as outdated and inappropriate for understanding how change happens in complex social systems.

What is needed is synthesis and adaptation and a meta-model process, not a singular one.

Saul’s approach is not in opposition to this, but it does get a little foggy how the recombination of parts into wholes gets realized. This is where the practical implications of using the genome model start to break down. However, this isn’t a reason to give up on it, but an invitation to ask more questions and to start testing the model out more fulsomely. It’s also a call for systems scientists to get involved, just like they did with the human genome project, which has given us great understanding of what influences our genes have and stressed the importance of the environment and how we create or design healthy systems for humans and the living world.

At present, the genomic approach to change is largely theoretical backed with ongoing development and experiments but little outcome data. There is great promise that bigger and better data, better coding, and a systemic approach to looking at social investment will lead to better outcomes, but there is little actual data on whether this approach works, for whom, and under what conditions. That is to come. In the meantime, we are left with questions and opportunities.

Among the most salient of the opportunities is to use this to inspire greater questions about the comparability and coordination of data. Evaluations as ‘one-off’ bespoke products are not efficient…unless they are the only thing that we have available. Wise, responsible evaluators know when to borrow or adapt from others and when to create something unique. Regardless of what design and tools we use however, this calls for evaluators to share what they learn and for programs to build the evaluative thinking and reflective capacity within their organizations.

The future of evaluation is going to include this kind of thinking and modeling. Evaluators, social change leaders, grant makers and the public alike ignore this at their peril, which includes losing opportunities to make evaluation and social impact development more accountable, more dynamic and impactful.

Photo credit (main): Genome by Quinn Dombrowski used under Creative Commons License via Flickr. Thanks for sharing Quinn!

About the author: Cameron Norman is the Principal of Cense Research + Design and assists organizations and networks in supporting learning and innovation in human services through design, program evaluation, behavioural science and system thinking. He is based in Toronto, Canada.

evaluationsocial innovation

Benchmarking change

The quest for excellence within social programs relies on knowing what excellence means and how programs compare against others. Benchmarks can enable us to compare one program to another if we have quality comparators and an evaluation culture to generate them – something we currently lack. 

5797496800_7876624709_b

A benchmark is something used by surveyors to provide a means of holding a levelling rod to determine some consistency in elevation measurement of a particular place that could be compared over time. A benchmark represents a fixed point for measurement to allow comparisons over time.

The term benchmark is often used in evaluation as a means of providing comparison between programs or practices, often taking one well-understood and high performing program as the ‘benchmark’ to which others are compared. Benchmarks in evaluation can be the standard to which other measures compare.

In a 2010 article for the World Bank (PDF), evaluators Azevedo, Newman and Pungilupp, articulate the value of benchmarking and provide examples for how it contributes to the understanding of both absolute and relative performance of development programs. Writing about the need for benchmarking, the authors conclude:

In most benchmarking exercises, it is useful to consider not only the nature of the changes in the indicator of interest but also the level. Focusing only on the relative performance in the change can cause the researcher to be overly optimistic. A district, state or country may be advancing comparatively rapidly, but it may have very far to go. Focusing only on the relative performance on the level can cause the researcher to be overly pessimistic, as it may not be sufficiently sensitive to pick up recent changes in efforts to improve.

Compared to what?

One of the challenges with benchmarking exercises is finding a comparator. This is easier for programs operating with relatively simple program systems and structures and less so for more complex ones. For example, in the service sector wait times are a common benchmark. In the province of Ontario in Canada, the government provides regularly updated wait times for Emergency Room visits via a website. In the case of healthcare, benchmarks are used in multiple ways. There is a target that is used as the benchmark, although, depending on the condition, this target might be on a combination of aspiration, evidence, as well as what the health system believes is reasonable, what the public demands (or expects) and what the hospital desires.

Part of the problem with benchmarks set in this manner is that they are easy to manipulate and thus raise the question of whether they are true benchmarks in the first place or just goals.

If I want to set a personal benchmark for good dietary behaviour of eating three meals a day, I might find myself performing exceptionally well as I’ve managed to do this nearly every day within the last three months. If the benchmark is consuming 2790 calories as is recommended for someone of my age, sex, activity levels, fitness goals and such that’s different. Add on that, within that range of calories, the aim is to have about 50% of those come from carbohydrates, 30% from fat and 20% from protein, and we a very different set of issues to consider when contemplating how performance relates to a standard.

One reason we can benchmark diet targets is that the data set we have to set that benchmark is enormous. Tools like MyFitnessPal and others operate to use benchmarks to provide personal data to its users to allow them to do fitness tracking using these exact benchmarks that are gleaned from having 10’s of thousands of users and hundreds of scientific articles and reports on diet and exercise from the past 50 years. From this it’s possible to generate reasonably appropriate recommendations for a specific age group and sex.

These benchmarks are also possible because we have internationally standardized the term calorie. We have further internationally recognized, but slightly less precise, measures for what it means to be a certain age and sex. Activity level gets a little more fuzzy, but we still have benchmarks for it. As the cluster of activities that define fitness and diet goals get clustered together we start to realize that it is a jumble of highly precise and somewhat loosely defined benchmarks.

The bigger challenge comes when we don’t have a scientifically validated standard or even a clear sense of what is being compared and that is what we have with social innovation.

Creating an evaluation culture within social innovation

Social innovation has a variety of definitions, however the common thread of these is that its about a social program aimed at address social problems using ideas, tools, policies and practices that differ from the status quo. Given the complexity of the environments that many social programs are operating, it’s safe to assume that social innovation** is happening all over the world because the contexts are so varied. The irony is that many in this sector are not learning from one another as much as they could, further complicating any initiative to build benchmarks for social programs.

Some groups like the Social Innovation Exchange (SIX) are trying to change that. However, they and others like them, face an uphill battle. Part of the reason is that social innovation has not established a culture of evaluation within it. There remains little in the way of common language, frameworks, or spaces to share and distribute knowledge about programs — both in description and evaluation — in a manner that is transparent and accessible to others.

Competition for funding, the desire to paint programs in a positive light, lack of expertise, not enough resources available for dissemination and translation, absence of a dedicated space for sharing results, and distrust or isolation from academia among certain sectors are some reasons that might contribute to this. For example, the Stanford Social Innovation Review is among the few venues dedicated to scholarship in social innovation aimed at a wide audience. It’s also a venue focused largely on international development and what I might call ‘big’ social innovation: the kind of works that attract large philanthropic resources. There’s lot of other types of social innovation and they don’t all fit into the model that SSIR promotes.

From my experiences, many small organizations or initiatives struggle to fund evaluation efforts sufficiently, let alone the dissemination of the work once it’s finished. Without good quality evaluations and the means to share their results — whether or not they cast a program in positive light or not — it’s difficult to build a culture where the sector can learn from one another. Without a culture of evaluation, we also don’t get the volume of data and access to comparators — appropriate comparators, not just the only things we can find — to develop true, useful benchmarks.

Culture’s feast on strategy

Building on the adage attributed to Peter Drucker that culture eats strategy for breakfast (or lunch) it might be time that we use that feasting to generate some energy for change. If the strategy is to be more evidence based, to learn more about what is happening in the social sector, and to compare across programs to aid that learning there needs to be a culture shift.

This requires some acknowledgement that evaluation, a disciplined means of providing structured feedback and monitoring of programs, is not something adjunct to social innovation, but a key part of it. This is not just in the sense that evaluation provides some of the raw materials (data) to make informed choices that can shape strategy, but that it is as much a part of the raw material for social change as enthusiasm, creativity, focus, and dissatisfaction with the status quo on any particular condition.

We are seeing a culture of shared ownership and collective impact forming, now it’s time to take that further and shape a culture of evaluation that builds on this so we can truly start sharing, building capacity and developing the real benchmarks to show how well social innovation is performing. In doing so, we make social innovation more respectable, more transparent, more comparable and more impactful.

Only by knowing what we are doing and have done can we really sense just how far we can go.

** For this article, I’m using the term social innovation broadly, which might encompass many types of social service programs, government or policy initiatives, and social entrepreneurship ventures that might not always be considered social innovation.

Photo credit: Redwood Benchmark by Hitchster used under Creative Commons License from Flickr.

About the author: Cameron Norman is the Principal of Cense Research + Design and works at assisting organizations and networks in supporting learning and innovation in human services through design, program evaluation, behavioural science and system thinking. He is based in Toronto, Canada.