Metrics are at the heart of evaluation of impact and value in products and services although they are rarely straightforward. What makes a good metric requires some thinking about what the meaning of a metric is, first.
I recently read a story on what makes a good metric from Chris Moran, Editor of Strategic Projects at The Guardian. Chris’s work is about building, engaging, and retaining audiences online so he spends a lot of time thinking about metrics and what they mean.
Chris – with support from many others — outlines the five characteristics of a good metric as being:
Relevant
Measurable
Actionable
Reliable
Readable (less likely to be misunderstood)
(What I liked was that he also pointed to additional criteria that didn’t quite make the cut but, as he suggests, could).
This list was developed in the context of communications initiatives, which is exactly the point we need to consider: context matters when it comes to metrics. Context also is holistic, thus we need to consider these five (plus the others?) criteria as a whole if we’re to develop, deploy, and interpret data from these metrics.
As John Hagel puts it: we are moving from the industrial age where standardized metrics and scale dominated to the contextual age.
Sensemaking and metrics
Innovation is entirely context-dependent. A new iPhone might not mean much to someone who has had one but could be transformative to someone who’s never had that computing power in their hand. Home visits by a doctor or healer were once the only way people were treated for sickness (and is still the case in some parts of the world) and now home visits are novel and represent an innovation in many areas of Western healthcare.
Demographic characteristics are one area where sensemaking is critical when it comes to metrics and measures. Sensemaking is a process of literally making sense of something within a specific context. It’s used when there are no standard or obvious means to understand the meaning of something at the outset, rather meaning is made through investigation, reflection, and other data. It is a process that involves asking questions about value — and value is at the core of innovation.
For example, identity questions on race, sexual orientation, gender, and place of origin all require intense sensemaking before, during, and after use. Asking these questions gets us to consider: what value is it to know any of this?
How is a metric useful without an understanding of the value in which it is meant to reflect?
What we’ve seen from population research is that failure to ask these questions has left many at the margins without a voice — their experience isn’t captured in the data used to make policy decisions. We’ve seen the opposite when we do ask these questions — unwisely — such as strange claims made on associations, over-generalizations, and stereotypes formed from data that somehow ‘links’ certain characteristics to behaviours without critical thought: we create policies that exclude because we have data.
The lesson we learn from behavioural science is that, if you have enough data, you can pretty much connect anything to anything. Therefore, we need to be very careful about what we collect data on and what metrics we use.
The role of theory of change and theory of stage
One reason for these strange associations (or absence) is the lack of a theory of change to explain why any of these variables ought to play a role in explaining what happens. A good, proper theory of change provides a rationale for why something should lead to something else and what might come from it all. It is anchored in data, evidence, theory, and design (which ties it together).
Metrics are the means by which we can assess the fit of a theory of change. What often gets missed is that fit is also context-based by time. Some metrics have a better fit at different times during an innovation’s development.
For example, a particular metric might be more useful in later-stage research where there is an established base of knowledge (e.g., when an innovation is mature) versus when we are looking at the early formation of an idea. The proof-of-concept stage (i.e., ‘can this idea work?’) is very different than if something is in the ‘can this scale’? stage. To that end, metrics need to be fit with something akin to a theory of stage. This would help explain how an innovation might develop at the early stage versus later ones.
Metrics are useful. Blindly using metrics — or using the wrong ones — can be harmful in ways that might be unmeasurable without the proper thinking about what they do, what they represent, and which ones to use.
Developmental evaluation holds promise for product and service designers looking to understand the process, outcomes, and strategies of innovation and link them to effects. It’s the great promise of DE that is also the reason to be most wary of it and beware the traps that are set for those unaware.
Developmental evaluation (DE), when used to support innovation, is about weaving design with data and strategy. It’s about taking a systematic, structured approach to paying attention to what you’re doing, what is being produced (and how), and anchoring it to why you’re doing it by using monitoring and evaluation data. DE helps to identify potentially promising practices or products and guide the strategic decision-making process that comes with innovation. When embedded within a design process, DE provides evidence to support the innovation process from ideation through to business model execution and product delivery.
This evidence might include the kind of information that helps an organization know when to scale up effort, change direction (“pivot”), or abandon a strategy altogether.
Powerful stuff.
Except, it can also be a trap.
It’s a Trap!
Star Wars fans will recognize the phrase “It’s a Trap!”as one of special — and much parodied — significance. Much like the Rebel fleet’s jeopardized quest to destroy the Death Star in Return of the Jedi, embarking on a DE is no easy or simple task.
DE was developed by Michael Quinn Patton and others working in the social innovation sector in response to the needs of programs operating in areas of high volatility, uncertainty, complexity, and ambiguity in helping them function better within this environment through evaluation. This meant providing the kind of useful data that recognized the context, allowed for strategic decision making with rigorous evaluation and not using tools that are ill-suited for complexity to simply do the ‘wrong thing righter‘.
The following are some of ‘traps’ that I’ve seen organizations fall into when approaching DE. A parallel set of posts exploring the practicalities of these traps are going up on the Cense site along with tips and tools to use to avoid and navigate them.
A trap is something that is usually camouflaged and employs some type of lure to draw people into it. It is, by its nature, deceptive and intended to ensnare those that come into it. By knowing what the traps are and what to look for, you might just avoid falling into them.
A different approach, same resourcing
A major trap is going into a DE is thinking that it is just another type of evaluation and thus requires the same resources as one might put toward a standard evaluation. Wrong.
DE most often requires more resources to design and manage than a standard program evaluation for many reasons. One the most important is that DE is about evaluation + strategy + design (the emphasis is on the ‘+’s). In a DE budget, one needs to account for the fact that three activities that were normally treated separately are now coming together. It may not mean that the costs are necessarily more (they often are), but that the work required will span multiple budget lines.
This also means that operationally one cannot simply have an evaluator, a strategist, and a program designer work separately. There must be some collaboration and time spent interacting for DE to be useful. That requires coordination costs.
Another big issue is that DE data can be ‘fuzzy’ or ambiguous — even if collected with a strong design and method — because the innovation activity usually has to be contextualized. Further complicating things is that the DE datastream is bidirectional. DE data comes from the program products and process as well as the strategic decision-making and design choices. This mutually influencing process generates more data, but also requires sensemaking to sort through and understand what the data means in the context of its use.
The biggest resource that gets missed? Time. This means not giving enough time to have the conversations about the data to make sense of its meaning. Setting aside regular time at intervals appropriate to the problem context is a must and too often organizations don’t budget this in.
The second? Focus. While a DE approach can capture an enormous wealth of data about the process, outcomes, strategic choices, and design innovations there is a need to temper the amount collected. More is not always better. More can be a sign of a lack of focus and lead organizations to collect data for data’s sake, not for a strategic purpose. If you don’t have a strategic intent, more data isn’t going to help.
The pivot problem
The term pivot comes from the Lean Startup approach and is found in Agile and other product development systems that rely on short-burst, iterative cycles with accompanying feedback. A pivot is a change of direction based on feedback. Collect the data, see the results, and if the results don’t yield what you want, make a change and adapt. Sounds good, right?
It is, except when the results aren’t well-grounded in data. DE has given cover to organizations for making arbitrary decisions based on the idea of pivoting when they really haven’t executed well or given things enough time to determine if a change of direction is warranted. I once heard the explanation given by an educator about how his team was so good at pivoting their strategy for how they were training their clients and students. They were taking a developmental approach to the course (because it was on complexity and social innovation). Yet, I knew that the team — a group of highly skilled educators — hadn’t spent nearly enough time coordinating and planning the course.
There are times when a presenter is putting things last minute into a presentation to capitalize on something that emerged from the situation to add to the quality of the presentation and then there is someone who has not put the time and thought into what they are doing and rushing at the last minute. One is about a pivot to contribute to excellence, the other is not executing properly. The trap is confusing the two.
Fearing success
“If you can’t get over your fear of the stuff that’s working, then I think you need to give up and do something else” – Seth Godin
A truly successful innovation changes things — mindsets, workflows, systems, and outcomes. Innovation affects the things it touches in ways that might not be foreseen. It also means recognizing that things will have to change in order to accommodate the success of whatever innovation you develop. But change can be hard to adjust to even when it is what you wanted.
It’s a strange truth that many non-profits are designed to put themselves out of business. If there were no more political injustices or human rights violations around the world there would be no Amnesty International. The World Wildlife Fund or Greenpeace wouldn’t exist if the natural world were deemed safe and protected. Conversely, there are no prominent NGO’s developed to eradicate polio anymore because pretty much have….or did we?
Seth Godin suggests you need to find something else if you’re afraid of success and that might work. I’d prefer that organizations do the kind of innovation therapy with themselves, engage in organizational mindfulness, and do the emotional, strategic, and reflective work to ensure they are prepared for success — as well as failure, which is a big part of the innovation journey.
DE is a strong tool for capturing success (in whatever form that takes) within the complexity of a situation and the trap is when the focus is on too many parts or ones that aren’t providing useful information. It’s not always possible to know this at the start, but there are things that can be done to hone things over time. As the saying goes: when everything is in focus, nothing is in focus.
Keeping the parking brake on
And you may win this war that’s coming
But would you tolerate the peace? – “This War” by Sting
You can’t drive far or well with your parking brake on. However, if innovation is meant to change the systems. You can’t keep the same thinking and structures in place and still expect to move forward. Developmental evaluation is not just for understanding your product or service, it’s also meant to inform the ways in which that entire process influences your organization. They are symbiotic: one affects the other.
Just as we might fear success, we may also not prepare (or tolerate) it when it comes. Success with one goal means having to set new goals. It changes the goal posts. It also means that one needs to reframe what success means going ahead. Sports teams face this problem in reframing their mission after winning a championship. The same thing is true for organizations.
This is why building a culture of innovation is so important with DE embedded within that culture. Innovation can’t be considered a ‘one-off’, rather it needs to be part of the fabric of the organization. If you set yourself up for change, real change, as a developmental organization, you’re more likely to be ready for the peace after the war is over as the lyric above asks.
It’s a reminder of the needs to put in the work to make change real and that DE is not just plug-and-play. To quote Martin Luther King Jr:
“Change does not roll in on the wheels of inevitability, but comes through continuous struggle. And so we must straighten our backs and work for our freedom. A man can’t ride you unless your back is bent.”
For more on how Developmental Evaluation can help you to innovate, contact Cense Ltd and let them show you what’s possible.
There is a point at which information ceases to increase knowledge and understand and begins to undermine it, creating a paradox. When information on nearly anything is more abundant than ever the choices we make about how to engage it become more important than ever.
The Information Age has been described as the period where industrial production was replaced by knowledge production as the key driver of social and economic benefit for society. Underpinning the thinking behind the information age (and the digital revolution that accompanies it) is that having more information, more access to it and improved tools to use it to solve problems will improve life for everyone. Presented with a choice to have access to more information or less people will almost always choose more.
More information leads to more options, which equals more choice and more choice is about freedom and that is seen as an inherent social good derived from the capitalist system, which further leads to better choices, more freedom and greater happiness overall. At least, this is what we’ve been led to believe and Barry Schwartz explains this quite eloquently in the opening of talk embedded later in this post.
This is the theory of change that underpins information theory as its played out in modern capitalist societies. It’s also the source of many of our modern paradoxes and problems.
Systems of influence: The case of the ePatient
I’ve stopped going to health-related hackathons and design jams altogether for the simple reason that one can almost always guarantee that one third or more of the solutions generated will be some form of app or information-focused tool. These well-meaning, creative tools are part of a consumer health movement that is all about putting information in the hands of patients with the idea that putting information in the hands of patients is the key to empowerment and better health outcomes, except they rarely lead to this promised land.
Few are better at explaining — and indeed living — this reality than Dave deBronkart or ‘e-Patient Dave’ who has been a tireless advocate for better information tools, access and engagement on health for patients. His Ted Talk captures the spirit of the movement nicely.
With all due respect to the positive sentiments around what the ePatient movement is about, it is based on a series of assumptions about health systems, patients and health itself in ways that don’t always hold. For certain patients, certain conditions, and certain contexts having more information delivered in the right format is indeed empowering and may be life saving as deBronkart’s story illustrates. But what’s often missing from these stories of success are the many layers of assumptions and conditions that underpin information-driven healthcare.
A few years ago I interviewed a patient who spoke about his experience with health care decision-making and information technology and his response was that having more information didn’t make his life much better, rather it made it even more complicated because with more access to more information he had more responsibility related to that information.
“I don’t know what to do with it all and there’s an assumption that once I know (this health information) I am in a position to do something. I don’t have the foggiest idea what to do, that’s why I am going to see (the health professionals) because that is what their job is for. They are the ones who are supposed to know what is to be done. It’s their world, not mine.”
This case is less about deferral to authority, but about resources (e.g., knowledge, skill, time, networks, etc..) and expectations around what comes with those resources. When you are unwell the last thing you want is to be told you have even more work to do.
The assumptions around personal health information and decision-making are that people have:
1) access to the data in the first place, 2) time, 3) information gathering tools, 4) knowledge synthesis tools, 5) skill and knowledge of how to sift, sort, synthesize and sense-make all the information obtained (because it may be in different formats, incomplete, or require conversions), 6) access to the people and other knowledge and skills required to appropriately sense-make the data, 7) the resources to act on whatever conclusions are drawn from that process, 8) a system that is able to respond to the actions that are needed and taken (and in a timely manner), 9) the personal willpower, energy, and resolve to persist through the various challenges and pushback from the system to resist the actions taken, 10) social support (because this is virtually impossible to do without any support at all) and 11) the motivation and interest in doing all of this in the first place.
Dave deBronkart and his peers are advocating for patient engagement on a broader level and that includes creating spaces for patients to have the choice as to what kind of information they use or not. This also means having choice to NOT have information. It’s not about technology pushing, but having a choice about what to access, when and how. That’s noble and often helpful to those who are used to not having much say in what happens, but that, too has problems of its own.
The paradox of choice
Barry Schwartz’s work (pdf) doing and synthesizing research on consumer decision-making puts truth to this lie that more choice is better. Choice options add value only to a certain point after which they degrade value and even subvert it altogether. The problem is that choice options are often ‘all or nothing’ and may be addictive if left unconstrained as we’ll see below.
Schwartz addresses the matter of decision-making in healthcare in the above video and points to the shifting of responsibility away from experts to everyone. Perhaps it is not surprising that we are seeing an incredible backlash against expert-driven knowledge and science in a way that we’ve not seen in over a hundred years. This is at a time when the public has access to more scientific data — the same data that scientists and other experts have — through open data and open access scientific publications to validate the claims by experts.
As discussed in a previous post, another feature of this wealth of information is that we are now living in what some call a post-truth political climate where almost anything goes. Speaking on the matter of science and climate change former Alaska Governor and Vice Presidential candidate Sarah Palin suggested that, when compared to Dr Bill Nye (the Science Guy and a rocket scientist — yes, a real rocket scientist ), she is as much of a scientist as he is.
Why have science when you can have opinion?
Distracted driving on the information superhighway
Recent data from Canada shows that year-over-year growth in smartphone use at 24% to over two thirds of the population with 85% reporting some form of mobile phone ownership. One of the key features of modern smartphones is the ‘always on’ nature of their tools and alert systems allowing you to bring maps, address books, a digital library, video and audio telephony, and the entire Internet in your pocket.
The distractions that come from the tools meant to deliver information are becoming crippling to some to the point of distancing us from our humanity itself. The title of a beautiful, sad piece in New York Magazine by Andrew Sullivan put this into perspective: I used to be a human being. (We will come back to this in a future post.)
But even if one still feels human using information technology, its a different experience of humanity than it once was. Behaviour change writer and coach Tony Schwartz (I’m not sure if he’s related to Barry), writing in the New York Times magazine, noted how his use of information technology was affecting his ability to, ironically, glean information from something simple as a book.
One evening early this summer, I opened a book and found myself reading the same paragraph over and over, a half dozen times before concluding that it was hopeless to continue. I simply couldn’t marshal the necessary focus.
He goes on to explain what is being exchanged for the books he had aspired to read:
Instead of reading them, I was spending too many hours online, checking the traffic numbers for my company’s website, shopping for more colorful socks on Gilt and Rue La La, even though I had more than I needed, and even guiltily clicking through pictures with irresistible headlines such as “Awkward Child Stars Who Grew Up to Be Attractive.”
We can laugh at the last bit because most of us have been online and lured by something we thought was impossible or ridiculous and had to inquire about. Link bait is not new or particularly clever, but it works. It works for a variety of reasons, but largely because we need to inhabit the same space to work as well as to play. The problem comes when these worlds cross-over into one another.
For example, I recently was shopping for a safe (no, it’s not to store my non-existent millions, but rather protect hard drives and enhance data security) and wanted to return to a story I’d read in the Guardian for a different blog post. As I returned to pull the URL for this I found the page looking like this:
All of a sudden I am confronted with shopping choices amidst a quest for a URL.
Information wealth: A Faustian bargain to knowledge poverty?
“We willingly accept the loss of concentration and focus, the division of our attention and the fragmentation of our thoughts, in return for the wealth of compelling or at least diverting information we receive.”
Tony Shwartz’s comments above and below point to what we know about how information works in our brain. We can try and resist, but the evolutionary reasons we pay attention to things and the biological limitations we have to processing it all are most likely to trump any efforts to resist it without substantial shifts to our practices.
Endless access to new information also easily overloads our working memory. When we reach cognitive overload, our ability to transfer learning to long-term memory significantly deteriorates. It’s as if our brain has become a full cup of water and anything more poured into it starts to spill out.
I wish I had the answers to what these are. Schwartz, has proposed a digital vacation. As beneficial as it was for him, he was also willing to admit that it’s not a perfect strategy and that he still spends too much time online, too distracted. But, its better.
Comedian Louis C.K. has taken to ‘quitting the internet’ altogether and, in a touching moment of reflection (as he often does with wit), notes how it has improved the relationship with his daughters.
It’s these relational aspects of the new information technology and how it impacts our world that concern me the most and creates the most troubling paradox: the tools that are designed to bring us together might just be making it harder to be together and pushing us apart from each other and ourselves. This is what I will look at in the next piece in this series on paradox.
Image credit:Information by Heath Brandon used under Creative Commons License and by author
The costs of books, materials, tuition, or conference fees often distort the perception of how much learning costs, creating larger distortions in how we perceive knowledge to benefit us. By looking at what price we pay for integrating knowledge and experience we might re-valuate what we need, what we have and what we pay attention to in our learning and innovation quest.
A quote paraphrased and attributed to German philosopher Arthur Schopenhauer points to one of the fundamental problems facing books:
Buying books would be a good thing if one could also buy the time to read them in: but as a rule the purchase of books is mistaken for the appropriation of their contents.
Schopenhauer passed away in 1860 when the book was the dominant media form of codified knowledge and the availability of books was limited. This was before radio, television, the Internet and the confluence of it all in today’s modern mediascape from Amazon to the iPhone and beyond.
Schopenhauer exposes the fallacy of thought that links having access to information to knowledge. This fallacy underpins the major challenges facing our learning culture today: quantity of information vs quality of integration.
Learning time
Consider something like a conference or seminar. How often have you attended a talk or workshop and been moved by what you heard and saw, took furious notes, and walked out of the room vowing to make a big change based on what you just experienced? And then what happened? My guess is that the world outside that workshop or conference looked a lot different than it appeared in it. You had emails piled up, phone messages to return, colleagues to convince, resources to marshall, patterns to break and so on.
Among the simple reasons is that we do not protect the time and resources required to actually learn and to integrate that knowledge into what we do. As a result, we mistakenly look at the volume of ‘things’ we expose ourselves to for learning outcomes.
One solution is to embrace what consultant, writer and blogger Sarah Van Bargen calls “intentional ignorance“. This approach involves turning away from the ongoing stream of data and accepting that there are things we won’t know and that we’ll just miss. Van Bargen isn’t calling for a complete shutting of the door, rather something akin to an information sabbatical or what some might call digital sabbath. Sabbath and sabbatical share the Latin root sabbatum, which means “to rest”.
Rebecca Rosen who writes on work and business for The Atlantic argues we don’t need a digital sabbath, we need more time. Rosen’s piece points to a number of trends that are suggesting the way we work is that we’re producing more, more often and doing it more throughout the day. The problem is not about more, it’s about less. It’s also about different.
Time, by design
One of the challenges is our relationship to time in the first place and the forward orientation we have to our work. We humans are designed to look forward so it is not a surprise that we engineer our lives and organizations to do the same. Sensemaking is a process that orients our gaze to the future by looking at both the past and the present, but also by taking time to look at what we have before we consider what else we need. It helps reduce or at least manage complex information to enable actionable understanding of what data is telling us by putting it into proper context. This can’t be done by automation.
It takes time.
It means….
….setting aside time to look at the data and discuss it with those who are affected by it, who helped generate it, and are close to the action;
….taking time to gather the right kind of information, that is context-rich, measures things that have meaning and does so with appropriate scope and precision;
….create organizational incentives and protections for people to integrate what they know into their jobs and roles and to create organizations that are adaptive enough to absorb, integrate and transform based on this learning — becoming a true learning organization.
By changing the practices within an organization we can start shifting the way we learn and increase the likelihood of learning taking place.
Buying time
Imagine buying both the book and the time to read the book and think about it. Imagine sending people on courses and then giving them the tools and opportunity to try the lessons (the good ones at least) in practice within the context of the organization. If learning is really a priority, what kind of time is given to people to share what they know, listen to others, and collectively make sense of what it means and how it influences strategy?
What we might find is that we do less. We buy less. We attend less. We subscribe to less. Yet, we absorb more and share more and do more as a result.
The cost of learning then shifts — maybe even to less than we spend now — but what it means is that we factor in time not just product in our learning and knowledge production activities.
Although Innovation is about producing value through doing something new or different than before, the concept is far from simple when applied in practice by individuals and institutions. This second in a series of articles on innovation ecology looks at the way we speak of innovation and how what we talk about new ideas and discovery shapes what we do about it.
“Language can be a way of hiding your thoughts and preventing communication” – Abraham Maslow
Innovation is one of the few concepts that offers little benefit contemplated in the abstract. We innovate on specific things with an eye to application, maybe even scaling that idea broadly. Humans innovate because the status quo is no longer satisfying, is unacceptable or has changed so we strive to come up with new ways of doing things, novel processes and tools to make the current situation a preferred one.
Thus, we are designers seeking our client, customer and creation through innovation and we do this through our words and actions — our language. Indeed, if one agrees with Marty Neumeier‘s assertion that design is the discipline of innovation and Greg Van Alystne & Bob Logan’s definition of design as “creation for reproduction” then our language of innovation is critical to ensuring that we design products and services that have the potential to reproduce beyond an idea.
Language matters in innovation.
To illustrate, lets look at how language manifests itself in the communication of ideas using an example from public health. In a paper entitled Knowledge integration: Conceptualizing communications in cancer control systems I co-authored with my colleagues Allan Best and Bob Hiatt, we looked at the way language was used within a deep and broad field like cancer control in shaping communications. This was not merely an academic exercise, but served to illustrate the values, practices and structures that are put in place to support communicating concepts and serves to illustrate how innovations are communicated.
Innovation as product
What we found was that there are three generations of cancer communications defined by their language and the practices and policies that are manifested in or representative of that language. The first generation of terms were traced up to the 1990’s and were characterized by viewing knowledge as a product. Indeed, the term knowledge products can be traced back to this period. Other key characteristics of this period include:
The terminology used to describe communications included the terms diffusion, dissemination, knowledge transfer, and knowledge uptake.
Focus on the handoff between knowledge ‘producers’ and knowledge (or research) ‘users’. These two groups were distinct and separate from one another
The degree of use is a function of effective packaging and presentation presuming the content is of high quality.
The language of this first generation makes the assumption that the ideas are independent of the context in which they are to be used or where they were generated. The communication represented in this generation of models relies on expertise and recognition of this. But what happens when expertise is not recognized? Or where expertise isn’t even possible? This is a situation we are increasingly seeing as we face new, complex challenges that require mass collaboration and innovation, something the Drucker Forum suggests represents the end of expertise.
Innovation as a contextual process
From the early and mid-1990’s through to the present we’ve seen a major shift from viewing knowledge or innovation as a product to that of a dynamic process where expertise resides in multiple places and sources and networks are valued as much as institutions or individuals. Some of the characteristics of this generation are:
Knowledge and good ideas come from multiple sources, not just recognized experts or leaders
Social relationships media what is generated and how it is communicated (and to whom)
Innovation is highly context-dependent
The degree of use of ideas or knowledge is a function of having strong, effective relationships and processes.
What happens when the context is changing consistently? What happens when the networks are dynamic and often unknown?
Systems-embedded innovation
What the paper argues is that we are seeing a shift toward more systems-oriented approaches to communication and that is represented in the term knowledge integration. A systems-oriented model views the design of knowledge structures as an integral to the support of effective innovation by embedding the activities of innovation — learning, discovery, and communication — within systems like institutions, networks, cultures and policies. This model also recognizes the following:
Both explicit and implicit knowledge is recognized and must be made visible and woven into policy making and practice decisions
Relationships are mediated through a cycle of innovation and must be understood as a system
The degree of integration of policies, practices and processes within a system is what determines the degree of use of an idea or innovation.
The language of integration suggests there is some systems-level plan to take the diverse aspects within a set of activities and connect, coordinate and, to some degree, manage to ensure that knowledge is effectively used.
Talking innovation
What makes language such a critical key to understanding innovation ecologies is that the way in which we speak about something is an indication of what we believe about something and how we act. As the quote from psychologist Abraham Maslow suggests above, language can also be used to hide things.
One example of this is in the realm of social innovation, where ideas are meant to be generated through social means for social benefit. This process can be organized many different ways, but it is almost never exclusively top-down, expert-driven. Yet, when we look at the language used to discuss social innovation, we see terms like dissemination regularly used. Examples from research, practice and connecting the two to inform policy all illustrate that the language of one generation continues to be used as new ones dawn. This is to be expected as the changes in language of one generation never fully supplants that of previous generations — at least not initially. Because of that, we need to be careful about what we say and how we say it to ensure that our intentions are reflected in our practice and our language. Without conscious awareness of what we say and what those words mean there is a risk that our quest to create true innovation ecosystems, ones where innovation is truly systems-embedded and knowledge is integrated we unwittingly create expectations and practices rooted in other models.
If we wish to walk the walk of innovation at a systems level, we need to talk the talk.
Tips and Tricks
Organizational mindfulness is a key quality and practice that embeds reflective practice and sensemaking into the organization. By cultivating practices that regularly check-in and examine the language and actions of an organization in reference to its goals, processes and outcomes. A recent article by Vogus and Sutcliffe (2012) (PDF) provides some guidance on how this can be understood.
Develop your sensemaking capacity by introducing space at regular meetings that bring together actors from different areas within an organization or network to introduce ideas, insights and observations and process what these mean with respect to what’s happened, what is happening and where its taking the group.
Some key references include:
Best, A., Hiatt, R. A., & Norman, C. D. (2008). Knowledge integration: Conceptualizing communications in cancer control systems. Patient Education and Counseling, 71(3), 319–327. http://doi.org/10.1016/j.pec.2008.02.013
Best, A., Terpstra, J. L., Moor, G., Riley, B., Norman, C. D., & Glasgow, R. E. (2009). Building knowledge integration systems for evidence‐informed decisions. Journal of Health Organization and Management, 23(6), 627–641. http://doi.org/10.1108/14777260911001644
Vogus, T. J., & Sutcliffe, K. M. (2012). Organizational Mindfulness and Mindful Organizing: A Reconciliation and Path Forward. Academy of Management Learning & Education, 11(4), 722–735. http://doi.org/10.5465/amle.2011.0002C
Weick, K. E., Sutcliffe, K. M., & Obstfeld, D. (2005). Organizing and the Process of Sensemaking. Organization Science, 16(4), 409–421. http://doi.org/10.1287/orsc.1050.0133
*** If you’re interested in applying these principles to your organization and want assistance in designing a process to support that activity, contact Cense Research + Design.
Learning simulacrum, simulation or something else?
Learning styles, technology-driven teaching, and self-direction are all concepts that anyone interested in education should be familiar with, yet the foundations for their adoption into the classroom, lab or boardroom are more suspect than you might think. Today we look at the three urban legends of learning and what that might mean for education, innovation and beyond.
What kind of learner are you? Are you a visual learner perhaps, where you need information presented in a particular visual style to make sense of it? Maybe you need to problem-solve to learn because that’s the way you’ve been told is best for your education.
Perhaps you are a self-directed learner who is one that, when given the right encouragement and tools, will find your way through the muck to the answers and that others just need to get out of the way. With tools like the web and social media, you have the world’s knowledge at your disposal and have little need to be ‘taught’ that stuff, because its online.
And if you’re a digital native (PDF), this is all second nature to you because you’re able to use multiple technologies simultaneously to solve multiple problems together with ease if given the ability to do so. After all, you’ve had these tools your entire life.
An urban legend, urban myth, urban tale, or contemporary legend, is a form of modern folklore consisting of stories that may or may not have been believed by their tellers to be true.
The authors are quick to point out that there are differences in the way people approach material and prefer to learn, but they also illustrate that there is relatively little evidence to support much of the thinking that surrounds these practices, confusing learning preferences for learning outcomes. I’ve commented on this before, noting that too often learning is conflated with interest and enjoyment when they are different things and if we were really serious about it we might change the way we do a great deal many things in life.
In the paper, the authors debunk — or at least question — the evidence that supports the ‘legends’ of digital natives as a type of learner, the presence of specific learning styles and the need to customize learning to suit such styles of learning, and that of the lone self-educator. In each case, the authors present much evidence to challenge these ideas so as not to take them as truths, but hypotheses that have little support for them in practice.
Science and its inconvenient truths about learning
Science has a funny way of revealing truths that we may find uncomfortable or at least challenge our current orthodoxy.
This reminds me of a terrific quote from the movie Men in Black that illustrates the fragility of ideas in the presence and absence of evidence after one of the characters (played by Will Smith) uncovers that aliens were living on earth (in the film) and is consoled by his partner (played by Tommy Lee Jones) about what is known and unknown in the world:
Fifteen hundred years ago everybody knew the Earth was the center of the universe. Five hundred years ago, everybody knew the Earth was flat, and fifteen minutes ago, you knew that humans were alone on this planet. Imagine what you’ll know tomorrow.
One of the problems with learning is that there is a lot to learn and not all of it is the same in content, format and situational utility. Knowledge is not a ‘thing’ in the way that potatoes, shoes, patio furniture, orange juice, and pencils are things where you can have more or less of it and measure the increase, decrease and change in it over time. But we often treat it that way. Further, knowledge is also highly contextualized and combines elements that are stable, emergent, and transformative in new, complex arrangements simultaneously over time. It is a complex adaptive system.
Learning (in practice) resists simple truths.
It’s why we can be taught something over and again and not get it, while other things get picked up quickly within the same person even if the two ‘things’ seem alike. The conditions in which a person might learn are cultural (e.g., exposure to teaching styles at school, classroom designs, educational systems, availability and exposure to technology, life experiences, emphasis on reflective living/practice within society, time to reflect etc..) and psycho-social/biological (e.g., attention, intelligence, social proximity, literacy, cognitive capacity for information processing, ability to engage with others) so to reduce this complex phenomena to a series of statements about technology, preference and perception is highly problematic.
Science doesn’t have all the answers — far from it — but at least it can test out what is consistent and observable over time and build on that. In doing so, it exposes the responsibility we have as educators and learners.
With great power comes great responsibility…?
Underpinning the urban legends discussed by Kirschner and van Merriënboer and not discussed is the tendency for these legends to create a hands-off learning systems where workplaces, schools, and social systems are freed from the responsibility of shaping learning experiences and opportunities. It effectively reduces institutional knowledge, wisdom and experience to mere variables in a panoply of info-bites treated as all the same.
It also assumes that design doesn’t matter, which undermines the ability to create spaces and places that optimize learning options for people from diverse circumstances.
This mindset frees organizations from having to give time to learning, provide direction (i.e., do their own homework and set the conditions for effective learning and knowledge integration at the outset). It also frees us up from having to choose, to commit to certain ideas and theories, which means some form of discernment, priority setting, and strategy. That requires work up front and leadership and hard, critical, and time-consuming conversations about what is important, what we value in our work, and what we want to see.
When we assume everyone will just find their way we abdicate that responsibility.
Divesting resources and increasing distraction
In my home country of Canada, governments have been doing this with social investment for years where the federal government divests interest to the provinces who divest it to cities and towns who divest it to the public (and private) sector, which means our taxes never go up even if the demands on services do and we find that individual citizens are responsible for more of the process of generating collective benefit without the advantage of any scaled system to support resource allocation and deployment throughout society (which is why we have governments in the first place). It also means our services and supports — mostly — get smaller, lesser in quality, more spread thinly, and lose their impact because there isn’t the scaled allocation of resources to support them.
Learning is the same way. We divest our interests in it and before you know it, we learn less and do less with it because we haven’t the cultural capital, traditions or infrastructure to handle it. Universities turn campus life to an online experience. Secondary schools stop or reduce teaching physical education that involves actual physical activity. Scholarly research is reduced to a Google search. Books are given up as learning vehicles because they take too long to read. It goes on.
It’s not that there are no advantages to some of these ideas in some bites, but that we are transforming the entire enterprise with next to no sense of the systems they are operating in, the mission they are to accomplish, a theory of change that is backed up by evidence, or the will to generate the evidence needed to advise and the resources to engage in the sensemaking needed to evaluate that evidence.
Science, systems and learning
It is time to start some serious conversations about systems, science and learning. It would help if we started getting serious about what we mean when we speak of learning, what theories we use to underpin that language and what evidence we have (or need) to understand what those theories mean in practice and for policy. This starts by asking better questions — and lots of them — about learning and its role in our lives and work.
Design thinking and systems thinking are two thinking tools that can help us find and frame these issues. Mindfulness and its ethics associated with non-judgement, open-mindedness, compassion and curiosity are also key tools. The less we judge, the more open we are to asking good questions about what we are seeing that can lead us to getting better answers rather than getting trapped by urban legends.
Doing this within a systems thinking frame also allows us to see how what we learn and where and how we learn is interconnected to better spot areas of leverage and problems in our assumptions.
Developmental evaluation is focused on real-time decision making for programs operating in complex, changing conditions, which can tax the attentional capacity of program staff and evaluators. Organizational mindfulness is a means of paying attention to what matters and building the capacity across the organization to better filter signals from noise.
Mindfulness is a means of introducing quiet to noisy environments; the kind that are often the focus of developmental evaluations. Like the image above, mindfulness involves remaining calm and centered while everything else is growing, crumbling and (perhaps) disembodied from all that is around it.
Mindfulness in Organizations and Evaluation
Mindfulness is the disciplined practice of paying attention. Bishop and colleagues (2004 – PDF), working in the clinical context, developed a two-component definition of mindfulness that focuses on 1) self-regulation of attention that is maintained on the immediate experience to enable pattern recognition (enhanced metacognition) and 2) an orientation to experience that is committed to and maintains an attitude of curiosity and openness to the present moment.
Mindfulness does not exist independent of the past, rather it takes account of present actions in light of a path to the current context. As simple as it may sound, mindfulness is anything but easy, especially in complex settings with high levels of information sources. What this means for developmental evaluation is that there needs to be a method of capturing data relevant to the present moment, a sensemaking capacity to understand how that data fits within the overall context and system of the program, and a strategy for provoking curiosity about the data to shape innovation. Without attention, sensemaking or interest in exploring the data to innovate there is little likelihood that there will be much change, which is what design (the next step in DE) is all about.
Organizational mindfulness is a quality of social innovation that situates the organization’s activities within a larger strategic frame that developmental evaluation supports. A mindful organization is grounded in a set of beliefs that guide its actions as lived through practice. Without some guiding, grounded models for action an organization can go anywhere and the data collected from a developmental evaluation has little context as nearly anything can develop from that data, yet organizations don’t want anything. They want the solutions that are best optimized for the current context.
Mindfulness for Innovation in Systems
Karl Weick has observed that high-reliability organizations are the way they are because of a mindful orientation. Weick and Karen Sutcliffe explored the concept of organizational mindfulness in greater detail and made the connection to systems thinking, by emphasizing how a mindful orientation opens up the perceptual capabilities of an organization to see their systems differently. They describe a mindful orientation as one that redirects attention from the expected to the unexpected, challenges what is comfortable, consistent, desired and agreed to the areas that challenge all of that.
Weick and Sutcliffe suggest that organizational mindfulness has five core dimensions:
Reluctance to simplify
Sensitivity to operations
Commitment to resilience
Deference to expertise
Preoccupation with failure
Ray, Baker and Plowman (2011) looked at how these qualities were represented in U.S. business schools, finding that there was some evidence for their existence. However, this mindful orientation is still something novel and its overlap with innovation output, unverified. (This is also true for developmental evaluation itself with few published studies illustrating that the fundamentals of developmental evaluation are applied). Vogus and Sutcliffe (2012) took this further and encouraged more research and development in this area in part because of the lack of detailed study of how it works in practice, partly due to an absence of organizational commitment to discovery and change instead of just existing modes of thinking.
Among the principal reasons for a lack of evidence is that organizational mindfulness requires a substantive re-orientation towards developmental processes that include both evaluation and design. For all of the talk about learning organizations in industry, health, education and social services we see relatively few concrete examples of it in action. A mistake that many evaluators and program planners make is the assumption that the foundations for learning, attention and strategy are all in place before launching a developmental evaluation, which is very often not the case. Just as we do evaluability assessments to see if a program is ready for an evaluation we may wish to consider organizational mindfulness assessments to explore how ready an organization is to engage in a true developmental evaluation.
Cultivating curiosity
What Weick and Sutcliffe’s five-factor model on organizational mindfulness misses is the second part of the definition of mindfulness introduced at the beginning of this post; the part about curiosity. And while Weick and Sutcliffe speak about the challenging of assumptions in organizational mindfulness, these challenges aren’t well reflected in the model.
Curiosity is a fundamental quality of mindfulness that is often overlooked (not just in organizational contexts). Arthur Zajonc, a physicist, educator and President of the Mind and Life Institute, writes and speaks about contemplative inquiry as a process of employing mindfulness for discovery about the world around us.Zajonc is a scientist and is motivated partly by a love and curiosity of both the inner and outer worlds we inhabit. His mindset — reflective of contemplative inquiry itself — is about an open and focused attention simultaneously.
Openness to new information and experience is one part, while the focus comes from experience and the need to draw in information to clarify intention and actions is the second. These are the same kind of patterns of movement that we see in complex systems (see the stitch image below) and is captured in the sensing-divergent-convergent model of design that is evident in the CENSE Research + Design Innovation arrow model below that.
Stitch of Complexity
CENSE Corkscrew Innovation Discovery Arrow
By being better attuned to the systems (big and small) around us and curiously asking questions about it, we may find that the assumptions we hold are untrue or incomplete. By contemplating fully the moment-by-moment experience of our systems, patterns emerge that are often too weak to notice, but that may drive behaviour in a complex system. This emergence of weak signals is often what shifts systems.
For organizations and evaluators the next step is determining whether or not they are willing (and capable) of doing something with the findings from this discovery and learning from a developmental evaluation, which will be covered in the next post in this series that looks at design.
References and Further Reading:
Bishop, S. R., Lau, M., Shapiro, S., & Carlson, L. (2004). Mindfulness: A Proposed Operational Definition. Clinical Psychology: Science and Practice, 11(N3), 230–241.
Ray, J. L., Baker, L. T., & Plowman, D. A. (2011). Organizational mindfulness in business schools. Academy of Management Learning & Education, 10(2), 188–203.
Vogus, T. J., & Sutcliffe, K. M. (2012). Organizational Mindfulness and Mindful Organizing : A Reconciliation and Path Forward. Academy of Management Learning & Education, 11(4), 722–735.
Weick, K. E., Sutcliffe, K. M., Obstfeld, D., & Wieck, K. E. (1999). Organizing for high reliability: processes of collective mindfulness. In R. S. Sutton & B. M. Staw (Eds.), Research in Organizational Behavior (Vol. 1, pp. 81–123). Stanford, CA: Jai Press.
Weick, K.E. & Sutcliffe, K.M. (2007). Managing the unexpected. San Francisco, CA: Jossey-Bass.
Zajonc, A. (2009). Meditation as contemplative inquiry: When knowing becomes love. Barrington, MA: Lindisfarne Books.
