Posted on September 23, 2014
Evaluation is supposed to be driven by a program’s needs and activities, but that isn’t always the case. What happens when the need for numbers, metrics, ‘outcomes’ and data shape the very activities programs do and how that changes everything is something that is worth paying some attention to.
Since the Second World War we’ve seen a gradual shift towards what has been called presence of neo-liberal values across social institutions, companies, government and society. This approach to the world is characterized, among other things, by its focus on personal and economic efficiency, freedom, and policies that support actions that encourage both. At certain levels of analysis, these policies have rather obvious benefits.
Who wouldn’t like to have more choice, more freedom, more perceived control and derive more value from their products, services and outputs? Not many I suspect. Certainly not me.
Yet, when these practices move to different levels and systems they start to produce enormous complications that are at odds with — and produce distortions of — the very values that they espouse. We’ve seen the same happen with other value systems that have produced social situations that are highly beneficial in some contexts and oppressive and toxic in others – capitalism and socialism both fit this bill.
Invisible tails and wags
What makes ‘isms’ so powerful is that they can become so prevalent that their purpose, value and opportunity stop being questioned at all. It is here that the tail starts to wag the dog.
Take our economy (or THE economy as it is somewhat referred to). An economy is intended to be a facilitator and product of activities used to create certain types of value in a society. We work and produce goods (or ideas), exchange and trade them for different things, and these allow us to fulfill certain human goals. It can take various shapes, be regulated more or less, and can operate at multiple scales, but it is a human construction — we invented it. Sometimes this gets forgotten and in times when we use the economy to justify behaviour we forget that it is our behaviour that is the economy.
We see over and again with neoliberalism (which is among the most dominant societal ‘ism’ of the past 50 years in the West and more reflected globally all the time) taken at the broadest level, the economy becomes the central feature of our social systems rather than a byproduct of what we do as social beings. Thus, things like goods, experiences, relations and so on we used to consider as having some type of inherent value suddenly become transformed into objects that judgements can be made.
The role of systems
This can make sense where there are purpose-driven reasons to assign particular value scores to something, but the nature of value is tied to the systems that surround what is valued. If we are dealing with simple systems, those where there are clear cause-and-effect connections between the product or service under scrutiny and its ability to achieve its purpose, then valuation measurement makes sense. We can assert that X brand of laundry detergent is better than Y on the basis of Z. We can conduct experiments, trials and repeated measures that can compare across conditions.
It is also safe to make an assumption of value based on the product’s purpose that can be generalized. In other words, our reason for using the product is clear and relatively unambiguous (e.g., to clean clothes using the above example). There may be additional reasons for choosing X brand over Y, but most of those reasons can be also controlled for and understood discretely (e.g., scent, price, size, bottle shape etc..).
This kind of thinking breaks down in complex systems. And to make it even more complex, it breaks down imperfectly so we have simple systems interwoven within complex ones. We have humans using simple products and services that operate in new, innovative and complex conditions. Unfortunately, what comes with simple systems is simple thinking. Because they are — by their nature — simple, these system dynamics are easy to understand. Returning to our example of the economy, classical micro-economic models of supply and demand as illustrated below.
Relationships and the systems that surround them
Using this model, we can do a reasonable job of predicting influence, ascertaining value and hypothesizing relationships between both.
In complex systems, the value links are often in flux, dynamic, and relative requiring a form of adaptive evaluation like developmental evaluation. But that doesn’t happen as much as it should, mostly because of a failure to question the systems and their influence. Without questioning the values and value that systems create — the isms that were mentioned earlier — and their supposed connection to outcomes, we risk measuring things that have no clear connection to value and worse, we create systems that get designed around these ineffective measures.
What this manifests itself in is mindless bureaucracy, useless meetings, pompous and intelligible titles, and innovation-squashing regulations that get divorced from the purpose that they are meant to solve. And in doing so, this undermines the potential benefit that the original purpose of a bureaucracy (to document and create an organizational memory to guide decisions), meetings (to discuss and share ideas and solve problems), titles (to denote role and responsibility — although these aren’t nearly as useful as people think in the modern organization), and regulations (to provide a systems lens to constrain uncoordinated individual actions from creating systems problems like the Tragedy of the Commons).
More importantly, this line of thinking also focuses us on measuring the things that don’t count. And as often quoted and misquoted, the phrase that is apt is:
Not everything that counts can be counted, and not everything that can be counted counts.
Counting what counts
It is critical to be mindful of the purpose — or to reconnect, rediscover, reinvent and reflect upon the purposes we create lest we allow our work to be driven by isms. Evaluators and their program clients and partners need to stand back and ask themselves: What is the purpose of this system I am dealing with?
What do we measure and is that important enough to matter?
Perhaps the most useful way of thinking about this is to ask yourself: what is this system being hired to do?
Regular mindful check-ins as part of reflective practice at the individual, organizational and, where possible, systems level are a way to remind ourselves to check our values and practices and align and realign them with our goals. Just as a car’s wheels go out of alignment every so often and need re-balancing, so too do our systems.
In engaging in reflective practice and contemplating what we measure and what we mean by it we can better determine what part of what we do is the dog, what is the tail and what is being wagged and by whom.
Systems thinking is a class of theories, models and methods for understanding human and non-human interactions as seen as wholes instead of parts. This focus on interconnections and relationships is precisely what makes it challenging for many when it comes to systemically considering what systems thinking is all about and the implications of this are many. This post provides an introduction to certain ideas in systems thinking and points to what makes it different than other non-systems thinking approaches to understanding something.
Perhaps the most popular aphorism about systems thinking is the statement that the whole is greater than the sum of its parts, something borrowed from Gestalt Psychology. That statement is intended to reflect system thinking’s principal focus on the system itself rather than on the actors and actions within it.
It’s a subtle difference, but a meaningful one. For example, psychology might look at why individuals make choices and act and what implications come from those actions. Systems thinking seeks to look at the combined interaction of these interactions as a unified whole.
Fundamental to this way of seeing things is the concept of boundaries. Boundaries are essentially where the differences that make a difference lie. In a closed system, everything that makes a difference is clearly contained and observed within a relatively solid set of boundary conditions. Mechanical systems often function this way, making them simple or complicated in that they have the potential to be understood clearly in terms of causal connections and relations. These systems are more amenable to things like “best practices” where we can reasonably expect similar outcomes from consistent actions.
This kind of systems thinking is not as useful when applied to human systems, because they are mostly characterized as open systems. Open systems are those where the boundaries require some form of negotiation and may actually be in flux.
A general shorthand rule for setting boundaries in this kind of environment is this:
If you find yourself lost over and again in trying to understand where the influences and relationships within the system are, then you’ve probably bound your system too loosely. If you are finding too many influences laying outside of your boundaries, you’ve probably bound it too tightly.
Perspective: Where you sit
Systems are all about where you sit in relation to them. For instance, let’s take the example of family and some of the boundary questions one might ask in understanding this social entity as a system.
- Firstly, who is family? You could define family as blood relationships. But is that immediate blood relations? For example, If parents and children count, then how do we consider grandparents who are the parents of the parents? Do they count as family when you bound the system? Do great grandparents? Should we use genes and, if so, what level of genetic similarity do we share? Are we all family?
- Can family be defined socially? For example, if people become family by marriage and that marriage breaks down, does it influence the family system as you define it? What if that marriage ends via someone passing away? What if they are not married at all, but common law?
- What about the roles that people play? Does an “Uncle” or “Aunt” who are close, intimate friends of the family, but not of blood ties still get included in the family? How about a trusted lifelong neighbour who has been a part of someone’s life the entire time, but was never genealogically connected to anyone?
- Can our neighbourhood be part of the family?
One can make a case for any of these conditions. In defining a system there is no ‘good’ or ‘bad’ way to do it, just perspectives that are more or less useful and more or less attentive to specific details.
The answers to the questions about boundaries also depend on what the purpose of the system is in the first place. Purpose is the means by which we determine the differences and how they make a difference. You can imagine that one could potentially answer “yes” to almost every one of the questions asked above depending on where someone sits in the system and what kind of purpose they see in that system.
Part of thinking systemically about systems is defining the purpose of the system and ascertaining a perspective. That means being strategic about what you wish your systems thinking to support. It is here that much of the use of systems thinking I’ve witnessed breaks down. Organizations seeking to employ systems thinking often jump in without doing the pre-work needed to ground their perspective into some sense of purpose and perspective. This requires a mindful, honest accounting of the perspectives being brought into the discussion and connecting those to the strategic intent of your enterprise.
Being mindful of what one values, what one seeks to accomplish, and what kind of activities your organization engages in (or wants to engage in), and where the reach of your organization extends is a key starting position to thinking more systemically about systems.
In a quest for getting more, faster we pursue strategies that aim to compress and challenge the physics of time. Education is one of these areas where the quest to learn more, faster and ‘better’ may actually be taking us away from knowledge and speeding us to folly.
What would you say or do if your physician or attending nurse in the hospital told you that they attended a medical school that distilled all the key sources of knowledge into packages that allowed them to complete their training in half the time?
Would you be comfortable being treated by them?
What if you were seated on your next flight and learned that the pilot of your aircraft was taught by a flight school that claimed it could train pilots without the thousands of flight hours by focusing on the essence of what it meant to fly and do that really well in a short period of time?
Would you still want to fly with them?
What if someone said that they had a formula for taking Ericcson’s near mythical 10,000 hour rule* on building expertise and could halve it to produce the exact same results?
Would you believe them? And would you follow them?
Packaged learning and the myths of efficiency
While we might say no to these, we say yes to a lot of other things that are perhaps just as hard to believe. One of these is the myth of online education. Major online learning platforms (MOOC’s) like EdX, Coursera, and Udacity along with global education pioneers Khan Academy are delivering educational content to millions along with universities and thousands of smaller or independent education providers with the promise of offering distance education, some with degrees attached to them.
There is a place for this type of learning, but as often happens, the enthusiasm for speed, efficiency and profit blind and blur. Correspondence classes and the earliest online or distance learning programs were designed to meet the educational needs of those who were geographically isolated from others where face-to-face learning was impractical. What had a practical idea to solving a specific set of problem existing in a particular set of constraint conditions it is suddenly morphing into a standard for everyone and that isn’t a good idea.
Look around and you will see more ‘packaging’ educational experiences so that they can be scaled and delivered efficiently to different audiences. This might be fine if the content is simple and can be matched with the educator, the learning space (physical or online), and the cognitive and emotional demands placed on the learner in the process of learning the material. Yet, frequently this isn’t the case. Now, we see efforts to create programs to teach complex, important topics in a weekend, a week or a short retreat with the idea that we can just get to the essence of what’s needed and the rest will take care of itself.
Doing the work, putting in the time
No better example of this is hyper-learning myth is found that with Timothy Ferris, author of the 4-hour workweek and other rapid-fire learning books. Ferris takes his readers through his journeys to be hyper-efficient and learn things in a compressed time along the way.
One example is how he became a champion in a martial arts tournament in a sport he knew nothing about before engaging in mere weeks of training before the event. This achievement was done through some clever exploitation of tournament rules and engaging in a near obscene dehydration plan that enabled him to lose weight prior to weigh ins to allow him to fight below his normally expected weight class. This doesn’t change the outcome, but it adds a very big asterisk to its notation in the record books. Ferris’ work is filled with these sleight of hand kind of efficiencies that might work for a one-off, who’s longer term is questionable**.
Ferris has made a career out of intense, hyper-condensed learning and, even if he does what he claims, his approach to learning is his job and life. For most people, learning is one of a great many things they have on the go. Further, the problems they are trying to solve might not be ones that have a clear answer or a way to circumvent using a close read of the rules, rather they may be the kind of protracted, complex, thorny and wicked problems that we see in healthcare, social policy, environmental action, and organizational development. These are spaces where sleights of hand aren’t well received.
Other sleights of hand
In professional circles it is the longer-term that matters. System change, social innovation, healthcare transformation and community or organizational development are all areas where learning needs to start and continue throughout a long process. It often involves consideration of complex scenarios, an understanding of theory, reflective practice and experimentation that simply take time to not only engage with, but to contemplate.
It is like the parable about the farmer who wakes up one morning to find all of his crops dead because his unknowing son spent the night pulling up every stalk of grain with the belief that he could make them grow faster.
We have not been able to circumvent time, no matter what we wish.
The sleight of hand is in making busywork and information disguised as active learning and knowledge. There are certainly ways we can improve teaching, learning, knowledge translation and exchange and knowledge integration in its effectiveness, reach and impact, but we won’t be finding the ‘killer app’ that gives us the ability to download knowledge to our heads like the Matrix. These are developmental problems and thus ought to be treated using developmental thinking.
But we still try. Apps are being developed that allow us to learn anything, anywhere, in real time, from our phone or change our behaviour with a couple simple clicks, except there is virtually no evidence that we actually learn, actually change or do anything other than buy more and worry more.
True learning innovation will come from being wide-eyed about what we mean by learning, what we seek to achieve through it and creating the developmental thinking around what it means to bring them together rather than subscribing to legends or quick-fixes that simply don’t work.
* Anders Ericsson’s research on deliberative practice (PDF), which shows that attentive, intentional learning over time is a key determinant in high performing individuals. Malcolm Gladwell’s book Outliers highlights this work in detail and has led to the popularization of what has been colloquially referred to as ‘The 10,000 hour rule’, which reflects the approximate number of hours of deliberative practice required to gain expert-level skill and knowledge in a field.
**Many of Ferris’ claims from learning languages in a few weeks to mastering other subjects are unverified.
Image Credit: The Time Keeper /El guardián del tiempo by Jesus Solano via Flickr used under Creative Commons License. Thanks Jesus for sharing your wonderful art with the world through Creative Commons.
Learning styles, technology-driven teaching, and self-direction are all concepts that anyone interested in education should be familiar with, yet the foundations for their adoption into the classroom, lab or boardroom are more suspect than you might think. Today we look at the three urban legends of learning and what that might mean for education, innovation and beyond.
What kind of learner are you? Are you a visual learner perhaps, where you need information presented in a particular visual style to make sense of it? Maybe you need to problem-solve to learn because that’s the way you’ve been told is best for your education.
Perhaps you are a self-directed learner who is one that, when given the right encouragement and tools, will find your way through the muck to the answers and that others just need to get out of the way. With tools like the web and social media, you have the world’s knowledge at your disposal and have little need to be ‘taught’ that stuff, because its online.
And if you’re a digital native (PDF), this is all second nature to you because you’re able to use multiple technologies simultaneously to solve multiple problems together with ease if given the ability to do so. After all, you’ve had these tools your entire life.
An urban legend, urban myth, urban tale, or contemporary legend, is a form of modern folklore consisting of stories that may or may not have been believed by their tellers to be true.
The authors are quick to point out that there are differences in the way people approach material and prefer to learn, but they also illustrate that there is relatively little evidence to support much of the thinking that surrounds these practices, confusing learning preferences for learning outcomes. I’ve commented on this before, noting that too often learning is conflated with interest and enjoyment when they are different things and if we were really serious about it we might change the way we do a great deal many things in life.
In the paper, the authors debunk — or at least question — the evidence that supports the ‘legends’ of digital natives as a type of learner, the presence of specific learning styles and the need to customize learning to suit such styles of learning, and that of the lone self-educator. In each case, the authors present much evidence to challenge these ideas so as not to take them as truths, but hypotheses that have little support for them in practice.
Science and its inconvenient truths about learning
Science has a funny way of revealing truths that we may find uncomfortable or at least challenge our current orthodoxy.
This reminds me of a terrific quote from the movie Men in Black that illustrates the fragility of ideas in the presence and absence of evidence after one of the characters (played by Will Smith) uncovers that aliens were living on earth (in the film) and is consoled by his partner (played by Tommy Lee Jones) about what is known and unknown in the world:
Fifteen hundred years ago everybody knew the Earth was the center of the universe. Five hundred years ago, everybody knew the Earth was flat, and fifteen minutes ago, you knew that humans were alone on this planet. Imagine what you’ll know tomorrow.
One of the problems with learning is that there is a lot to learn and not all of it is the same in content, format and situational utility. Knowledge is not a ‘thing’ in the way that potatoes, shoes, patio furniture, orange juice, and pencils are things where you can have more or less of it and measure the increase, decrease and change in it over time. But we often treat it that way. Further, knowledge is also highly contextualized and combines elements that are stable, emergent, and transformative in new, complex arrangements simultaneously over time. It is a complex adaptive system.
Learning (in practice) resists simple truths.
It’s why we can be taught something over and again and not get it, while other things get picked up quickly within the same person even if the two ‘things’ seem alike. The conditions in which a person might learn are cultural (e.g., exposure to teaching styles at school, classroom designs, educational systems, availability and exposure to technology, life experiences, emphasis on reflective living/practice within society, time to reflect etc..) and psycho-social/biological (e.g., attention, intelligence, social proximity, literacy, cognitive capacity for information processing, ability to engage with others) so to reduce this complex phenomena to a series of statements about technology, preference and perception is highly problematic.
Science doesn’t have all the answers — far from it — but at least it can test out what is consistent and observable over time and build on that. In doing so, it exposes the responsibility we have as educators and learners.
With great power comes great responsibility…?
Underpinning the urban legends discussed by Kirschner and van Merriënboer and not discussed is the tendency for these legends to create a hands-off learning systems where workplaces, schools, and social systems are freed from the responsibility of shaping learning experiences and opportunities. It effectively reduces institutional knowledge, wisdom and experience to mere variables in a panoply of info-bites treated as all the same.
It also assumes that design doesn’t matter, which undermines the ability to create spaces and places that optimize learning options for people from diverse circumstances.
This mindset frees organizations from having to give time to learning, provide direction (i.e., do their own homework and set the conditions for effective learning and knowledge integration at the outset). It also frees us up from having to choose, to commit to certain ideas and theories, which means some form of discernment, priority setting, and strategy. That requires work up front and leadership and hard, critical, and time-consuming conversations about what is important, what we value in our work, and what we want to see.
When we assume everyone will just find their way we abdicate that responsibility.
Divesting resources and increasing distraction
In my home country of Canada, governments have been doing this with social investment for years where the federal government divests interest to the provinces who divest it to cities and towns who divest it to the public (and private) sector, which means our taxes never go up even if the demands on services do and we find that individual citizens are responsible for more of the process of generating collective benefit without the advantage of any scaled system to support resource allocation and deployment throughout society (which is why we have governments in the first place). It also means our services and supports — mostly — get smaller, lesser in quality, more spread thinly, and lose their impact because there isn’t the scaled allocation of resources to support them.
Learning is the same way. We divest our interests in it and before you know it, we learn less and do less with it because we haven’t the cultural capital, traditions or infrastructure to handle it. Universities turn campus life to an online experience. Secondary schools stop or reduce teaching physical education that involves actual physical activity. Scholarly research is reduced to a Google search. Books are given up as learning vehicles because they take too long to read. It goes on.
It’s not that there are no advantages to some of these ideas in some bites, but that we are transforming the entire enterprise with next to no sense of the systems they are operating in, the mission they are to accomplish, a theory of change that is backed up by evidence, or the will to generate the evidence needed to advise and the resources to engage in the sensemaking needed to evaluate that evidence.
Science, systems and learning
It is time to start some serious conversations about systems, science and learning. It would help if we started getting serious about what we mean when we speak of learning, what theories we use to underpin that language and what evidence we have (or need) to understand what those theories mean in practice and for policy. This starts by asking better questions – and lots of them — about learning and its role in our lives and work.
Design thinking and systems thinking are two thinking tools that can help us find and frame these issues. Mindfulness and its ethics associated with non-judgement, open-mindedness, compassion and curiosity are also key tools. The less we judge, the more open we are to asking good questions about what we are seeing that can lead us to getting better answers rather than getting trapped by urban legends.
Doing this within a systems thinking frame also allows us to see how what we learn and where and how we learn is interconnected to better spot areas of leverage and problems in our assumptions.
This might allow us to make many of our urban legends obsolete instead of allowing them to grow like the alligators that live in the sewers of New York City.