Category: education & learning

Education and learning

complexityeducation & learningevaluationsystems thinking

Developmental Evaluation: Questions and Qualities

Same thing, different colour or different thing?

Same thing, different colour or different thing?

Developmental evaluation, a form of real-time evaluation focused on innovation and complexity, is gaining interest and attention with funders, program developers, and social innovators. Yet, it’s popularity is revealing fundamental misunderstandings and misuse of the term that, if left unquestioned, may threaten the advancement of this important approach as a tool to support innovation and resilience. 

If you are operating in the social service, health promotion or innovation space it is quite possible that you’ve been hearing about developmental evaluation, an emerging approach to evaluation that is suited for programs operating in highly complex, dynamic conditions.

Developmental evaluation (DE) is an exciting advancement in evaluative and program design thinking because it links those two activities together and creates an ongoing conversation about innovation in real time to facilitate strategic learning about what programs do and how they can evolve wisely. Because it is rooted in both traditional program evaluation theory and methods as well as complexity science it takes a realist approach to evaluation making it fit with the thorny, complex, real-world situations that many programs find themselves inhabiting.

I ought to be excited at seeing DE brought up so often, yet I am often not. Why?

Building a better brand for developmental evaluation?

Alas, with rare exception, when I hear someone speak about the developmental evaluation they are involved in I fail to hear any of the indicator terms one would expect from such an evaluation. These include terms like:

  • Program adaptation
  • Complexity concepts like emergence, attractors, self-organization, boundaries,
  • Strategic learning
  • Surprise!
  • Co-development and design
  • Dialogue
  • System dynamics
  • Flexibility

DE is following the well-worn path laid by terms like systems thinking, which is getting less useful every day as it starts being referred as any mode of thought that focuses on the bigger context of a program (the system (?) — whatever that is, it’s never elaborated on) even if there is no structure, discipline, method or focus to that thinking that one would expect from true systems thinking. In other words, its thinking about a system without the effort of real systems thinking. Still, people see themselves as systems thinkers as a result.

I hear the term DE being used more frequently in this cavalier manner that I suspect reflects aspiration rather than reality.

This aspiration is likely about wanting to be seen (by themselves and others) as innovative, as adaptive, and participative and as being a true learning organization. DE has the potential to support all of this, but to accomplish these things requires an enormous amount of commitment. It is not for the faint of heart, the rigid and inflexible, the traditionalists, or those who have little tolerance for risk.

Doing DE requires that you set up a system for collecting, sharing, sensemaking, and designing-with data. It means being willing to — and competent enough to know how to — adapt your evaluation design and your programs themselves in measured, appropriate ways.

DE is about discipline, not precision. Too often, I see quests to get a beautiful, elegant design to fit the ‘social messes‘ that represent the programs under evaluation only to do what Russell Ackoff calls “the wrong things, righter” because they apply a standard, rigid method to a slippery, complex problem.

Maybe we need to build a better brand for DE.

Much ado about something

Why does this fuss about the way people use the term DE matter? Is this not some academic rant based on a sense of ‘preciousness’ of a term? Who cares what we call it?

This matters because the programs that use and can benefit from DE matter. If its just gathering some loose data, slapping it together and saying its an evaluation and knowing that nothing will ever be done with it, then maybe its OK (actually, that’s not OK either — but let’s pretend here for the sake of the point). When real program decisions are made, jobs are kept or lost, communities are strengthened or weakened, and the energy and creative talents of those involved is put to the test because of evaluation and its products, the details matter a great deal.

If DE promises a means to critically, mindfully and thoroughly support learning and innovation than it needs to keep that promise. But that promise can only be kept if what we call DE is not something else.

That ‘something else’ is often a form of utilization-focused evaluation, or maybe participatory evaluation or it might simply be a traditional evaluation model dressed up with words like ‘complexity’ and ‘innovation’ that have no real meaning. (When was the last time you heard someone openly question what someone meant by those terms?)

We take such terms as given and for granted and make enormous assumptions about what they mean that are not always supported). There is nothing wrong with any of these methods if they are appropriate, but too often I see mis-matches between the problem and the evaluative thinking and practice tools used to address them. DE is new, sexy and a sure sign of innovation to some, which is why it is often picked.

Yet, it’s like saying “I need a 3-D printer” when you’re looking to fix a pipe on your sink instead of a wrench, because that’s the latest tool innovation and wrenches are “last year’s” tool. It makes no sense. Yet, it’s done all the time.

Qualities and qualifications

There is something alluring about the mysterious. Innovation, design and systems thinking all have elements of mystery to them, which allows for obfuscation, confusion and well-intentioned errors in judgement depending on who and what is being discussed in relation to those terms.

I’ve started seeing recent university graduates claiming to be developmental evaluators who have almost no concept of complexity, service design, and have completed just a single course in program evaluation. I’m seeing traditional organizations recruit and hire for developmental evaluation without making any adjustments to their expectations, modes of operating, or timelines from the status quo and still expecting results that could only come from DE. It’s as I’ve written before and that Winston Churchill once said:

I am always ready to learn, but I don’t always like being taught

Many programs are not even primed to learn, let alone being taught.

So what should someone look for in DE and those who practice it? What are some questions those seeking DE support ask of themselves?

Of evaluators

  • What familiarity and experience do you have with complexity theory and science? What is your understanding of these domains?
  • What experience do you have with service design and design thinking?
  • What kind of evaluation methods and approaches have you used in the past? Are you comfortable with mixed-methods?
  • What is your understanding of the concepts of knowledge integration and sensemaking? And how have you supported others in using these concepts in your career?
  • What is your education, experience and professional qualifications in evaluation?
  • Do you have skills in group facilitation?
  • How open and willing are you to support learning, adapt, and change your own practice and evaluation designs to suit emerging patterns from the DE?

Of programs

  • Are you (we) prepared to alter our normal course of operations in support of the learning process that might emerge from a DE?
  • How comfortable are we with uncertainty? Unpredictability? Risk?
  • Are our timelines and boundaries we place on the DE flexible and negotiable?
  • What kind of experience do we have truly learning and are we prepared to create a culture around the evaluation that is open to learning? (This means tolerance of ambiguity, failure, surprise, and new perspectives?)
  • Do we have practices in place that allow us to be mindful and aware of what is going on regularly (as opposed to every 6-months to a year)?
  • How willing are we to work with the developmental evaluator to learn, adapt and design our programs?
  • Are our funders/partners/sponsors/stakeholders willing to come with us on our journey?

Of both evaluators and program stakeholders

  • Are we willing to be open about our fears, concerns, ideas and aspirations with ourselves and each other?
  • Are we willing to work through data that is potentially ambiguous, contradictory, confusing, time-sensitive, context-sensitive and incomplete in capturing the entire system?
  • Are we willing/able to bring others into the journey as we go?

DE is not a magic bullet, but it can be a very powerful ally to programs who are operating in domains of high complexity and require innovation to adapt, thrive and build resilience. It is an important job and a very formidable challenge with great potential benefits to those willing to dive into it competently. It is for these reasons that it is worth doing and doing well.

In order for us to get there this means taking DE seriously and the demands it puts on us, the requirements for all involved, and the need to be clear in our language lest we let the not-good-enough be the enemy of the great.

 

Photo credit: Highline Chairs by the author

behaviour changeeducation & learninginnovation

Isolation: The New Innovator’s Dilemma

It's can be a long, lonely climb

It’s can be a long, lonely climb

 Innovators transform the world around them in big and small ways and while a successful effort can be lauded by pundits, politicians and the public there is a long road to making change happen. That road is also a lonely one and doing things different means more than just innovating and experiencing what it means to be resilient firsthand. 

Clayton Christensen’s seminal book The Innovator’s Dilemma has been one of the leading sources of thinking-inspriation in business and social innovation. The book reflects the challenges with those seeking to introduce new ideas, products or services into established markets (or ecosystems) in the aim of addressing both people’s present and future needs.

These innovators — change-makers — risk disrupting the very markets they seek to influence bringing uncertainty for everyone. What innovators bet on is that the changes they introduce will have wide-ranging, positive benefits even if they don’t fully know what those are before setting out. Not surprisingly, these efforts are not always welcome at first and the road toward understanding and acceptance is a long one.

Innovation means doing something new and while we like to talk about new, many don’t actually like doing ‘new’ because that means questioning and changing things. Indeed, change — profound change — in thinking is often vigorously opposed as Albert Einstein pointed out in a quote that is paraphrased as:

Great spirits have always encountered violent opposition from mediocre minds

This opposition is a challenge for anyone, but the long slog towards innovation is not only hard on the spirit, it is often a lonely path.

The lonely lives of leaders

To innovate means to lead through ideas and products. We live in a society that admires and elevates the innovators. No better or perhaps inspiring example is the 1997 advertisement from Apple as part of the Think Different campaign in the 1990’s.

What is missing from the platitudes, plaudits and celebrations is the quiet, often lonely, life away from the attention that successful innovations bring (nevermind those that are not deemed successful). To innovate is to lead and to lead is often to be lonely by definition because there are few leading and more following. This leadership by thought or action is often what makes leaders appear creative, innovative and — as Seth Godin affectionately calls being weird. A study discussed in the Harvard Business Review and dissected in Forbes pointed to high rates of loneliness among those at the CEO level, which is among those who “made it”. Consider those who haven’t yet “made it”, who haven’t had their idea “succeed” or take off and it might feel even more lonely.

At a recent workshop I conducted a participant expressed publicly a sense of gratitude for simply having the opportunity to connect with others who were simply open to seeing the world in the same way that they were. In hosting a learning workshop for social innovators a positive byproduct was that attendees who might have been isolated in their activities and thinking in one context could come together in another.

Innovation, because it is new, means that innovators have few peers available to directly commiserate with and may need to find ways to connect on idea, method, philosophy or role, but rarely something direct. That requires extra work in the search and more effort to connect in the finding, which takes time and energy — two things innovators are often short of.

But that doesn’t diminish the value and importance of time and energy and directing it towards efforts to reduce isolation.

Creating deep community

Paul Born, Director of the Tamarack Institute for Community Engagement, recently published a book on creating deep community connections as a necessary means of fostering transformative change. Born offers four pillars to a deepening community are:  1) sharing stories, 2) taking the time to enjoy one another, 3) taking care of one another, and 4) working together towards a bigger social goal.

While there is little to argue with here, these pillars rest on the ability to locate, co-locate and create the space to share, enjoy, care and collaborate in the first place. For many innovators this is the hardest part. Where do we find the others like ourselves and how do begin to frame this journey?

There is a reason that innovators have flocked to tools like the Business Model Canvas and the Lean Startup method to help people define, refine and develop their products and mission. It’s easy to point to firms like Apple as examples of clear-focused innovators now, but 20 or 30 years ago it wasn’t so clear. Apple’s overall mission and vision are easy to see lived out in hindsight, not at the beginning. A read of Steve Jobs’ biography illustrates how often his way of approaching the world clashed with nearly everyone and everything and how difficult life was for him.

But Steve Jobs happened to be challenging the world in a place that would come to be known as Silicon Valley. For the last thirty years the San Francisco bay area has been a spark for creative thinking and innovation, one of many hotbeds of business and cultural transformation that Richard Florida documented as home of the Creative Class(es). But not all innovation takes place in these centres and even within such centres it might be hard to connect when an idea is ill-formed or new. We lose out when innovation is only done in certain places by certain people.

(Social) innovators are part of a diffuse and sometimes lost tribe.

Troubled language

If you look at the language that we frame innovation we reveal many of the problems with not only our ideas, but what we do with them. As mentioned in previous posts, we privilege terms like creativity, but often ignore craft. We aspire to be learners, but often don’t like real learning. We tout the role of failure in design and innovation, yet our overloaded cultural baggage attached to the term prevents us from really failing (or asking such tepid questions we don’t really stretch ourselves).

Having access to social media and electronic communities offer a lot and something we didn’t have before, but its very difficult to forge strong, connective bonds mediated through a technological interface. Technology is good at initiating superficial connections or maintaining deeper connections, but not so good at creating deep connections. Those deeper connections as Paul Born points out are the things that sustain us and allow us to do our best work.

The dilemma is how to allocate time and resources in cultivating uniqueness, depth and connecting to similar innovators when that pool is small or integrating more with those in the convention system. Of course innovators need to relate to both groups at some level because an innovation doesn’t grow if we only connect to ‘true believers’, but at different stages it matters how we’re allocating our time, energy and enthusiasm particularly along that journey up Mt. Isolation.

Options

There is no ready answer for this problem. Indeed, the lonely path to being different, weird or constructively challenge the harmful or less effective parts of the status quo may be one of the most wicked ones innovators face.

For those interested in social innovation there are a few examples for those who want to find peers and connect:

  • The Tamarack Institute for Community Engagement (mentioned earlier) has different communities of practice focused on various aspects of community building and social innovation. They host events and have created a vibrant community of learners and action-oriented professionals across Canada and the United States;
  • LinkedIn has a number of topical groups that have evolved on a variety of social and innovation topics that include local, global and topical foci;
  • The Social Innovation Generation Group convenes formal and informal events connecting those working in the social innovation space in the Greater Toronto Area and across Canada;
  • Meetups are self-organized gatherings on virtually every topic under the sun in communities across the globe. Check out and see if there is something near you;
  • In Toronto and New York City, the Centre for Social Innovation is a part co-working space, social action community, and venture incubation support group that connects and enlivens the work that social innovators do. They have many events (many are free and low cost) organized by their members that seek to bring people together and offer skill development. If you’re in Ottawa, check out The Hub. In Calgary? Check out EpicYYC ;  In Vancouver, visit the great folk at the HiVE. Throughout the United States Impact Hub spaces offer innovators options to work and connect and in Cambridge, MA there is the amazing Cambridge Innovation Centre for innovation more broadly. MaRS in Toronto offers another option.
  • Lastly, CENSE Research + Design hosts a series of webinars and free and paid workshops to create capacity for social innovation. For more information visit: www.cense.ca/learning .

References:

Born, P. (2014). Deepening Community: Finding Joy Together in Chaotic Times (p. 216). San Francisco, CA: Berrett-Koehler Publishers.

Wheatley, M. (2006). Leadership and the New Science: Discovering Order in a Chaotic World (3rd. ed., p. 218). San Francisco, CA: Berrett-Koehler Publishers.

Wheatley, M. (2007). Finding Our Way: Leadership for an Uncertain Time (p. 300). San Francisco, CA: Berrett-Koehler Publishers.

Wheatley, M. (2010). Perseverance (p. 168). San Francisco, CA: Berrett-Koehler Publishers.

Photo: Mt. Isolation This Way on Flickr by Tim Sackton used under Creative Commons License. (Thanks for the great shot Tim and making it available for others to use!)

behaviour changecomplexityeducation & learningpsychologysystems science

Asking Better and More Beautiful Questions

Why__by_WhiteSpeed

Beautiful answers require beautiful (and better) questions and Warren Berger’s new book looks at this very phenomenon of inquiry and asks: What does it mean to ask better questions and what does that mean for the answers we seek and receive?  

Warren Berger recently published  A More Beautiful Question, a book looking at something we take for granted and yet is the foundational building block for all great designs and innovations: the question.

Perhaps more specifically, Berger is looking at hundreds of questions as he delves into the process of questioning, the kind of questions that lead to provocative and insightful answers, and the habits of good questioning that make for sustained innovation over time.  Berger is well suited to this inquiry having penned the book Glimmer, which profiled designer Bruce Mau and explored the concept of design thinking in great detail.

Asking good questions is perhaps the (often unstated, missed and neglected) foundation of what design thinking is all about and seeing that design is the foundation of innovation it therefore means that questioning is at that foundation, too. This is important stuff.

Finding the right problem by asking better questions

A look at any bookstore, blog roll, or journal dealing with the topic of innovation and you’ll inevitably find the word “creativity” used a lot. Creativity — the act and process of creating things — is highly correlated with the questions that spur the creation in the first place. Education professor J.W. Getzels did some of the earliest research on creativity and questioning (which is interestly absent from Berger’s book) and found that those who took more time to find the best problem to solve – and thus, asked better and deeper questions of their world and subject matter — came up with more creative ideas than those who dove quickly into solving the problem as they initially saw it.

The simple take-away is: 

At the root of an answer is a question – J.W. Getzels

The better the question, the better the answer.

In complexity terms, the questions asked often create the path dependencies that entrench practices that come after it. So by asking better or ‘more beautiful’ questions and giving that attention we are not only doing ourselves a service, but are acting more ethical as well. This ethical foundation is what underlies mindfulness practice. Jon Kabat Zinn has written extensively on the importance of grounding oneself to ask better questions of the world, something that I’ve done through CENSE Research + Design in developing a mindful organization model.

In his 2004 presidential address to the Canadian Psychological Association Pat O’Neill looked at how sub-fields like community psychology changed the nature of how many “problems” in psychology were framed at the outset. Issues like poverty, drug addiction and unemployment were often (and still are in many domains) framed as personal, moral failings or just bad choices. By asking different questions of these problems, community psychologists were able to see how social policies, neighbourhood structures, social networks, and historical social exclusion — all systems issues — factor in to frame and constrain individual’s choices and risk behaviours. Suddenly, what had been framed as a personal problem, became a shared one that we all had at least some stake in.

It is this thinking that has led to greater awareness of how social change is inextricably linked to systems change and why we need to understand systems at the individual, organizational, community and societal level if we wish to address many of our social problems. Asking systems questions is asking different, sometimes more beautiful questions that get at the root of problems and inspire social innovation.

Finding the beautiful question

In his book, Berger finds that those best equipped to solve or at least address these big wicked questions in business, philanthropy and social innovation are those that ask ‘beautiful questions’ and do it often. Berger cites studies that have shown a clear relationship between success in leadership and a propensity to ask good questions. Asking good questions however takes time and the willingness to take time to question, think and question some more is another stand-out feature of these successful leaders.

It is why good questioning is also a leadership issue. Effective leaders often take the time needed to fully process the most important decisions to form what Gary Hamel and C.K. Pralahad refer to as strategic intent. Psychologist Daniel Goleman recently summarized the research linking mindfulness to focus and leadership, showing how leaders are able to better focus on what they do by being mindful. This mindful attention clears away much of the cognitive clutter to enable better question finding and asking.

Berger shows that finding the question requires some persistence. Good questioners are able to live with not having an answer or even the right question for a while. They have great patience. That ability to stand back and think, see, reflect and think some more while prototyping questions is what separates those who ask the better questions from those who don’t.

Creative collisions also helps. By mixing up ideas and connections with others, good questioners give themselves the raw material to work with. However, many of the best questioners that Berger spoke to also advocated for the need for some solitude and time to process these ideas and questions on their own. This mix of collaboration, collision, and independence is a key factor in developing the beautiful idea.

Designing better question-making

What jumped out at me in this book was how little support most organizations offer themselves for asking better, beautiful questions. Berger noted that the need for ‘serial mastery’ and constant learning is a staple of the new work environment, which should lend itself to question asking. However, if organizations are unwilling or unable to provide time for reflection, training, knowledge integration and ongoing discovery through better questions how likely is it that the workforce is going to respond to this need for new skills?

Are organizations willing to invest in a culture of inquiry? Are organizations able to make the leap from knowing things to asking things? How many public sector, non-profit, social and health service organizations (let alone industry groups) would be willing to follow companies like Google who create space — literally and figuratively — for questioning? These are some of the questions I asked myself as I read Berger’s book.

These are design questions. Berger notes how Google’s founders, Larry Page and Sergey Brin, were both Montessori school graduates. The Montessori system of education is based on question asking and Google is run as an organization largely framed around questions (and queries as noted by the very notion of “googling” something). Google has been designed to support better questions in its literal architecture of its software, its hardware, its office space, and the ‘20 per cent time‘ they offer employees to explore questions they have and projects that are of personal importance to them.

True to the idea of questions being worthy of paying attention to, Warren Berger’s book is filled with them including some answers. I liked the book and believe that he has tapped into something very big. Whether or not organizations and leaders will be inspired to ask better questions from this or simply try to find better answers in the processes they have is perhaps the big question next.

On a related note, March 14th has been dubbed Question Day by Berger and his colleagues at the Right Question Institute, a non-profit organization that provides support for teachers and students to ask better questions in school as a foundation for a lifetime of learning. 

References:

Berger, W. (2009). Glimmer: How Design Can Transform Your Life, Your Business, and Maybe Even the World. Toronto, ON: Random House Canada.

Berger, W. (2014). A more beautiful question: The power of inquiry to spark breakthrough ideas. New York, N.Y.: Bloomsbury Publishing.

Chand, I., & Runco, M. A. (1993). Problem finding skills as components in the creative process. Personality and Individual Differences, 14(1), 155–162.

Getzels, J. W. (1979). Problem Finding: a Theoretical Note. Cognitive Science, 3(2), 167–172. doi:10.1207/s15516709cog0302_4

Getzels, J. W. (1980). Problem Finding and Human Thought. The Educational Forum, 44(2), 243–244.

Goleman, D. (2013). Focus: The Hidden Driver of Excellence. New York, N.Y.: Harper Collins.

O’Neill, P. (2005). The ethics of problem definition. Canadian Psychology, 46(13-22).

Photo credit: Why? by Whitespeed via DeviantArt

complexityeducation & learningemergenceevaluationsystems thinking

Developmental Evaluation and Mindfulness

Mindfulness in Motion

Mindfulness in Motion?

Developmental evaluation is focused on real-time decision making for programs operating in complex, changing conditions, which can tax the attentional capacity of program staff and evaluators. Organizational mindfulness is a means of paying attention to what matters and building the capacity across the organization to better filter signals from noise.

Mindfulness is a means of introducing quiet to noisy environments; the kind that are often the focus of developmental evaluations. Like the image above, mindfulness involves remaining calm and centered while everything else is growing, crumbling and (perhaps) disembodied from all that is around it.

Mindfulness in Organizations and Evaluation

Mindfulness is the disciplined practice of paying attention. Bishop and colleagues (2004 – PDF), working in the clinical context, developed a two-component definition of mindfulness that focuses on 1) self-regulation of attention that is maintained on the immediate experience to enable pattern recognition (enhanced metacognition) and 2) an orientation to experience that is committed to and maintains an attitude of curiosity and openness to the present moment.

Mindfulness does not exist independent of the past, rather it takes account of present actions in light of a path to the current context. As simple as it may sound, mindfulness is anything but easy, especially in complex settings with high levels of information sources. What this means for developmental evaluation is that there needs to be a method of capturing data relevant to the present moment, a sensemaking capacity to understand how that data fits within the overall context and system of the program, and a strategy for provoking curiosity about the data to shape innovation. Without attention, sensemaking or interest in exploring the data to innovate there is little likelihood that there will be much change, which is what design (the next step in DE) is all about.

Organizational mindfulness is a quality of social innovation that situates the organization’s activities within a larger strategic frame that developmental evaluation supports. A mindful organization is grounded in a set of beliefs that guide its actions as lived through practice. Without some guiding, grounded models for action an organization can go anywhere and the data collected from a developmental evaluation has little context as nearly anything can develop from that data, yet organizations don’t want anything. They want the solutions that are best optimized for the current context.

Mindfulness for Innovation in Systems

Karl Weick has observed that high-reliability organizations are the way they are because of a mindful orientation. Weick and Karen Sutcliffe explored the concept of organizational mindfulness in greater detail and made the connection to systems thinking, by emphasizing how a mindful orientation opens up the perceptual capabilities of an organization to see their systems differently. They describe a mindful orientation as one that redirects attention from the expected to the unexpected, challenges what is comfortable, consistent, desired and agreed to the areas that challenge all of that.

Weick and Sutcliffe suggest that organizational mindfulness has five core dimensions:

  1. Reluctance to simplify
  2. Sensitivity to operations
  3. Commitment to resilience
  4. Deference to expertise
  5. Preoccupation with failure

Ray, Baker and Plowman (2011) looked at how these qualities were represented in U.S. business schools, finding that there was some evidence for their existence. However, this mindful orientation is still something novel and its overlap with innovation output, unverified. (This is also true for developmental evaluation itself with few published studies illustrating that the fundamentals of developmental evaluation are applied). Vogus and Sutcliffe (2012) took this further and encouraged more research and development in this area in part because of the lack of detailed study of how it works in practice, partly due to an absence of organizational commitment to discovery and change instead of just existing modes of thinking. 

Among the principal reasons for a lack of evidence is that organizational mindfulness requires a substantive re-orientation towards developmental processes that include both evaluation and design. For all of the talk about learning organizations in industry, health, education and social services we see relatively few concrete examples of it in action. A mistake that many evaluators and program planners make is the assumption that the foundations for learning, attention and strategy are all in place before launching a developmental evaluation, which is very often not the case. Just as we do evaluability assessments to see if a program is ready for an evaluation we may wish to consider organizational mindfulness assessments to explore how ready an organization is to engage in a true developmental evaluation. 

Cultivating curiosity

What Weick and Sutcliffe’s five-factor model on organizational mindfulness misses is the second part of the definition of mindfulness introduced at the beginning of this post; the part about curiosity. And while Weick and Sutcliffe speak about the challenging of assumptions in organizational mindfulness, these challenges aren’t well reflected in the model.

Curiosity is a fundamental quality of mindfulness that is often overlooked (not just in organizational contexts). Arthur Zajonc, a physicist, educator and President of the Mind and Life Institute, writes and speaks about contemplative inquiry as a process of employing mindfulness for discovery about the world around us.Zajonc is a scientist and is motivated partly by a love and curiosity of both the inner and outer worlds we inhabit. His mindset — reflective of contemplative inquiry itself — is about an open and focused attention simultaneously.

Openness to new information and experience is one part, while the focus comes from experience and the need to draw in information to clarify intention and actions is the second. These are the same kind of patterns of movement that we see in complex systems (see the stitch image below) and is captured in the sensing-divergent-convergent model of design that is evident in the CENSE Research + Design Innovation arrow model below that.

Stitch of Complexity

Stitch of Complexity

CENSE Corkscrew Innovation Discovery Arrow

CENSE Corkscrew Innovation Discovery Arrow

By being better attuned to the systems (big and small) around us and curiously asking questions about it, we may find that the assumptions we hold are untrue or incomplete. By contemplating fully the moment-by-moment experience of our systems, patterns emerge that are often too weak to notice, but that may drive behaviour in a complex system. This emergence of weak signals is often what shifts systems.

Sensemaking, which we discussed in a previous post in this series, is a means of taking this information and using it to understand the system and the implications of these signals.

For organizations and evaluators the next step is determining whether or not they are willing (and capable) of doing something with the findings from this discovery and learning from a developmental evaluation, which will be covered in the next post in this series that looks at design.

 

References and Further Reading: 

Bishop, S. R., Lau, M., Shapiro, S., & Carlson, L. (2004). Mindfulness: A Proposed Operational Definition. Clinical Psychology: Science and Practice, 11(N3), 230–241.

Ray, J. L., Baker, L. T., & Plowman, D. A. (2011). Organizational mindfulness in business schools. Academy of Management Learning & Education, 10(2), 188–203.

Vogus, T. J., & Sutcliffe, K. M. (2012). Organizational Mindfulness and Mindful Organizing : A Reconciliation and Path Forward. Academy of Management Learning & Education, 11(4), 722–735.

Weick, K. E., Sutcliffe, K. M., Obstfeld, D., & Wieck, K. E. (1999). Organizing for high reliability: processes of collective mindfulness. In R. S. Sutton & B. M. Staw (Eds.), Research in Organizational Behavior (Vol. 1, pp. 81–123). Stanford, CA: Jai Press.

Weick, K.E. & Sutcliffe, K.M. (2007). Managing the unexpected. San Francisco, CA: Jossey-Bass.

Zajonc, A. (2009). Meditation as contemplative inquiry: When knowing becomes love. Barrington, MA: Lindisfarne Books.

complexityeducation & learningevaluationsystems science

Evaluation, Evidence and Moving Beyond the Tyranny of ‘I Think’

I think, you think

The concrete evidence for ‘I think’

Good evidence provides a foundation for decision-making in programs that is dispassionate, comparable and open to debate and view, yet often it is ignored in favour of opinion. Practice-based evidence allows that expert opinion in, however the way to get it into the discussion is through the very means we use to generate traditional evidence. 

Picture a meeting of educators or health practitioners or social workers discussing a case or an issue about a program. Envision those people presenting evidence for a particular decision based on what they know and can find. If they are operating in a complex, dynamic field the chances are that the evidence will be incomplete, but there might be some. Once these studies and cases are presented, invariably the switch comes when someone says “I think...” and offers their opinion.

Incomplete evidence is not useful and complex systems require a very different use of evidence. As Ray Pawson illustrates in his most recent book, science in the realm of complexity requires a type of sensemaking and deliberations that differ greatly from extrapolating findings from simple or even complicated program data.

Practice-based evidence: The education case

Larry Green, a thought leader in health promotion, has been advocating to the health services community that if we want more evidence based practice we need more practice based evidence (video). Green argues that systems science has much to offer by drawing in connections between the program as a system and the systems that the program operates in. He further argues that we are not truly creating evidence-based programs without adding in the practice-based knowledge to the equation.

Yet, practice-based evidence can quickly devolve into “I think” statements that are opinion based on un-reflective bias, personal prejudice, convenience and lack of information. To illustrate, I need only consider a curriculum decision-making process at universities (and likely many primary and secondary schools too). Having been a part of training programs at different institutions — in-house degree programs and multi-centre networks between universities — I can say I’ve rarely seen evidence come into play in decisions about what to teach, how to teach, what is learned and what the purpose of the programs are, yet always see “I think”.

Part of the reason for this is that there is little useful data for designing programs. In post-secondary education we use remarkably crude metrics to assess student learning and progress. Most often, we use some imperfect time-series data like assignments that are aggregated together to form a grade. If this is undergraduate education, most likely we are using multiple-choice exams because they are easier to distribute and grade and use within the resource constraints. For graduate students, we still use exams, but perhaps we use papers as well. But rarely, do we have any process data to make decisions on.

Yet, we recruit students based on quotas and rarely look to where the intellectual and career ‘markets’ are going to set our programs. Instead, we use opinion. “I think we should be teaching X” or “I think we should teach X this way” with little evidence for why these decisions are made. It is remarkable how learning — whether formal or informal — in organizations is left without a clear sense of what the point is. Asking why we teach something, why people need to learn something, and what they are expected to do with that knowledge is a question that is well beyond a luxury. To get a sense of this absurdity, NYC principal Scott Conti’s talk at TEDX Dumbo is worth a watch. He points to the mis-match between what we seek to teach and what we expect from students in their lives.

Going small to go big

There is, as Green points out, a need for a systems approach to understanding the problem of taking these anecdotes and opinion and making them useful. Many of the issues with education have to do with resources and policy directions made at levels well beyond the classroom, which is why a systems approach to evaluation is important.Evaluation applied across the system using a systems approach that takes into account the structures and complexity in that system can be an enormous asset. But how does this work for teachers or anyone who operates at the front line of their profession?

Evaluation can provide the raw materials for discussion in a program. By systematically collecting data on the way we form decisions, design our programs, and make changes we create a layer of transparency and an opportunity to better integrate practice-based evidence more fully. The term “system” in evaluation or programming often invokes a sense of despair due to a perception of size. Yet, systems happen at multiple scales and a classroom and the teaching within it are systems.

One of the best ways to cultivate practice-based evidence is to design evaluations that take into account the way people learn and make decisions. It requires initially using a design-oriented approach to paying attention to how people operate in the classroom — both as teachers and learners. From there, we can match those activities to the goals of the classroom — the larger goals, the ones that ask “what’s the point of people being here” and also what the metrics for assessment are within the culture of the school. Next, consider ways to collect data on a smaller scale through things like reflective practice journals, recorded videos of teaching, observational notes, and markers of significance such as moments of insight, heightened emotion, or decisions.

By capturing the small decisions it is possible to generate practice-based evidence that goes beyond “I think”. It also allows others in. Rather than ideas being formed exclusively in one’s own head, we can illustrate where people’s knowledge comes from and permit greater learning from those around that person. Too often, the talents and tools of great leaders and teachers are accessible only in formal settings — like lectures or discussions — and not evident to others in the fire of everyday practice.

What are you doing to support evaluation at a small scale and allowing others to access your practice-based knowledge to create that practice-based evidence?

References:

Green, L. W. (2006). Public health asks of systems science: to advance our evidence-based practice, can you help us get more practice-based evidence? American Journal of Public Health, 96(3), 406–9.

Green, L. W. (2008). Making research relevant: if it is an evidence-based practice, where’s the practice-based evidence? Family practice, 25 Suppl 1(suppl_1), i20–4.

Pawson, R. (2013). The science of evaluation: A realist manifesto. London, UK: Sage Publications.

complexitydesign thinkingeducation & learningsystems thinking

Integrative Thinking And Empathy in Systems

Seeing What You're Reaching For And With

Seeing What You’re Reaching For And With

Award-winning Canadian author and University of Toronto professor David Gilmour came under social/media fire for comments made about his stance of only including male, middle-aged writers in his list of readings for his undergraduate English courses because that is the experience he resonates with most. Drawing on what you know is both wise and foolish when looking at it from the perspective of systems change and by looking within and beyond our own boundaries we can see how. 

Richard Katz knows what it is like to be an outsider and see the world from deep within and far from outside a culture. Katz, a former professor and elder with the First Nations University of Canada and Harvard-trained anthropologist, was one of the first non-native individuals to be welcomed into the lives of the Kalahari Ju|’hoansi peoples of central Africa. The Ju|’hoansi are known to Westerners as ‘the Bushmen‘ and were the ‘stars’ of the film The Gods Must Be Crazy. His journey and decades-long experience with these peoples are chronicled in two remarkable books on healing and culture.

Dr Katz worked closely with my undergraduate advisor and mentor, Dr. Mary Hampton, a remarkable community psychologist and her husband (and elder) Dr. Eber Hampton, and would occasionally come to meet and speak with us eager students and the healing communities in Regina, where I studied. In life, but particularly in working with affairs of the heart and soul (which is the stuff of healing and community), Katz would say:

Talk only of what you know

I didn’t fully understand the meaning of this when I heard it until much later in life. As one interested in the science as well as the art of healing I struggled to understand how we couldn’t speak of things unknown if we were seeking discovery — which is about making the unknown, known. Over time I came to ‘know’ more about what Katz meant:  that our perspective is one of many in a system and it is one, that if contemplated and welcomed with an open mind and heart, is valid and true while also being apart and unique. While we hold a stance those around you have their own perspective and stance that is both the same and different and in this lies the heart of healing.

Katz was trying to warn students and other researchers against the idea that we can just go into some place and ‘know it’ without being in it and that even in immersing ourselves in the worlds of others we are still but a traveller, just as they are in ours. He also suggested that we can’t know other systems without knowing our own (my words, not his).

It is the paradox that we can connect on a fundamental human level and still hold an independent, personal account. Being at one and apart at the same time. This is a hallmark feature of a complex system. It is also what makes integrative thinking and empathy so critical in such systems.

Knowing me, knowing you

This brings us back to professor Gilmour. Speaking to the online culture magazine Hazlitt, David Gilmour said that he doesn’t teach books written by women, just men. This has caused a predictable uproar in the social/ media (see Storify link below).

In an interview with the Toronto Star, Gilmour tried to clarify his comments:

“My only point is that I tend to teach people whose lives are close to my own,” said Gilmour, who has taught at the university for seven years. “I’m an old guy and I understand about old guys.”

On the surface, Gilmour is doing just what Dick Katz implored us all to do: speak of what you know. Gilmour knows ‘old guys’ (who are White and straight) and not women or other ‘groups’. He is being authentic and true to his experience.

What Gilmour is missing on this topic is the empathy that is so important in working with complexity. Teaching English to undergraduates might not be an obvious example of systems thinking and complexity, but it can be. As Gilmour points out, English is about a point of view, which is another way to say its about where you stand. The writing of the ‘old guys’ Gilmour includes in his courses are able telling a narrative from a point of view. That makes for good literature.

Yet, it is the reader’s ability to adopt, interpret, experience and critique the point of view of a story character that makes a literary work compelling. That is in large part about empathy. Great writers make empathy easy. By being empathic, we see a setting or context — a system — that might be unfamiliar to us in ways that seem familiar by bringing us momentarily into the world of the other. This familiarity allows us to draw on the experience we have in other settings and contexts and apply them to the new one.

To the degree this has harmony and congruence with the narrative being told is the measure of fit between data from one context to another.

This is what we do in systems work. For Gilmour, the complexity in his system comes not from his perspective, but that of his students. They are women, maybe GLBT, most certainly from other age and cultural groups and geographic contexts. Gilmour is asking his students to empathize with his ‘old guy’ narrative while forgetting that he can empathize with the narrative of someone who is Asian, queer, or speaks Catalan in drawing narratives that can be welcomed into the classroom without it being the perspective he’s most familiar with. Indeed, it is when we extend ourselves beyond the most familiar narratives to finding something resonant in other narratives that we learn, discover and innovate.

Integrative (Design) Thinking

Integrative thinking is a concept that Roger Martin, also from the University of Toronto, has made popular and integrated into the teaching at the Rotman School of Business. (Indeed, Rotman’s marketing material brands itself as providing “a new way to think”). This style of thinking, which Martin has written about extensively through his research on CEO decision making, has been closely linked with design thinking, which is also tied closely to thinking about systems. It is about holding different ideas together at the same time and building models of reality through the exploration of these opposable thoughts.

It is a vehicle for empathy to flow through connecting feelings and observations with thoughts and prototyping actions. This is ultimately what we do when we design for engagement in complex systems. We aim to place ourselves in the system we seek to influence, learn where we are in relation to the boundaries we see, set those boundaries (maintaining flexibility throughout) and then build mechanisms to get feedback and probe the culture we are a part of — organizationally, individually and so on — to enable us to take some action. This continues in an iterative manner throughout our engagement with the system.

Integrative thinking combined with empathy allows us to engage human systems we don’t fully know in a meaningful way that recognizes our limits — speaking to Katz’s point about ‘talking about what we know’ — while opening up possibilities for communion on issues of shared concern.

This means that we can know others, but also that we can only know them as ourselves. It also means that the systems change we seek in our social world is both an intensely personal journey and one that shares our common humanity, regardless of whether we are looking at shifting an organization, a community or a global culture.

Perhaps by taking a bigger view, professor Gilmour might find the same passion in literature that is from a different perspective and ultimately find how its also very much the same.

education & learningevaluationsystems thinkingUncategorized

Scaling Education: The Absurd Case of the MOOC

Theatre at the Temple of Apollo

Theatre at the Temple of Apollo

The Chronicle of Higher Education (online) recently reported results of a survey looking at faculty teaching on MOOC’s (massive open online course) and found much interest and expectation around this new format.

The survey, conducted by The Chronicle, attempted to reach every professor who has taught a MOOC. The online questionnaire was sent to 184 professors in late February, and 103 of them responded.

Hype around these new free online courses has grown louder and louder since a few professors at Stanford University drew hundreds of thousands of students to online computer-science courses in 2011. Since then MOOCs, which charge no tuition and are open to anybody with Internet access, have been touted by reformers as a way to transform higher education and expand college access. Many professors teaching MOOCs had a similarly positive outlook: Asked whether they believe MOOCs “are worth the hype,” 79 percent said yes.

The survey of professors was not scientific, particularly because it was of those who are already teaching MOOC’s, but it paints a picture of enthusiasm among those — many who were initially reticent about the potential of online education at that scale – engaged with the medium.

Global Potential vs Global Hype

NY Times columnist Thomas Friedman, never one to resist enthusiasm for global movements, speaks of the MOOC as a revolution. Friedman suggests how a MOOC-driven education system could potentially change foreign aid given its promise to do good things.

Anant Agarwal, the former director of M.I.T.’s artificial intelligence lab, is now president of edX, a nonprofit MOOC that M.I.T. and Harvard are jointly building. Agarwal told me that since May, some 155,000 students from around the world have taken edX’s first course: an M.I.T. intro class on circuits. “That is greater than the total number of M.I.T. alumni in its 150-year history,” he said.

Yes, only a small percentage complete all the work, and even they still tend to be from the middle and upper classes of their societies, but I am convinced that within five years these platforms will reach a much broader demographic. Imagine how this might change U.S. foreign aid. For relatively little money, the U.S. could rent space in an Egyptian village, install two dozen computers and high-speed satellite Internet access, hire a local teacher as a facilitator, and invite in any Egyptian who wanted to take online courses with the best professors in the world, subtitled in Arabic.

For Friedman and many of those writing on MOOCs, the potential for this new format to bring the world’s best higher education to anyone, anywhere, in any country is enormous. It is an example of taking an innovation to scale on perhaps the most extreme. With the click of a mouse the entire world can learn together easily.

Except, it is a lie.

Writing as a guest on the Worldwise blog as part of the Chronicle of Higher Education, professor of writing and rhetoric Ghanashyam Sharma  puts truth to the lie of the modern online education movement’s hype. In an article called A MOOC Delusion: Why Visions to Educate the World Are Absurd, Sharma illustrates the faulty thinking that underpins much of the MOOC enthusiasm for transforming global education. His critique is less about MOOC’s as an educational vehicle in itself, but the global, culture-free scale at which they seek to operate.

There is a dire need for some healthy skepticism among educators about the idea that MOOCs are a wonderful means to go global in order to do good. For our desire to educate the whole world from the convenience of our laptops to be translated into any meaningful effect, we need more research about how students learn in massive open online platforms, and a better understanding of how students from different academic, cultural, social, and national backgrounds fare in such spaces.

Education at Scale

Drawing on his own personal experience with teaching in different contexts, beginning with Nepal and then moving to the University of Louisville and now SUNY Stony Brook, Sharma points to the subtleties in cultural learning styles that did not translate from space to space. He speaks of enormous challenges and, fortunately, the opportunity to meet these with resources to aid him in adapting his teaching from context to context. In doing so, he points to the myth that the global classroom will be filled with people all learning the same way, from a compatible perspective, and using the same language. Even simple differences in the way he presents as a teacher can change the manner in which students learn – and those differences are rooted in culture.

As a teacher myself, I know much of what he writes. Even a ‘simple’ face-to-face graduate course presents considerable pedagogical challenges for me. As an instructor  in public health I need to consider things like:

  • Disciplinary background. Each discipline has communication ‘sub-cultures’ and traditions that differ. A student with a sociology background might be used to rhetoric while one from the basic sciences may be used to communicating through technical reports. Each discipline also uses language in different ways, with terms unique to that discipline.
  • Cultural backgrounds. Students from around the world will present differently in the way they approach the material, the presentation of arguments, and the level of participation in class. Even within a narrow band of cultural contexts that present within my classroom (usually there is perhaps 10-20 per cent of students are international) I am always amazed at the nuances that play out in the classroom. Race, gender and language all add additional complex layers that would require volumes to unpack here.
  • Personality and motivation. Students that are more reserved in class will experience each lesson differently than those who are outgoing. Personality and motivation change the type of discussions learners engage in and shape their interactions with other students. This is not to say that one personality style is better than the other, but whether you’re introverted or extraverted, confident, clear spoken or highly social shapes the classroom. Face-to-face encounters allows for some modulation of these effects to encourage participation with those of different needs.
  • Literacy. Whether it prose, numeracy, or discipline-specific language use, the literacy of students is tested when they have to take in material — whether through lecture, small groups, video, audio or text — and convey arguments.

These are constraints (and opportunities) in most modern universities, whether the course is delivered face-to-face or online. The amount of effort to engage a room full of eager learners is enormous if I attend to these various issues. As an educator, it is effort that I believe makes for a good experience for everyone and is a joyous part of the job. But I am speaking to a room of about 25 graduate students at one of the best universities in the world who are all present and sharing the same environmental context even if it is as a visiting student.

What happens when we are teaching to a ‘room’ of 150,000 people from all over the world?

Understanding scale

I do believe that we can learn much from online courses and that much can be conveyed via the MOOC that brings content to the world in ways that are appropriate and useful for a general audience. I am not against the MOOC, but I do think we need to carefully consider what it means to take education to scale and ask some deeper questions about what the learning experience is intended to achieve.

We need to design our educational experience using the same principles or design thinking we would apply to any other service of value.

The lie is in believing true education of a global audience in the rich way that a university or college intends is not only possible, but appropriate. What is being lost in the effort to make a common experience among a global classroom? Are we just sending out information or are we creating a learning environment? To what degree does the MOOC serve this purpose well. That is what a designer might start with as they move to asking how might we bring learning from one context to another.

Ghanashyam Sharma points to the folly in those who think the jump from culture to culture can be easily made when teaching the world at one time. The MOOC offers enormous potential to re-shape the way people learn and promoting access to content and expertise that many in this world could have only dreamed of years ago. But the naivete that such learning can be done in an monocultural way without losing something special about the context in which people learn and use what they learn might lead to some expensive lessons.

Sharma is right to ask for more research. Thomas Friedman is right to inspire us to think of creative ways to bring the wealth of the west’s educational treasures to the globe. The key is to figure out what parts of this global vision are real or possible, which are illusions, and which are delusions.

Unlike the auditorium at the Temple of Apollo at Delphi (pictured above) the audience for what we teach now are not from one place, time and shared cultural perspective; they represent the world. Without an understanding of what scale means in education we might be producing more ignorance than knowledge.

Photo: Cameron Norman