Category: education & learning

Education and learning

education & learningevaluation

Learning: The Innovators’ Guaranteed Outcome

on November 22, 2018

Innovation involves bringing something new into the world and that often means a lot of uncertainty with respect to outcomes. Learning is the one outcome that any innovation initiative can promise if the right conditions are put into place. 

Innovation — the act of doing something new to produce value — in human systems is wrought with complications from the standpoint of evaluation given that the outcomes are not always certain, the processes aren’t standardized (or even set), and the relationship between the two are often in an ongoing state of flux. And yet, evaluation is of enormous importance to innovators looking to maximize benefit, minimize harm, and seek solutions that can potentially scale beyond their local implementation. 

Non-profits and social innovators are particularly vexed by evaluation because there is an often unfair expectation that their products, services, and programs make a substantial change to social issues such as poverty, hunger, employment, chronic disease, and the environment (to name a few). These are issues that are large, complex, and for which no actor has complete ownership or control over, yet require some form of action, individually and collectively. 

What is an organization to do or expect? What can they promise to funders, partners, and their stakeholders? Apart from what might be behavioural or organizational outcomes, the one outcome that an innovator can guarantee — if they manage themselves right — is learning. 

Learning as an Outcome

For learning to take place, there need to be a few things included in any innovation plan. The first is that there needs to be some form of data capture of the activities that are undertaken in the design of the innovation. This is often the first hurdle that many organizations face because designers are notoriously bad at showing their work. Innovators (designers) need to capture what they do and what they produce along the way. This might include false starts, stops, ‘failures’, and half-successes, which are all part of the innovation process. Documenting what happens between idea and creation is critical.

Secondly, there needs to be some mechanism to attribute activities and actions to indicators of progress. Change only can be detected in relation to something else so, in the process of innovation, we need to be able to compare events, processes, activities, and products at different stages. Some of the selection of these indicators might be arbitrary at first, but as time moves along it becomes easier to know whether things like a stop or start are really just ‘pauses’ or whether they really are pivots or changes in direction. 

Learning as organization

Andrew Taylor and Ben Liadsky from Taylor Newberry Consulting recently wrote a great piece on the American Evaluation Association’s AEA 365 blog outlining a simple approach to asking questions about learning outcomes. Writing about their experience working with non-profits and grantmakers, they comment on how evaluation and learning require creating a culture that supports the two in tandem:

Given that organizational culture is the soil into which evaluators hope to plant seeds, it may be important for us to develop a deeper understanding of how learning culture works and what can be done to cultivate it.

What Andrew and Ben speak of is the need to create the environment for which learning can occur at the start. Some of that is stirred by asking some critical questions as they point out in their article. These include identifying whether there are goals for learning in the organization and what kind of time and resources are invested to regularly gathering people together to talk about the work that is done. This is the third big part of evaluating for learning: create the culture for it to thrive. 

Creating Consciousness

It’s often said that learning is a natural as breathing, but if that were true much more would be gained from innovation than there is. Just like breathing, learning can take place passively and can be manipulated or controlled. In both cases, there is a need to create a consciousness around what ‘lessons’ abound. 

Evaluation serves to make the unconscious, conscious. By paying attention — being mindful — of what is taking place and linking that to innovation at the level of the organization (not just the individual) evaluation can be a powerful tool to aid the process of taking new ideas forward. While we cannot always guarantee that a new idea will transform a problem into a solution, we can ensure that we learn in our effort to make change happen. 

The benefit of learning is that it can scale. Many innovations can’t, but learning is something that can readily be added to, built on, and transforms the learner. In many ways, learning is the ultimate outcome. So next time you look to undertake an innovation, make sure to evaluate it and build in the kind of questions that help ensure that, no matter what the risks are, you can assure yourself a positive outcome. 

Image Credit: Rachel on Unsplash

education & learningevaluation

The Quality Conundrum in Evaluation

on November 16, 2018

lex-sirikiat-469013-unsplash

One of the central pillars of evaluation is assessing the quality of something, often described as its merit. Along with worth (value) and significance (importance), assessing the merit of a program, product or service is one of the principal areas that evaluators focus their energy.

However, if you think that would be something that’s relatively simple to do, you would be wrong.

This was brought home clearly in a discussion I took part in as part of a session on quality and evaluation at the recent conference of the American Evaluation Association entitled: Who decides if it’s good? How? Balancing rigor, relevance, and power when measuring program quality. The conversation session was hosted by Madeline Brandt and Kim Leonard from the Oregon Community Foundation, who presented on some of their work in evaluating quality within the school system in that state.

In describing the context of their work in schools, I was struck by some of the situational variables that came into play such as high staff turnover (and a resulting shortage among those staff that remain) and the decision to operate some schools on a four-day workweek instead of five as a means of addressing shortfalls in funding. I’ve since learned that Oregon is not alone in adopting the 4-day school week; many states have begun experimenting with it to curb costs. The argument is, presumably, that schools can and must do more with less time.

This means that students are receiving up to one fifth less classroom time each week, yet expecting to perform at the same level as those with five days. What does that mean for quality? Like much of evaluation work, it all depends on the context.

Quality in context

The United States has a long history of standardized testing, which was instituted partly as a means of ensuring quality in education. The thinking was that, with such diversity in schools, school types, and populations there needed to be some means to compare the capabilities and achievement across these contexts. A standardized test was presumed to serve as a means of assessing these attributes by creating a benchmark (standard) to which student performance could be measured and compared.

While there is a certain logic to this, standardized testing has a series of flaws embedded in its core assumptions about how education works. For starters, it assumes a standard curriculum and model of instruction that is largely one-size-fits-all. Anyone who has been in a classroom knows this is simply not realistic or appropriate. Teachers may teach the same material, but the manner in which it is introduced and engaged with is meant to reflect the state of the classroom — it’s students, physical space, availability of materials, and place within the curriculum (among others).

If we put aside the ridiculous assumption that all students are alike in their ability and preparedness to learn each day for a minute and just focus on the classroom itself, we already see the problem with evaluating quality by looking back at the 4-day school week. Four-day weeks mean either that teachers are creating short-cuts in how they introduce subjects and are not teaching all of the material they have or they are teaching the same material in a compressed amount of time, giving students less opportunity to ask questions and engage with the content. This means the intervention (i.e., classroom instruction) is not consistent across settings and thus, how could one expect things like standardized tests to reflect a common attribute? What quality education means in this context is different than others.

And that’s just the variable of time. Consider the teachers themselves. If we have high staff turnover, it is likely an indicator that there are some fundamental problems with the job. It may be low pay, poor working conditions, unreasonable demands, insufficient support or recognition, or little opportunity for advancement to name a few. How motivated, supported, or prepared do you think these teachers are?

With all due respect to those teachers, they may be incompetent to facilitate high-quality education in this kind of classroom environment. By incompetent, I mean not being prepared to manage compressed schedules, lack of classroom resources, demands from standardized tests (and parents), high student-teacher ratios, individual student learning needs, plus fitting in the other social activities that teachers participate in around school such as clubs, sports, and the arts. Probably no teachers have the competency for that. Those teachers — at least the ones that don’t quit their job — do what they can with what they have.

Context in Quality

This situation then demands new thinking about what quality means in the context of teaching. Is a high-quality teaching performance one where teachers are better able to adapt, respond to the changes, and manage to simply get through the material without losing their students? It might be.

Exemplary teaching in the context of depleted or scarce resources (time, funding, materials, attention) might look far different than if conducted under conditions of plenty. The learning outcomes might also be considerably different, too. So the link between the quality of teaching and learning outcomes is highly dependent on many contextual variables that, if we fail to account for them, will misattribute causes and effects.

What does this mean for quality? Is it an objective standard or a negotiated, relative one? Can it be both?

This is the conundrum that we face when evaluating something like the education system and its outcomes. Are we ‘lowering the bar’ for our students and society by recognizing outstanding effort in the face of unreasonable constraints or showing quality can exist in even the most challenging of conditions? We risk accepting something that under many conditions is unacceptable with one definition and blaming others for outcomes they can’t possibly achieve with the other.

From the perspective of standardized tests, the entire system is flawed to the point where the measurement is designed to capture outcomes that schools aren’t equipped to generate (even if one assumes that standardized tests measure the ‘right’ things in the ‘right’ way, which is another argument for another day).

Speaking truth to power

This years’ AEA conference theme was speaking truth to power and this situation provides a strong illustration of that. While evaluators may not be able to resolve this conundrum, what they can do is illuminate the issue through their work. By drawing attention to the standards of quality, their application, and the conditions that are associated with their realization in practice, not just theory, evaluation can serve to point to areas where there are injustices, unreasonable demands, and areas for improvement.

Rather than assert blame or unfairly label something as good or bad, evaluation, when done with an eye to speaking truth to power, can play a role in fostering quality and promoting the kind of outcomes we desire, not just the ones we get. In this way, perhaps the real measure of quality is the degree to which our evaluations do this. That is a standard that, as a profession, we can live up to and that our clients — students, teachers, parents, and society — deserve.

Image credit:  Lex Sirikiat

complexityeducation & learningpsychologysystems thinking

Complex problems and social learning

on September 14, 2017

6023780563_12f559a73f_o.jpg

Adaptation, evolution, innovation, and growth all require that we gain new knowledge and apply it to our circumstances, or learn. While much focus in education is on how individuals attend, process and integrate information to create knowledge, it is the social part of learning that may best determine whether we simply add information to our brains or truly learn. 

Organizations are scrambling to convert what they do and what they are exposed to into tangible value. In other words: learn. A 2016 report from the Association for Talent Development (ATD) found that “organizations spent an average of $1,252 per employee on training and development initiatives in 2015”, which works out to an average cost per learning hour of $82 based on an average of 33 hours spent in training programs per year. Learning and innovation are expensive.

The massive marketplace for seminars, keynote addresses, TED talks, conferences, and workshops points to a deep pool of opportunities for exposure to content, yet when we look past these events to where and how such learning is converted into changes at the organizational level we see far fewer examples.

Instead of building more educational offerings like seminars, webinars, retreats, and courses, what might happen if they devoted resources to creating cultures for learning to take place? Consider how often you may have been sent off to some learning event, perhaps taken in some workshops or seen an engaging keynote speaker, been momentarily inspired and then returned home to find that yourself no better off in the long run. The reason is that you have no system — time, resources, organizational support, social opportunities in which to discuss and process the new information — and thus, turn a potential learning opportunity into neural ephemera.

Or consider how you may have read an article on something interesting, relevant and important to what you do, only to find that you have no avenue to apply or explore it further. Where do the ideas go? Do they get logged in your head with all the other content that you’re exposed to every day from various sources, lost?

Technical vs. Social

My colleague and friend John Wenger recently wrote about what we need to learn, stating that our quest for technical knowledge to serve as learning might be missing a bigger point: what we need at this moment. Wenger suggests shifting our focus from mere knowledge to capability and states:

What is the #1 capability we should be learning?  Answer: the one (or ones) that WE most need; right now in our lives, taking account of what we already know and know how to do and our current situations in life.

Wenger argues that, while technical knowledge is necessary to improve our work, it’s our personal capabilities that require attention to be sufficient for learning to take hold. These capabilities are always contingent as we humans exist in situated lives and thus our learning must further be applied to what we, in our situation, require. It’s not about what the best practice is in the abstract, but what is best for us, now, at this moment. The usual ‘stuff’ we are exposed to is decontextualized and presented to us without that sense of what our situation is.

The usual ‘stuff’ we are exposed to under the guise of learning is so often decontextualized and presented to us without that sense of how, whether, or why it matters to us in our present situation.

To illustrate, I teach a course on program evaluation for public health students. No matter how many examples, references, anecdotes, or colourful illustrations I provide them, most of my students struggle to integrate what they are exposed to into anything substantive from a practical standpoint. At least, not at first. Without the ability to apply what they are learning, expose the method to the realities of a client, colleague, or context’s situation, they are left abstracting from the classroom to a hypothetical situation.

But, as Mike Tyson said so truthfully and brutally: “Every fighter has a plan until they get punched in the mouth.”

In a reflection on that quote years later, Tyson elaborated saying:

“Everybody has a plan until they get hit. Then, like a rat, they stop in fear and freeze.”

Tyson’s quote applies to much more than boxing and complements Wenger’s assertions around learning for capability. If you develop a plan knowing that it will fail the moment you get hit (and you know you’re going to get hit), then you learn for the capability to adapt. You build on another quote attributed to Dwight D. Eisenhower, who said:

“I have always found that plans are useless but planning is indispensable.”

Better social, better learning

Plans don’t exist in a vacuum, which is why they don’t always turn out. While sometimes a failed plan is due to poor planning, it is more likely due to complexity when dealing with human systems. Complexity requires learning strategies that are different than those typically employed in so many educational settings: social connection.

When information is technical, it may be simple or complicated, but it has a degree of linearity to it where one can connect different pieces together through logic and persistence to arrive a particular set of knowledge outcomes. Thus, didactic classroom learning or many online course modules that require reading, viewing or listening to a lesson work well to this effect. However, human systems require attention to changing conditions that are created largely in social situations. Thus, learning itself requires some form of ‘social’ to fully integrate information and to know what information is worth attending to in the first place. This is the kind of capabilities that Wenger was talking about.

My capabilities within my context may look very much like that of my colleagues, but the kind of relationships I have with others, the experiences I bring and the way I scaffold what I’ve learned in the past with what I require in the present is going to be completely different. The better organizations can create the social contexts for people to explore this, learn together, verify what they learn and apply it the more likely they can reap far greater benefits from the investment of time and money they spend on education.

Design for learning, not just education

We need a means to support learning and support the intentional integration of what we learn into what we do: it fails in bad systems.

It also means getting serious about learning, meaning we need to invest in it from a social, leadership and financial standpoint. Most importantly, we need to emotionally invest in it. Emotional investment is the kind of attractor that motivates people to act. It’s why we often attend to the small, yet insignificant, ‘goals’ of every day like responding to email or attending meetings at the expense of larger, substantial, yet long-term goals.

As an organization, you need to set yourself up to support learning. This means creating and encouraging social connections, time to dialogue and explore ideas, the organizational space to integrate, share and test out lessons learned from things like conferences or workshops (even if they may not be as useful as first thought), and to structurally build moments of reflection and attention to ongoing data to serve as developmental lessons and feedback.

If learning is meant to take place at retreats, conferences or discrete events, you’re not learning for human systems. By designing systems that foster real learning focused on the needs and capabilities of those in that system, you’re more likely to reap the true benefit of education and grow accordingly. That is an enormous return on investment.

Learning requires a plan and one that recognizes you’re going to get punched in the mouth (and do just fine).

Can this be done for real? Yes, it can. For more information on how to create a true learning culture in your organization and what kind of data and strategy can support that, contact Cense and they’ll show you what’s possible. 

Image credit: Social by JD Hancock used under Creative Commons license.

behaviour changeeducation & learningsystems thinking

Learning fails in bad systems

on August 28, 2017

2348137226_2d6536745e_o_Edits.jpgEnormous energy is spent on developing strategies to accomplish things with comparatively little paid to the systems that they are being deployed in. A good strategy works by design and that means designing systems that improve the likelihood of their success rather than fight against them and this is no truer than in the effort to learn on the job.

 

A simple search of the literature — gray or academic — will find an enormous volume of resources on how to design, implement and support learning for action in organizations. At an individual level, there are countless* articles on personal change, self-improvement, and performance ‘hacks’ that individuals can do to better themselves and supposedly achieve more in what they do.

Psychology and related behavioural sciences have spent inordinate time learning how individuals and organizations change by emphasizing specific behaviours, decision processes, and data that can support action. A close inspection will find that relatively few strategies produce consistent results and this has to do less with execution, skill or topic and more with the system in which these strategies are introduced.

To illustrate this, consider the role of learning in the organization and how our strategies to promote it ultimately fail when our systems are not designed to support it.

Knowledge integration: A case study

Consider the example of attending a conference as a means of learning and integrating knowledge into practice.

Surajit Bhattacharya published a primer for how to get value from conferences in medicine, pointing to tips and strategies that a medical practitioner can take such as arriving a day early (so you’re not groggy), planning out your day, and be social. These are all practical, logical suggestions, yet they are premised upon a number of things that we might call system variables. These include:

  • The amount of control you have over your schedule week-to-week.
  • The availability of transportation and accommodation options that suit your schedule, budget, and preferences.
  • The nature and type of work you do, including the amount of hours and intensity of the work you perform in a typical week. This will determine the amount of energy you have and the readiness to be attentive.
  • The volume of email and other digital communications (e.g., messages and updates via social media, chat, project management platforms) you receive on a daily basis and the nature of those kinds of messages (e.g.urgency and importance).
  • The amount and nature of travel required to both attend the event and the amount you had prior to attending the event.
  • The level of rest you’ve had. Sleep amount, timing, and quality all factor into how much rest you get. Add in the opportunity to engage in an activity like walking, exercise or stretching that one might do and we see a number of factors that could influence learning performance.
  • The setting. The lighting, air quality and air flow, seat arrangement, room acoustics, and access to some natural light are all factors in our ability to attend to and engage with a learning event.
  • The quality and format of the content and its delivery. Speaker quality, preparation, content and overall performance will all contribute to the ability to convey information and engage the audience.
  • Food and drink. Are you eating the kinds of foods and beverages that enable your body’s performance? Do you have access to these foods and drinks? Are they served at times that suit your body?
  • Your level of comfort and skill at engaging strangers. This matters if you’re more introverted, dislike small talk, or are not energized by others.

These are all platform issues: those in which motivation and energy can be channeled to focus on and engage with learning content. The fewer of these factors present the greater the energy expenditure needed on the part of the learner.

Learning within systems

W. Edwards Deming noted that most of the issues of performance in any organization were due to processes and systems (estimated to be up to 85% or more) rather than individual employees. While Deming was referring largely to manufacturing contexts, the same might be said for learning.

Consider our example from earlier about the conference. We’ve already outlined the factors that could contribute to learning at the conference itself, but let’s extend the case further to what happens after the conference. After all, a surgeon, engineer, computer programmer, law clerk, or carpenter isn’t going to practice her or his craft at the conference; they’ll do it when they return to regular work.

Now consider what our attendee encounters after they have made the trip home to apply this newfound learning:

  • A backlog of emails, phone messages and other correspondence that has either been left untouched, scantly attended to, or fully managed. In the first case, the backlog might be high and requires a considerable amount of time and energy to ‘catch up’ on upon return, however at least the learner was fully present to perform the many activities suggested byBhattacharya in the earlier article. In the second case, there is a higher than usual amount to attend to and the learner might have been selectively disengaged from the learning event. In the third, the learner returns to usual life without a backlog but may have sacrificed considerable attention toward the usual correspondence than actually learning.
  • A backlog of meetings. Scheduled meetings, calls or other events that require a co-presence (virtual or physical) that were put off due to travel are now picked up.
  • A backlog of administrative tasks. Submitting receipts and conference expenses, regular accounting or administrative tasks are all things that either was left untouched or, in the case of submitting expenses, unlikely or impossible to do until the trip has returned.
  • Fatigue. Sitting in a conference can be exhausting, particularly because of the conditions of the rooms, the volume of content and the break in the routine of every day (which can be energizing, too). Add in any travel issues that might arise and there is a reasonable chance that a person is not in an optimal state to take what they have been exposed to and apply it.
  • The usual organization processes and structures. Are there are opportunities to reflect upon, discuss, and process what has been learned with others and spaces to apply those lessons directly with appropriate feedback? How often have we been exposed to inspiring or practical content only to find few opportunities to apply it in practice upon our return in enough time before the details of the lessons fade?

It’s not reasonable to expect to have optimal conditions in our work much of the time, if ever. However, as you can see there are a lot of factors that contribute to our capacity to learn and the required energy needed to take what we’ve been exposed to and integrate it into our work. The fewer of these situations in place, the greater the likelihood that the investment in the learning experience will be lost.

An organization or individual requires a platform for learning that includes systems that allow for learners to be at their best and to provide a means for them to take what they learn and apply it — if it’s valuable. Otherwise, why invest in it?

This isn’t to say that no good can come from a conference, but if the main focus is on actual learning and the application of knowledge to the betterment of an organization and individual why would we not invest in the platform to make use of that rather than discarding it.

Rethinking our systems

When I was doing evaluation work in continuing medical education I was amazed to see how often learning events were held at 7 or 8 am. The rationale was that this was often tied to shift changes at hospitals and were the one time of day when most physicians were least likely to have other appointments. This was also the time when physicians were either highly fatigued from a night shift or having battled traffic on their commute to work or were planning the rest of their day ahead — all circumstances when they might be least focused on actually learning.

This choice of time was done for scheduling purposes, not for learning purposes. Yet, the stated purpose of continuing education was to promote learning and its various outcomes. Here, the strategy was to expose medical professionals to necessary, quality content to keep them informed and skilled and doing it at a time that appeared most convenient for all is an example of an idea that had logic to it, but ultimately failed in most regards.

How? If one looked at the evaluation data, typically the results suggested this strategy wasn’t so bad. Most often post-event surveys suggested that the overall ratings were consistently high. Yet a closer look at the data yields some questions.

For example, the questions asked to assess impact were things like: did the presenter speak clearly? or did the presenter provide the content they said they would? In most cases, participants were asked if the speaker arrived on time, presented what they said they would, were intelligible and whether there was a chance the learner might find useful what was presented. It had little to no bearing on whether the content was appropriate, impactful or applied in practice. This is because the system for evaluation was based on a model of knowledge transmission: content is delivered to a person and, assuming the content is good, the lesson is learned.

We know this to be among the weakest forms of moving knowledge to action and certainly not something suited to more complex situations or conditions, particularly in health systems. This is still what prevails.

Design for learning

If you’re seeking to promote learning and create a culture where individuals across an organization can adapt, develop, and grow learning requires much more than simply sending people to conferences, hosting seminars, providing books and other materials or watching some instructional videos. Without a means to integrate and promote that new knowledge as part of a praxis, organizations and individuals alike will continue to get frustrated, lag in their efforts to anticipate and respond to changing conditions and will ultimately fail to achieve anything close to their potential.

Designing for learning is as much about a curriculum as the context for how that curriculum is delivered and how learners are set up to engage with it all in their organizations and everyday lives.

*This is literally the case because the volume of new articles being published daily is so high.

If you’re looking to create learning systems in your organization, visit Cense to explore what it can do for you in shaping your strategy and evaluation to support sustainable, impactful learning for complex conditions. 

Image credit: “Platform” by Martin L is licensed under CC BY 2.0

 

complexityeducation & learningevaluationinnovationknowledge translation

You Want It Darker?

on November 11, 2016

shutterstock_150465248

It is poetic irony on many levels that weeks after Leonard Cohen releases his album about the threat of death that he passes on, mere days after we saw the least poetic, most crass election campaign end in the United States with an equally dramatic outcome. This points to art, but also to the science of complexity and how we choose to approach this problem of understanding– and whether we do at all — will determine whether we choose to have things darker or not. 

A million candles burning for the love that never came
You want it darker
We kill the flame

Canadian-born and citizen-of-the-world poet, literary author, and songwriter Leonard Cohen passed away last night and the words above were part of his final musical contribution to the world. It is fitting that those words were penned at time not only when Cohen was ill and dying, but also as we’ve witnessed the flames of social progress, inclusion, and diversity fall ill.

Donald Trump is the president-elect of the United States, a fact that for many is not only unpalatable, but deeply troubling for what it represents. A Trump presidency and the social ills that have been linked to his campaign are just the latest sign that we are well into a strange, fear-ful, period of history within Western democracies. His was not a win for ideas, policy, but personality and as a vector for many other things that simply cannot be boiled down exclusively to racism, sexism, celebrity, or education — although all of those things played some part. It was about the complexity of it all and the ability for simplicity to serve as a (false) antidote.

No matter what side of the political spectrum you sit, it’s hard to envision someone less suited to the job of President of a diverse, powerful nation like the United States than Donald Trump using any standard measure of leadership, personality, experience, personal integrity or record of public conduct. Yet, he’s in and his election provides another signal that we are living in complex times and, like with Brexit, the polls got it very wrong.

We are seeing global trade shrink at a time when globalization is thought to be at its highest. We are witnessing high-profile acts of hatred, discrimination and abuse at at time when we have more means to be socially connected across contexts than ever before. We are lonely when the world and connection is at our fingertips.  It is a time of paradox and when we have so many means to cast light on the world, we seem to find new ways to kill the flame.

It is for this reason that those who deal with complexity and seek positive social change in the world need to take action lest things get darker.

Complexity just got real

The election of Donald Trump and the Brexit vote are two examples that should serve to wake-up anyone who seeks greater accounting of complexity in the making of social decisions.

This is not about voting for a Republican President or for citizens wanting greater control of Britain, it’s about understanding the premise of which those decisions were based on. The amount of cognitive dissonance required to assume that Donald Trump has the qualities befitting a leader of a country like the United States is truly astounding. And just like Brexit, the theories and models proposed post-event by the same people who predicted the opposite outcome pre-event will be just words, backed with too little understanding of complexity or why things actually happened.

Those who understand complexity know that these simplistic explanations are likely to be problematic. But that doesn’t make us better people, but it does mean we have certain responsibilities.

Complexity rhetoric vs science

For those who rely on complexity science as a means of understanding these kinds of events its now time to start matching the science to our rhetoric so we can back up the talk. In crude, but truth-speaking pop culture parlance: “This shit just got real“.

As complexity and systems thinking has gained attention in social science and policy studies we are seeing much more attention to the idea of complexity. Yet, the level of rhetoric on social complexity has overwhelmed any instances of evidence of how complexity actually is manifest, emergent, harnessed, or accounted for in practical means.

This isn’t to say that the tenets of complexity for understanding social systems aren’t true, but rather we don’t know that it’s true for sure and to what extent in what situations. I write this as a true-believer, but also as one who believes in science. Science is about challenging our beliefs and only if we cannot refute our theories through our best efforts can claim something is true. Thus, if we can’t show consistently how the principles of complexity are employed to make useful choices and inform the documentation of some of the outcomes related to our actions based on those choices, we are simply making fables not flourishing organizations, communities and societies.

Showing our work

Without something more than rhetoric to back our claims up we become no better than a politician claiming to make America great again because we’ve got great ideas and will be the greatest president ever because we have great ideas.

This is not about reverting to positivist science to understand the entire world, but about responsible practice in evaluation and research that allows us to document what we do and explore the consequences in context. Powered by complexity theory and the appropriate methods, we can do this. Yet, too often I hear reference to complexity theories in presentations, discussions and papers without any reference to how its been used in real terms (and not just extracted from some other realm of science like bee colonies, natural ecosystems and simulation models) to influence something of value beyond serving as an organizing framework.

Like little kids in math class: we need to show our work.

How did complexity manifest in practice in this case? What methods were used to systematically document the process? How does this fit / challenge the theories we know? These are questions that are what responsible scientists and evaluators ask of their subjects and its time to do this with complexity, regularly and often. No longer can we give it the relatively unchallenged ride it’s been given since first being introduced as a viable contributor to social theory about 20 years ago.

The reasons have to do with what happens when we stop trying to understand complex systems.

Evaluators and social sciences’ new moral imperative

As the US election was unfolding I became aware of some prescient, wise words that were uttered by former US Supreme Court Justice David Souter speaking at a town hall prior to the last election. His words were chilling to anyone paying attention to the world today. In the quote and interview (see link) he says on the matter of government and democracy:

What I worry about is that when problems are not addressed, people will not know who is responsible.

His words are not just about the United States or even politics alone. The further we get from understanding how our social, economic, political and environmental systems work the more we all become vulnerable to the kind of simplistic thinking that leads us to someone that embodies H.L. Mencken’s mis-paraphrased words*:

There is always an easy solution to every human problem — neat, plausible, and wrong

It is our duty as scientists and evaluators to show the world the work of the programs, policies and initiatives that are aimed at changing systems — no matter what that system is. We need to be better at telling the story of programs using data and communicating what we learn to the world. It’s our role to show the work of others and to let others see our work in the process. By doing so we can make a contribution to helping address what Justice Souter meant about people not knowing who is responsible.

And like Mencken’s message, our answer won’t be one that is all that neat, but we if we approach our work with the wisdom and knowledge of how systems work we can avoid Mencken’s trap and avoid presenting the complex as simple, but we will go further and illustrate what complexity means.

It is our moral duty to do this. For if not us, who?

People do understand complexity. Anyone with a child or garden knows that there is no ‘standard practice’ that applies to all kids or any years’ crop of vegetables all the time in all cases. It’s evident all around us. We have the tools, theories and models to help illuminate this in the world and a duty to test them and make this visible to help shed that light on how our increasingly complex world works. Without that we are at risk of demagogues and the darker forces of our nature taking hold.

We have the means for people to see light through the work of those who build programs, policies and communities to illuminate our world. In doing so we not only create the candles as Leonard Cohen speaks of, but the curiosity and love that keeps that flame burning. We can’t kill the flame.

And we could use some love right now.

Thanks Leonard for sharing your gifts with us. I hope your art inspires us to reflect on what world you left to better create a world we move to.

*Mencken’s original quote was: “Explanations exist; they have existed for all time; there is always a well-known solution to every human problem — neat, plausible, and wrong.” Alas, this doesn’t make as pithy, Powerpoint worthy comment. Despite the incorrectness of the paraphrased quote attributed to Mencken, it’s fair to say that in many organizations we see this as a true statement nonetheless.

Image Credit: Shutterstock, used under licence.

education & learningevaluation

Reflections said, not done

on May 27, 2016

27189389172_cb3f195829_k

 

Reflective practice is the cornerstone of developmental evaluation and organizational learning and yet is one of the least discussed (and poorly supported) aspects of these processes. It’s time to reflect a little on reflection itself. 

The term reflective practice was popularized by the work of Donald Schön in his book The Reflective Practitioner, although the concept of reflecting while doing things was discussed by Aristotle and serves as the foundation for what we now call praxis. Nonetheless, what made reflective practice as a formal term different from others was that it spoke to a deliberative process of reflection that was designed to meet specific developmental goals and capacities. While many professionals had been doing this, Schön created a framework for understanding how it could be done — and why it was important — in professional settings as a matter of enhancing learning and improving innovation potential.

From individual learners to learning organizations

As the book title suggests, the focus of Schön’s work was on the practitioner her/himself. By cultivating a focus, a mindset and a skill set in looking at practice-in-context Schön (and those that have built on his work) suggest that professionals can enhance their capacity to perform and learning as they go through a series of habits and regular practices by critically inquiring about their work as they work.

This approach has many similarities to mindfulness in action or organizational mindfulness, contemplative inquiry, and engaged scholarship among others. But, aside from organizational mindfulness, these aforementioned approaches are designed principally to support individuals learning about and reflecting about their work.

There’s little question that paying attention and reflecting on what is being done has value for someone seeking to improve the quality of their work and its potential impact, but it’s not enough, at least in practice (even if it does in theory). And the evidence can be found in the astonishing absence of examples of sustained change initiatives supported by reflective practice and, more particularly, developmental evaluation, which is an approach for bringing reflection to bear on the way we evolve programs over time. This is not a criticism of reflective practice or developmental evaluation per se, but the problems that many have in implementing it in a sustained manner. From professional experience, this comes down largely to the matter of what is required to actually do reflective practice or any in practice. 

For developmental evaluation it means connecting what it can do to what people actually will do.

Same theories, different practices

The flaw in all of this is that the implementation of developmental evaluation is often predicated on implicit assumptions about learning, how it’s done, who’s responsible for it, and what it’s intended to achieve. The review of the founding works of developmental evaluation (DE) by Patton and others point to practices and questions that that can support DE work.

While enormously useful, they make the (reasonable) assumption that organizations are in a position to adopt them. What is worth considering for any organization looking to build DE into their work is: are we really ready to reflect in action? Do we do it now? And if we don’t, what makes us think we’ll do it in the future? 

In my practice, I continually meet organizations that want to use DE, be innovative, become adaptive, learn more deeply from what they do and yet when we speak about what they currently do to support this in everyday practice few examples are presented. The reason is largely due to time and the priorities and organization of our practice in relation to time. Time — and its felt sense of scarcity for many of us — is one of the substantive limits and reflective practice requires time.

The other is space. Are there places for reflection on issues that matter that are accessible? These twin examples have been touched on in other posts, but they speak to the limits of DE in affecting change without the ability to build reflection into practice. Thus, the theory of DE is sound, but the practice of it is tied to the ability to use time and space to support the necessary reflection and sensemaking to make it work.

The architecture of reflection

If we are to derive the benefits from DE and innovate more fully, reflective practice is critical for without one we can’t have the other. This means designing in reflective space and time into our organizations ahead of undertaking a developmental evaluation. This invites questions about where and how we work in space (physical and virtual) and how we spend our time.

To architect reflection into our practice, consider some questions or areas of focus:

  • Are there spaces for quiet contemplation free of stimulation available to you? This might mean a screen-free environment, a quiet space and one that is away from traffic.
  • Is there organizational support for ‘unplugging’ in daily practice? This would mean turning off email, phones and other electronic devices’ notifications to support focused attention on something. And, within that space, are there encouragements to use that quiet time to focus on looking at and thinking about evaluation data and reflecting on it?
  • Are there spaces and times for these practices to be shared and done collectively or in small groups?
  • If we are not granting ourselves time to do this, what are we spending the time doing and does it add more value than what we can gain from learning?
  • Sometimes off-site trips and scheduled days away from an office are helpful by giving people other spaces to reflect and work.
  • Can you (will you?) build in — structurally — to scheduled work times and flows committed times to reflect-in-action and ensure that this is done at regular intervals, not periodic ones?
  • If our current spaces are insufficient to support reflection, are we prepared to redesign them or even move?

These are starting questions and hard ones to ask, but they can mean the difference between reflection in theory and reflection in practice which is the difference between innovating, adapting and thriving in practice, not just theory or aspiration.

 

 

 

 

education & learningknowledge translationpsychologysystems thinking

Bullying, the market for education and the damaged quest for learning

on September 11, 2014

Dark classroom, light minds

Dark classroom, light minds

A recent study found looked into the experience of cyberbullying by university professors at the hands of their students. This disturbing phenomenon points to much larger issues beyond mental health promotion and calls into question many of the assumptions we have about the systems we’ve designed to foster education and what it means to be a learner at university. 

The university is one of our oldest cultural institutions and its instructors are considered to have among societies most respected jobs, even if not always well compensated. In the past, students often approached their professors with a mixed sense of wonder, respect, curiosity and fear and that, in healthy situations, was reciprocated by faculty to create a space where people could explore ideas, learn, and challenge themselves and others to grow. That relationship has started to change as evidenced by the rise of cyberbullying in the classroom.

A recent article in Macleans Magazine looked at the changing state of the post-secondary classroom and the role of cyberbullying. Only this was not about student victims, but students as the perpetrators against their professors. The effects of cyberbullying are crippling and professors are bearing the burden of having hundreds of eyes watching them, writing about them and writing ‘consumer reviews’ about them in anonymous and sometimes unflattering, inflammatory and questionable terms on sites like RateMyProfessor.com .

Researchers at the University of California, Riverside found that as students age the incidence of face-to-face bullying decreases and cyberbullying increases, which might partly explain why we’re seeing this in university settings when face-to-face bullying goes subterranean. Yet, the notion that professors that are getting bullied by their students belies some other issues that require further investigation, namely those related to the nature of education and the role of students-as-consumers.

Consuming knowledge, producing expectations

If you pay for something, should you not expected to get something rather specific for that experience or product? Aside from some rare experiences of profane/profound personal challenge/punishment like Tough Mudder and its peers or dental work, there are few things we willingly pay for that we don’t derive pleasure from or achieve a very specific (anticipated) outcome.

Education is problematic because we might not know what we’ll get from it going in, what kind of experiences or ideas will emerge, and how our relationship to those experiences will change us. That is its great gift.

Many of us have had profound life changes because of something we experienced through our education and writing as one who has completed four different degree programs and a post-doc I can confidently say that I didn’t receive a lot of what I expected in any of those programs and I am a better person for it. Indeed, if I go to a specific learning event (aside from those focused on a specific technique or technology) I am disappointed if I actually come away with exactly what I expected.

That is part of the point. We don’t know what we don’t know.

But when you start viewing education as a thing that resembles any other market-driven product or services, you begin to focus on learning as a consumable good and your students as customers. In following this line of thought, it makes some sense to focus the delivery of this product on the desires of the consumer.

Increasingly, teachers (of various stripes) are being asked to consider a range of student-related variables in their education. Things like learning styles and preferences are now being woven into classroom instruction and students have come to learn to expect and are increasingly demanding to be taught in ways that match their unique learning preferences and styles. While there is reason to imagine that this approach is useful in stimulating engagement of students in the lessons, there is increasing evidence much of it does little to enhance actual learning. Many of the life lessons we’ve gained that shape what we do and who we are were not delivered in the manner of our choosing, conformed with our preferences and were not desired, expected or enjoyed in the moment. We risk confusing enjoyment with learning; they can be aligned but one isn’t necessary for the other to take place.

However, when we are viewing education from a consumer model, the specific outcomes become part of the contract. If I come to get a degree in X because I believe that the job market demands the skills and knowledge that X brings and I am paying tens of thousands of dollars and spending four or more years acquiring X then I feel entitled to expect all the benefits that X brings. Further, I expect that my journey to acquiring X will be enjoyable, because why would I spend more money than I’ve ever seen on something I don’t enjoy.

Particularly when that is money I don’t have.

A debt to pay

In Canada and the United States, student debt rates have dramatically increased. The Canadian Federation of Students note that Canadian’s attending post-secondary education now owe more than $15B to the Canadian federal government (PDF) as part of their student loan program, a number that doesn’t include debt accumulated from borrowing from banks, family, credit cards and other means. In Canada’s largest province, Ontario, the rate of graduate employment has decreased since 2001 and the overall youth unemployment rate continues to be the highest, despite the province having one of the most educated youth population in the country (and arguably, the world). And while Ontario universities continue to promote the fact that education is a better pathway to success, it is a hard pill for many students to swallow when many can’t apply what they trained for and paid for after they graduate.

Satirist John Oliver has an informative, humorous and distressing take on student debt and the state of consumer-oriented education for those who want to learn more.

None of these reasons are excuses for cyberbullying, but it does give a more complicated picture of those that might feel they are entitled to bully others and their reasoning behind it.

What we are seeing is a systems change in the way education is being produced, consumed and experienced. Even the mere fact that we can now reasonably use the language of consumerism to speak to something like education should give us pause and concern. I’ve been involved in post-secondary education for nearly 20 years and there has always been students who simply wanted the ‘piece of paper’ (degree) as a stepping stone to a job and little more than that from their time at school. They were willing to do the work — often the minimum possible — to graduate, but they knew they had to put the effort in to be successful. There was never an expectation that one was entitled to anything from going to school, although that might be changing.

Market identities and education systems

Belgian psychotherapist Paul Verhaeghe has explored the role of identity in market-based economies in his new book What About Me? In the book, Verhaeghe illustrates how we construct our identities as people drawing on the research that reflects (and often contradicts or obscures) the two major perspectives on personality and identity: the person-as-blank-slate and the person as a reflection of the environment. The former perspective assumes we come into the world as we are while the latter assumes the world makes us who we are and both have enormous amount of moral, cultural and evidentiary baggage attached to them.

What Verhaeghe does is point to the ways in which both have elements of truth to them, but that they are mediated by the manner in which we construct the very questions about who we are and what our purpose is. These questions are (for many cultural, historical, economic and political reasons that he elaborates on) frequently market-based. Thus, who we are is defined by what we do, what we own, what we produce, and how we use such things once out into the world and that the value that come with such ways of defining ourselves is considered self-evident. He makes a disturbing and convincing case when one stops to reflect on the way we think about how we think (metacognition + mindfulness) .

When viewed from the perspective of a market, knowledge and its products soon become the goal and not the journey. Indeed, I’ve even written about this in support of an argument for better research-to-action and knowledge translation. Much of the knowledge-to-action discourse is about viewing knowledge as a product even if the more progressive models also view this as part of a process and even more as part of a system. But it is the last part — the system — that we often give the shortest shrift to in our discussions. What Verhaeghe and others are doing is encouraging us to spend more time thinking about this and the potential outcomes that emerge from this line of thinking.

Unless we are willing to talk more about the systems we create to learn, explore and relate we will continue to support Verhaeghe’s thesis and uphold the conditions for the kind of education-as-a-product thinking that I suspect is contributing to students’ changing behaviour with their professors and creating a climate at universities that is toxic instead of inspiring.

Photo credit: Classroom by Esparta Palma used under Creative Commons License via Flickr. Check out Esparta’s remarkable work here.

1 2 3 … 13