Developmental evaluation (DE) is a problematic concept because it deals with a complex set of conditions and potential outcomes that differ from and challenge the orthodoxy in much of mainstream research and evaluation and makes it difficult to communicate. At a recent gathering of DE practitioners in Toronto, we were charged with coming up with an elevator pitch to describe DE to someone who wasn’t familiar with it; this is what I came up with.
Developmental evaluation is an approach to understanding the activities of a program operating in dynamic, novel environments with complex interactions. It focuses on innovation and strategic learning rather than standard outcomes and is as much a way of thinking about programs-in-context and the feedback they produce. The concept is an extension of Michael Quinn Patton’s original concept of Utilization Focused Evaluation with concepts gleaned from complexity science to account for the dynamism and novelty. While Utilization Focused Evaluation has a series of steps to follow (PDF), Developmental Evaluation is less prescriptive, which is both its strength and its challenge for describing it to people (things I’ve discussed in earlier posts).
So with that in mind, our group was charged with coming up with a way to explain DE to someone who is not familiar with it using anything we’d like — song, poetry, dance, slides, stories and beyond. While my colleague Dan chose to lead us all in song, I opted to go with a simple analogy by comparing DE to a hybrid of Trip Advisor and the classic Road Trip (due to lack of good vocalizing skills).
Trip Advisor has emerged as one of the most popular tools for travellers seeking advice on everything from hotel rooms to airlines to resorts and all the destinations along the way. Trip Advisor is averaging more than 13 million unique visitors per month and, unlike its competitors, focuses on user-generated content to support its service. Thus, your fellow travellers are the source of the recommendations not some professional travel agent or journalist. At its heart are stories of varies tones, detail and quality. People upload various accounts of their stay, chronicling even the most minute detail through photos, links to their blogs, video, and narrative. If you want to get the inside details on what a hotel is really like, check Trip Advisor and you’ll likely find it.
However, like any self-organizing set of ideas, the quality of the content will vary along with the level of reportage and the conclusions will be different depending on the context and experience of the person doing the reporting. For example, if you are a North American who is used to having even the most basic hotel chain offer a room with full-service linens, a bathroom, closet, desk and separate shower, you’ll have a hard time adjusting to something like EasyHotel in Europe.
The Road Trip part (capitalization intended here to denote something different than a regular trip by road), denotes the experience that comes from a journey with a desired destination, but not a pre-determined route and only a generalized timeline. A Road Trip is something that is more than just traveling from Point A to Point B, which is usually accomplished by taking the shortest route, the fastest route or a combination of the two; rather it is a journey. Movies like National Lampoon’s Vacation (and, European Vacation), Thelma and Louise, Harold & Kumar Go to White Castle, and (surprise!) Road Trip all capture this spirit to some effect. I suppose one might even find a more grim example of a Road Trip in the Lord of the Rings Trilogy or The Road.
“Road trips are fun when they are not planned point-to-point. As long as you have accommodation booked, that is enough. Its better not to have agendas; get as spontaneous and adventurous as you can. My friends and I went on a road trip to Goa last year. It was loads of fun as it was the first time we took off on our own without parents. To me, it was more than just a trip with friends. It showed that I could take care of myself and that I was now a grown-up, free to do what I wanted,” says Siddharth, who is doing his engineering.
Chance favours the prepared mind
“Road-tripping is a great way to bond with the people you are travelling with and I would strongly recommend it to people. It not only makes you appreciate yourself as an individual but is an amazing experience as you get to meet new people, know different cultures and sample different cuisines. I can never forget biking on sleet, riding though torrential rains, gobbling hot rotis at dhabas, the beautiful snow-capped mountains and guy talk with friends on the trip,” says Dheeraj, who recently went to Ladakh.
all is not hunky dory during these trips. You have to be way about accidents and mishaps. And, realise that freedom comes with responsibility. Says Arjun: “I had borrowed my friend’s bike for the trip, and though it looked good, it gave problems on the foothills of Kodaikanal and we couldn’t do the climb. Being a weekend, there were no mechanics. It helps to know your machine. A passion for road-tripping is not enough. You need to be equipped to take care of yourself also.”
Here, the story parallel is about being prepared. Know evaluation methods, know how to build and sustain relationships and to deal with conflict. A high tolerance for ambiguity and the flexibility to adapt is also important. Knowing a little about systems thinking and complexity doesn’t hurt either. Developmental evaluation is not healthy for those who need a high degree of predictability, are not flexible in their approach, and adhere to rigid timelines. Complex systems collapse under rigid boundary conditions and do evaluators working with such restrictions in developmental contexts.
So why do people do it? “Well, my memories of my favourite road trip were an injured leg, chocolates, beautiful photographs and a great sense of fulfilment,” recalls Arjun.
It is youngsters like these who have transformed road-tripping from just a hobby to an art.
After all, friendship and travel is a potent combination that you can’t say no to.
Clowning might seem either silly or scary to some, but the art of non-verbal communication is just that: an art. And like art, it opens the door to myriad interpretations, but also to greater empathy and that only benefits design.
Tonight I attended my first of what I hope will be many monthly meetings of the Design With Dialogue community of practice being held at OCAD University in Toronto. The topic of the evening was What do clowns know that you don’t ? The hosts were an international clown troupe comprised of Patricia Kambitsch, Heidi Madsen, and Elsa Lam. The answer to the question posed by the evening is: a lot.
The night began with a series of exercises done first in pairs, then pairs of pairs, and then as teams of four. What struck me was that, prior to this evening and a few Twitter follows, I didn’t know a single person at this event. Yet, after the course of two and a half hours, I felt I had a room full of new “peeps”. I was thrilled to find an interesting, engaged and dynamic group of people who could perform for each other without the safety net provided by familiarity.
So what brought this about and what does it have to do with clowns? Actually, the clowns were not made up nor was there even mention of clowning beyond the introduction of the hosts. In the case of tonight’s activity, the clowning was due more to physical performance, and particularly the use of non-verbal communication. Over the course of two hours we went through four sets of activities:
1. In pairs, determined by how tall you were (which isn’t relevant, it just allows for a creative way of splitting the room up), introduce yourself using gestures — particularly exaggerated ones — and then mirror that response back to your partner with no words expressed between you.
2. As a pair, join with another pair and use the non-verbal communication rapport generated from the first exercise to work together to non-verbally communicate a particular emotion (including some tough ones like passed over and pity — try acting these out, you’ll see how hard that can be).
3. Working with both pairs together as a foursome, the new group of four is asked to act out a particular phrase. They are to do this while walking across the room where the rest of the participants are asked to guess the phrase. There is little communication between the four people in this new group.
4. The four individuals sit on four chairs and acts out a skit called “four clowns on a bench” where one person is whispered a scenario and the other three are asked to follow along, not knowing what the actually phrase is.
What happens after all of these is remarkable. I found myself acting in a group on something I didn’t know, yet perceived because of the empathy that I developed over the course of two short hours of working non-verbally. As a result, my group — team — and all the others put on performances that were funny, coherent, and creative with little to no verbal sharing of information.
This stoking of empathy and the insight it produced demonstrated enormous potential for design and teamwork. Building on work that Keith Sawyer has done looking at improv and creativity, this session demonstrated just how powerful non-verbal, emergent communication can be and how us designers — in whatever situation we inhabit — dismiss such opportunities for learning, creative expression and community building at our peril.
I wasn’t really a fan of clowns before tonight; now, I am.
** Photo by Peter Pearson under Creative Commons License from the Flickr pool
Grading is the tyrrany of higher education and this week I had to face it full-force and get reminded why we value the appearance of education rather than true learning.
This week I submitted the final grades for my graduate course on health behaviour change. Submitting grades is always an emotional time for me. I’ve watched students do poorly not due to lack of understanding, but circumstance. I’ve also seen students turn things around after having started out slow and ending on a high note. In every case, I end up assigning a letter (sometimes with a + or – attached to it) to assess the quality of the work done, which is supposed to be a proxy for learning. The truth is far from that.
More than any other semester in my teaching history I found myself struggling with grading. Grades are holdovers from a system designed to produce good little factory workers who would have enough knowledge not to hurt themselves and do the job right, but not quite enough to truly challenge the system that said they had to work the way that they did. Unfortunately, old habits are hard to break (which is, ironically perhaps, the theme of the course I teach).
I am fortunate enough to have a room to teach with moveable chairs and tables, although there is really only a few designs open to me given that the room is literally filled to the capacity set by the fire marshall. It’s still better than the circumstance illustrated in the above photo, with students sitting in rows all looking the same. I am pleased that many of the students in my course call me “Cameron” and not just “Dr. Norman” or “Professor”. We’ve made a lot of progress, but at the same time there is much illusion about the nature of education today.
I teach up to 30 graduate students — both masters and doctoral level students — at a time. When you pile 30+ learners (including the TA’s, guest lecturers, guest students) into one room, the type of teaching and learning you are able to do is seriously limited by the size of the class, the room, the complexity of the material being presented, and the time you have available to explore that material. I do my best and the students do theirs, but it is limited. And yet, we call this graduate education. We call it education; period.
It is as if individuals have no prior experience of their own and couldn’t possibly add to the discussion in any meaningful way. As such, we set up a system of evaluation that suggest that I, as the implied smartest person in the room, can truly judge the worthiness of any idea with complete objectivity, precision, and efficiency and that is worth something. Well, when it comes to health behaviour change or systems thinking (the two courses I teach) I can confidently say that I have more codified, structured, academically acceptable knowledge than any one person in my classroom. But do I have more than the class combined? No way. I’m not even close.
So it would surmise that some method of tapping into that knowledge of the 10, 20, or 30 students is a good idea. But doing so means acknowledging that the professor — the said “smart person” in the room — might not have all the answers and maybe some of the students have those answers. Or in the case of complex and novel problems that we see in public health more often these days, maybe no one has the answer. Maybe the answer needs to be generated by collaboration, discussion and bringing diverse groups together.
But what does this mean for grading? If five people help derive a solution to a problem, who gets the grade? Some models might suggest the leader gets more credit than the others as we see in the academic peer-reviewed publication traditions . While others might have some form of negotiated hierarchy of authors. A systems thinking perspective might throw the whole authorship issue out altogether because it was the contribution of the team acting as such that generated the knowledge. Yes, some may have worked longer hours, taken bigger roles, but the entire product is the sum of the whole of the parts, therefore every component is considered vital. If we took this into account, we’d have to award the same grade to everyone if producing knowledge that was useful was the goal of the activity.
That’s pretty heretical stuff where I come from. Yes, it is true we can have group grades, but this is speaking to a fundamental issue of contribution and acknowledging that not everyone will add the same value and that is OK, because in the end it is what people add in its totality that is most important the whole.
In complexity terms, grading is anathema. It suggests that we can know what is “right” and “wrong” and “effective” and “ineffective” in each circumstance. In simple systems, that might be true. When we have “best practice” that is reliable and valid and can be assessed consistently, then grades are perfectly reasonable. Yet, when we work in spaces where the context changes, the variables multiply and shift, and the outcomes can, at best, be anticipated but not predicted, the idea of assessing people based on concrete, objective standards seems silly at best, dangerous at worst. But that’s what we do all the time.
Complexity does endorse — and indeed, thrives on — feedback. Getting some form of assessment is great so long as it is provides opportunities for adaptation. Without it, complex systems would become simple ones.
Imagine a system where we gave students feedback, allowed them to adapt, and to take the information they learn and apply it in ways that fit the context they are working in? Consider what that might look like in terms of grades and grading and how the absence of such almost arbitrary assessments could lead to knowledge that could truly advance the health and wellbeing of everyone, not just propose to do so.
This is not just systems thinking, but true systems change and that is what education is all about in my books.
The fire, not as large as one could imagine when you hear “6-alarm blaze”, was still far greater in its impact than its size would suggest. As of last night, there remain 1700 people unable to return home. The building’s structural integrity is now in question, which could pose even further problems for a community that is not well prepared to cope with it. Reading through the stories of what happened in the community, which is where much my research group‘s work is focused on, it is hard to imagine how difficult it must be for people living in a modern city to be camped out in makeshift shelters that are propped up throughout the downtown.
One quote from the Torontoist’s coverage points at the cascading set of problems that these problems cause:
“I just want to know what’s going on,” said Romaniuk, a fourteen-year resident of 200 Wellesley Street with long red hair and an accent that was difficult to place. She had arrived home from work last night only to be denied access to her apartment by emergency responders. “At some point I need to get in. I need to go back to work. I have no clothes to go back to work.” She said she’d slept at her cousin’s home, and that she’d do so again tonight, if necessary. For those who had nowhere to go, the Community Centre was filled with cots, draped with Red Cross blankets. Some residents slept at other ad-hoc downtown shelters last night.
Here we see a remarkable dichtomy between the a part of the world where such sites are rare and those parts where such sites are common, perhaps even semi-permanent (PDF). In Toronto, emergency services have done a decent job of handling the crisis and moving quickly to find places to hold residents who are without a home. But what passes for good in these situations is usually a matter of perspective.
The cascading set of problems that these problems cause are usually examples of complexity in action. The interconnectedness between events and the unintended consequences that emerge from simple actions have ramifications that our post-event analyses only scratch the surface upon. They also cause much discussion about the suitability of emergency preparedness plans. Such plans, often designed to help communities respond quickly in a disaster, tend to work well when the parameters are known and the system constraints are reasonably tight. Airplane emergency safety planning is one area. In an emergency, those in a plane have very few options for escape and in those situations where a problem occurs and there is a chance of survival, most of the strategies, imperfect as they are, will do the job of getting people to safety. A plane is a closed system.
Communities are more troublesome beasts. They are open systems and it is virtually impossible to imagine the variety of scenarios that could unfold in the event that a large scale disaster takes place. The aftermath of Hurricane Katrina showed clearly the flaws in both their plan, but also in the mindset that goes into planning in the first place. The mere act of planning is problematic when you consider a complex system.
The Oxford English Dictionary defines a plan as:
1 a detailed proposal for doing or achieving something : the UN peace plan.
• [with adj. ] a scheme for the regular payment of contributions toward a pension, savings account, or insurance policy : a personal pension plan.
2 (usu. plans) an intention or decision about what one is going to do : I have no plans to retire.
3 a detailed diagram, drawing, or program, in particular
• a fairly large-scale map of a town or district : a street plan.
• a drawing or diagram made by projection on a horizontal plane, esp. one showing the layout of a building or one floor of a building. Compare with elevation (sense 3).
• a diagram showing how something will be arranged : look at the seating plan.
Consider the terms. The first is a detailed plan of what you are going to do. This means having some idea of what the context will be, what the parameters are, and the agents involved. How often can we do this reliably?
The second part, intention, is far easier. This is something that one can develop abstract, but focused sets of ideas about what is to be achieved.
The last part is about as problematic as the first.
Colin Powell had a more realistic, complex view of planning:
No battle plan survives contact with the enemy
Since Katrina and as the potential spectre of a pandemic influenza sits in our minds, public health has been focusing on emergency preparedness. Thinking in complex terms might enable us to get the best of our intentions to gel with what Powell speaks of: contact. The Toronto fire example provides a decent case for planning, but as the unplanned for consequences begin to reveal themselves (lack of ability to work, loss of pets, missing medication schedules, eating nothing but pizza for three days straight to name a few) the strength of this plan will be forgotten. Considering things as complex from the outset means that plans are no longer solid documents, but fluid, adaptive processes that require new ways of engaging this complexity.
I don’t see much of that. But then, I’m too busy planning for other events that are equally as ludicrous (classes, papers, research projects). Perhaps we all would be wise to heed John Lennon:
Life is what happens when you’re busy making other plans.
For those interested in learning more or doing more for those affected by the fire in Toronto, here are some links:
ongoing activities and news: http://search.twitter.com/search?q=wellesleyfire
co-ordination wiki: http://www.torontopedia.ca/200_Wellesley_Fire
fundraising opportunities: http://twitter.com/WellesleyFireTO
Simple, straightforward and predictable things are pretty boring, but they at least can be understood without much effort. And sometimes that simplicity provides comfort that we can’t find in complicated, complex or chaotic events. As we find ourselves working long hours eating badly and sleeping less hours than our body would like its no surprise that we find a lot of organizations trying to make complex change using simple processes (that won’t work). It’s tiring thinking about complexity and simplicity is, well, simple. We don’t need to consider the pushback that could come from making our morning coffee, we need not worry about the unintended consequences of ironing our shirts, or contemplate the emergent patterns that come from picking a green M&M out of the holiday party bowl over the red one. After a long day at the office or an emotional conversation with a loved one, these ‘simple pleasures’ as they are often referred to provide us comfort that can’t be found in complexity.
But change is rarely a pleasure, but always an adventure; When it comes we need to be ready and have the energy to tackle it.
It is perhaps for that reason that people try to deny it or over-simplify problems. Its the very reason why the self-help book section of a store is so big, why New Year’s resolutions are so popular (do you have yours yet?), and why late night infomercials and daytime talkshows still persist in their efforts to sell us the quick and easy change. Change your life in three, five, seven, 10 or 12 easy steps!
It is never that easy. If it was, I could teach my students health behaviour change in an evening seminar at a hotel airport instead of a semester-long graduate course that is, at best, showing the ice floating above the waterline. However, in that proverbial sea of self-help resources one of the few ideas that stands out comes from The Power of Full Engagement. In the book, authors Jim Loehr and and Tony Schwartz point out that a key to change is managing energy as much as it is our cognitions, emotions and behaviour. It is the energy we bring to situations that is the necessary precondition to becoming fully engaged and able to change. It’s why its so hard to pay attention in class or a meeting when you’re tired. Or why you tune out when the message itself is tired; the same old stuff trotted out again and again.
Change in human systems is complex.
Tired individuals and organizations tend to opt for those solutions to complex problems that are simple and, as H.L. Menken said, wrong, — see my last post. Ever seen profound change take root in an exhausted environment? Not me. It’s one of the reasons why effective leaders are those that aim to spark emotion and raise the energy level of those that follow them as much as instill new ideas. Indeed, if you look at many of the best leaders out there, they tend to create environments where new ideas come from introducing new ways to see the complex and make it exciting. A terrific example of this is Benjamin Zander’s talk at TED looking at how the complicated structure and complexity of classical music can enliven the spirit.
So perhaps our first strategy to change is to take a nap, play some Chopin and watch an inspirational movie than try and solve it otherwise we might end up with simple and wrong solutions to complex problems and be no better off for it.
The benefits of standing still and looking around at the systems around us never cease to reveal themselves.
Mindfulness is something that is most often associated with individuals. Mindfulness is a pillar of Buddhist practice and is increasingly being used in clinical settings to help people deal with stress and pain.
Mindfulness sometimes get unfairly linked to individuals, groups and movements that, for lack of a better term, could be described as ‘flaky’. Its association with many spiritual movements can also be problematic for those who are looking for something more aligned with science and less about religion or spirituality. Yet, the spiritual and scientific benefits of mindfulness need not be incompatible. Google, while innovative and often unusual in the way it runs its business, is certainly not flaky. As a company, it understands the power of mindfulness and has hosted a few talks on its application to everyday life and its neuroscientific foundations and benefits. For companies like Google, promoting mindfulness yields health benefits to its individual staff members, but also to its bottom line because being mindful as a company allows them to see trends and the emergence of new patterns in how people use the Internet and search for information. Indeed, one could say that Google with its search engine and productivity tools could be the ultimate mindfulness company, aiding us to become aware of the world around us (on the Internet anyway).
We are often profoundly ignorant of the systems that we are a part of and while the idea of having us all sit and mediate might sound appealing (particularly those of us who could use a moment of peace!) it is not a reasonable proposition. One of the things that meditation does is enable the mediator to become aware of themselves and their surroundings often through a type of mental visualization. Visualization allows the observer to see the relationships between entities in a system, their proximity, and the extended relationships beyond themselves. In systems research and evaluation, this might be done through the application of social network analysis or a system dynamics model. Through these kinds of tools that allow us to enhance visualization potential of systems, this is almost akin to creating a mindful systems thinking tool.
My colleague Tim Huerta and I have been developing methods and strategies to incorporate social network analysis into organizational decision making and published a paper in 2006 on how this could be done to support the development of communities of practice in tobacco control. I’m also working on creating a system dynamics model of the relationships within the gambling system in Ontario with David Korn and Jennifer Reynolds.
By creating visuals of what the system looks like consciousness raising takes place and the invisible connections become visible. And by making things visible the impact, reach, scope and potential opportunities for collaboration and action are made aware. And with awareness comes insight into the connections between actions and consequences (past, current and potential) and that allows us to strategize ways to minimize or amplify such effects as necessary.