What is Developmental Evaluation?

Posted: November 19, 2011 | Author: | Filed under: complexity, evaluation, innovation, systems thinking | Tags: , , , , , , , | 5 Comments »

Developmental evaluation (DE) is a problematic concept because it deals with a complex set of conditions and potential outcomes that differ from and challenge the orthodoxy in much of mainstream research and evaluation and makes it difficult to communicate. At a recent gathering of DE practitioners in Toronto, we were charged with coming up with an elevator pitch to describe DE to someone who wasn’t familiar with it; this is what I came up with. 

Developmental evaluation is an approach to understanding the activities of a program operating in dynamic, novel environments with complex interactions. It  focuses on innovation and strategic learning rather than standard outcomes and is as much a way of thinking about programs-in-context and the feedback they produce. The concept is an extension of Michael Quinn Patton’s original concept of Utilization Focused Evaluation with concepts gleaned from complexity science to account for the dynamism and novelty. While Utilization Focused Evaluation has a series of steps to follow (PDF), Developmental Evaluation is less prescriptive, which is both its strength and its challenge for describing it to people (things I’ve discussed in earlier posts).

So with that in mind, our group was charged with coming up with a way to explain DE to someone who is not familiar with it using anything we’d like — song, poetry, dance, slides, stories and beyond. While my colleague Dan chose to lead us all in song, I opted to go with a simple analogy by comparing DE to a hybrid of Trip Advisor and the classic Road Trip (due to lack of good vocalizing skills).

Trip Advisor has emerged as one of the most popular tools for travellers seeking advice on everything from hotel rooms to airlines to resorts and all the destinations along the way. Trip Advisor is averaging more than 13 million unique visitors per month and, unlike its competitors, focuses on user-generated content to support its service. Thus, your fellow travellers are the source of the recommendations not some professional travel agent or journalist. At its heart are stories of varies tones, detail and quality. People upload various accounts of their stay, chronicling even the most minute detail through photos, links to their blogs, video, and narrative. If you want to get the inside details on what a hotel is really like, check Trip Advisor and you’ll likely find it.

However, like any self-organizing set of ideas, the quality of the content will vary along with the level of reportage and the conclusions will be different depending on the context and experience of the person doing the reporting. For example, if you are a North American who is used to having even the most basic hotel chain offer a room with full-service linens, a bathroom, closet, desk and separate shower, you’ll have a hard time adjusting to something like EasyHotel in Europe.

The Road Trip part (capitalization intended here to denote something different than a regular trip by road), denotes the experience that comes from a journey with a desired destination, but not a pre-determined route and only a generalized timeline. A Road Trip is something that is more than just traveling from Point A to Point B, which is usually accomplished by taking the shortest route, the fastest route or a combination of the two; rather it is a journey. Movies like National Lampoon’s Vacation (and, European Vacation), Thelma and Louise, Harold & Kumar Go to White Castle, and (surprise!) Road Trip all capture this spirit to some effect. I suppose one might even find a more grim example of a Road Trip in the Lord of the Rings Trilogy or The Road.

Road Trips have a long history and are not just a North American phenomenon as this article from the Indian Newspaper, The Hindu reports in some detail:

“Road trips are fun when they are not planned point-to-point. As long as you have accommodation booked, that is enough. Its better not to have agendas; get as spontaneous and adventurous as you can. My friends and I went on a road trip to Goa last year. It was loads of fun as it was the first time we took off on our own without parents. To me, it was more than just a trip with friends. It showed that I could take care of myself and that I was now a grown-up, free to do what I wanted,” says Siddharth, who is doing his engineering.

The idea of spontaneity and adventure are part of the process, not an unexpected problem to be solved like in a traditional evaluation. Indeed, some of these unplanned and unusual departures are not only part of the learning, but essential to it. It is akin to what Thor Muller describes as planned serendipity; you might not know what is going to come, but it is possible to set the conditions up to increase the likelihood of and preparedness for moments of discovery and learning. This is like setting out on a journey with a mindset of developmental and strategic learning to fit with what Louis Pasteur stated about discovery:
Chance favours the prepared mind
Thus, as Developmental Evaluators and program implementation leaders we are creating conditions to learn en route to a general destination, but without a clear path and an open mind towards what might unfold. This attention to the emergence of new patterns and then the sensemaking to understand what these new patterns mean in the context to which they emerged and the goals, directions and resources that surround the discovery is a important facet of what separates Developmental Evaluation from other forms of evaluation and research.
So in describing DE to others, I proposed combining these two ideas of Trip Advisor and the Road Trip to create: Road Trip Advisor.

Road Trip Advisor for Developmental Evaluation

Road Trip Advisor would involve going on a journey that has a general destination, but with no single path to it. Along the way, the Developmental Evaluator would work with those taking the journey with him — likely the program staff, stakeholders and others interested in strategic learning and feedback — and systematically capture the decision points to take a particular path, the process that unfolded in making decisions, the outcomes or events connected to those decisions inasmuch as one can draw such linkages, and then continually dialogue with the program team about what she or he or they are seeing, sensing and experiencing. This includes what innovations are being produced.
Returning to the article on road tripping from The Hindu:
“Road-tripping is a great way to bond with the people you are travelling with and I would strongly recommend it to people. It not only makes you appreciate yourself as an individual but is an amazing experience as you get to meet new people, know different cultures and sample different cuisines. I can never forget biking on sleet, riding though torrential rains, gobbling hot rotis at dhabas, the beautiful snow-capped mountains and guy talk with friends on the trip,” says Dheeraj, who recently went to Ladakh.
Here the focus is on relationships, learning new things and taking that learning onward. That is what DE is all about. My colleague Remi illustrated this in our meeting by having us all spread out throughout the room and go through a pantomime-type skit where he collected information from each participant about where the wisdom was and then bringing this person along for the journey. So as he started out alone as the Developmental Evaluator, he wound up at the destination of wisdom with everyone.
Road Trip Advisor requires documenting the journey along the way, sharing what you learn with others, and continuing learning and revisiting your notes — while checking out what notes others have (including use of evidence from other projects and academic research) — and integrating that together on an ongoing basis.
But as my other colleagues pointed out in their presentations, the journey isn’t always about feeling good. Sometimes there are challenges as the Hindu article adds:

all is not hunky dory during these trips. You have to be way about accidents and mishaps. And, realise that freedom comes with responsibility. Says Arjun: “I had borrowed my friend’s bike for the trip, and though it looked good, it gave problems on the foothills of Kodaikanal and we couldn’t do the climb. Being a weekend, there were no mechanics. It helps to know your machine. A passion for road-tripping is not enough. You need to be equipped to take care of yourself also.”

Here, the story parallel is about being prepared. Know evaluation methods, know how to build and sustain relationships and to deal with conflict. A high tolerance for ambiguity and the flexibility to adapt is also important. Knowing a little about systems thinking and complexity doesn’t hurt either. Developmental evaluation is not healthy for those who need a high degree of predictability, are not flexible in their approach, and adhere to rigid timelines. Complex systems collapse under rigid boundary conditions and do evaluators working with such restrictions in developmental contexts.

So why do people do it? “Well, my memories of my favourite road trip were an injured leg, chocolates, beautiful photographs and a great sense of fulfilment,” recalls Arjun.

It is youngsters like these who have transformed road-tripping from just a hobby to an art.

After all, friendship and travel is a potent combination that you can’t say no to.

 In DE, the “youngsters” are everyone. But as we (my DE colleagues) all pointed out: DE is fun. It is fun because we learn and grow and challenge ourselves and the programs that we are working with. It’s collaborative, instructive, and promotes a level of connection between people, programs and ideas that other methods of evaluation and learning are less effective at. DE is not for everyone or every program and Micheal Quinn Patton has pointed this out repeatedly. But for those programs where innovation, strategic learning and collaboration count, it is pretty good way to journey from where you are to where you want to go.

Design Thinking and Zombies

Posted: November 11, 2011 | Author: | Filed under: design thinking, innovation, research | Tags: , , , , | 2 Comments »

Stopping to sip the brains of sloppy design thinkers

The concept of design thinking has been much maligned in some circles; declared dead, brought to life, and now, or like a zombie, walking in a state somewhere between. If the concept is to live or die it must do so based on evidence from research and practice, not rhetoric as it’s been up until now. 

The FastCo Design blog has posted an article proclaiming: Is Design Thinking Dead? Hell No!  by Grant McCracken from C3 at MIT. McCracken’s post was in response to the oft-cited editorial by Bruce Nussbaum a few months ago that I’ve commented on many times in this blog space about how design thinking is a failed experiment; basically dead.

According to McCracken, DT is still alive, or at least undead.

Few concepts have engendered such a strong reaction from so many. Writing in the Harvard Business Review blog, Peter Merholz made the case that design thinking has been an oversold concept and is not the tool some think it is.  The Design Sojourn blog went so far to suggest that design thinking kills creativity. These articles run counter to a series of books, special issues and conferences that have sought to promote design thinking widely.

The concern I have for much of the discourse on design thinking — its life, death and zombie-like undead state — is that it is nearly all based on rhetoric alone.

Definitions of design thinking tend toward: Design thinking is what I say it is and I am a designer, therefore I know design thinking. While maybe true for an individual designer, such claims to a concept become problematic when, as McCracken points out, entire programs of activity from U of T Rotman’s b-school to Stanford’s d-school to IDEO and Jump have embraced this concept wholeheartedly and focused their business around it. The stakes are getting higher for design and design thinking with little attention being paid to what designers and non-designers actually do or think about what they do.

As an academic, I don’t declare something alive or dead until its been thoroughly examined. A concept like design thinking, if it is to have worth, must withstand scrutiny through both theoretical and empirical examination. A review of the literature (academic and grey) so far, suggests that neither has been done sufficiently. What is the theory of design thinking? We don’t really have one. It appears to be a set of strategies and a stance that are loosely connected to the process of exercising creative, intentional control in the pursuit of a useful problem solution. Does this set of processes or the stance produce good solutions or better solutions than other ways of doing things? We don’t know. In part, this is because we really don’t know what it is. That is the first step towards answering the bigger question about what it does.

Some, like Nigel Cross, have sought to do studies looking at what designers do, while others like Roger Martin, have tried to articulate how design thinkers think, but neither have done so in a systematic way that extends beyond a few case studies. A true, synthetic and empirically supported evidence base is what is missing. It is time to change this if the concept of design thinking is to have a future or is to be rightfully put down.

My colleague Andrea Yip and I are seeking to change this. Our project, Design Thinking Foundations, is focusing on a synthesis of the literature and interviews with leading professionals from different fields within design, branding, media, and business. Combined with observations and reflexive practice within our own design work, we intend to bring more than just rhetoric to design thinking, but data.  Our stake is less in the name design thinking, but more to determine what it is, how it is practiced, and what value it brings in an empirical and theoretically robust manner. Through research we hope to answer the question about whether design thinking is alive and well or simply the walking undead.

** Photo by Dance Photographer Brendan Lally used under Creative Commons License from Flickr.

Follow

Get every new post delivered to your Inbox.

Join 1,477 other followers