More Design Thinking & Evaluation
Posted: November 15, 2010 Filed under: design thinking, Education & Learning, evaluation, innovation | Tags: creativity, design thinking, developmental evaluation, evaluation, innovation Leave a comment »
Capturing Design in Evaluation (CameraNight by Dream Sky, Used under Creative Commons License)
On the last day of the American Evaluation Association conference, which wrapped up on Saturday, I participated in an interactive session on design thinking and evaluation by a group from the Savannah College of Art and Design.
One of the first things that was presented was some of the language of design thinking for those in the audience who are not accustomed to this way of approaching problems (which I suspect was most of those in attendance).
At the heart of this was the importance of praxis, which is the link between theory, design principles and practice (see below)
Design Thinking
As part of this perspective is the belief that design is less a field about the creation of things on their own, but rather a problem solving discipline.
Design is a problem solving discipline
When conceived of this way, it becomes easier to see why design thinking is so important and more than a passing fad. Inherent in this way of thinking are principles that demand inclusion of multiple perspectives on the problem, collaborative ideation, and purposeful wandering through a subject matter.
Another is to view design as serious play to support learning.
Imagine taking the perspective of design as serious play to support learning?
The presenters also introduced a quote from Bruce Mau, which I will inaccurately capture here, but is akin to this:
One of the revelations in the studio is that life is something that we create every single day. We create our space and place.
Within this approach is a shift from sympathy with others in the world, to empathy. It is less about evaluating the world, but rather engaging with it to come up with new insights that can inform its further development. This is really a nod (in my view) to developmental evaluation.
The audience was enthralled and engaged and, I hope, willing to take the concept of design and design thinking further in their work as evaluators. In doing so, I can only hope that evaluation becomes one of the homes for design thinking beyond the realm of business and industrial arts.
Design Thinking & Evaluation
Posted: November 13, 2010 Filed under: design thinking, Education & Learning, evaluation, innovation, research | Tags: conference, design, design thinking, developmental evaluation, education, empathy, evaluation, innovation, social innovation, utilization-focused evaluation 2 Comments »
Design Thinking Meets Evaluation (by Lumaxart, Creative Commons Licence)
This morning at the American Evaluation Association meeting in San Antonio I attended a session very near and dear to my heart: design thinking and evaluation.
I have been a staunch believer that design thinking ought to be one of the most prominent tools for evaluators and that evaluation ought to be one of the principal components of any design thinking strategy. This morning, I was with my “peeps”.
Specifically, I was with Ching Ching Yap, Christine Miller and Robert Fee and about 35 other early risers hoping to learn about ways in which the Savannah College of Art and Design (SCAD) uses design thinking in support of their programs and evaluations.
The presenters went through a series of outlines for design thinking and what it is (more on that in a follow-up post), but what I wanted to focus on here was the way in which evaluation and design thinking fits together more broadly.
Design thinking is an approach that encourages participatory engagement in planning and setting out objectives, as well as in ideation, development, prototyping, testing, and refinement. In evaluation terms, it is akin to action research and utilization-focused evaluation (PDF). But perhaps its most close correlate is with Developmental evaluation (DE). DE is an approach that uses complexity-science concepts to inform an iterative approach to evaluation that is centred on innovation, the discovery of something new (or adaptation of something into something else) and the application of that knowledge to problem solving.
Indeed, the speakers today positioned design thinking as a means of problem solving.
Evaluation , at least DE, is about problem solving by collecting the data used as a form of feedback to inform the next iteration of decision making. It also is a form of evaluation that is intimately connected to program planning.
What design thinking offers is a way to extend that planning in new ways that optimizes opportunities for feedback, new information, participation, and creative interaction. Design thinking approaches, like the workshop today, also focuses on people’s felt needs and experiences, not just their ideas. In our session today, six audience members were recruited to play the role of either three facets of a store clerk or three facets of a customer — the rational, emotional and executive mind of each. A customer comes looking for a solution to a home improvement/repair problem, not sure of what she needs, while the store clerk tries to help.
What this design-oriented approach does is greatly enhance the participant’s sense of the whole, what the needs and desires and fears both parties are dealing with, not just the executive or rational elements. More importantly, this strategy looks at how these different components might interact by simulating a condition in which they might play out. Time didn’t allow us to explore what might have happened had we NOT done this and just designed an evaluation to capture the experience, but I can confidently say that this exercise got me thinking about all the different elements that could and indeed SHOULD be considered if trying to understand and evaluate an interaction is desired.
If design thinking isn’t a core competency of evaluation, perhaps we might want to consider it.
Systems Thinking, Logic Models and Evaluation
Posted: November 12, 2010 Filed under: complexity, evaluation, research, systems thinking | Tags: complexity, conference, evaluation, systems thinking 1 Comment »
San Antonio at Night, by Corey Leopold (CC License)
The American Evaluation Association conference is on right now in San Antonio and with hundreds of sessions spread over four days it is hard to focus on just one thing. For those interested in systems approaches to evaluation, the conference has had a wealth of learning opportunities.
The highlight was a session on systems approaches to understanding one of evaluation’s staples: the program logic model.
The speakers, Patricia Rogers from the Royal Melbourne Institute of Technology, and consultants Richard Hummelbrunner and Bob Williams spoke to the challenges posed with the traditional forms of logic models by looking at the concepts of beauty, truth and justice. These model forms tend to take the shape of the box model (the approach most common in North America), the outcome hierarchy model, and the logic framework, which is popular in international development work.
The latter model was the focus of Hummelbrunner’s talk, which critiqued the ‘log frame’ approach and showed how its highly structured approach to conceptualizing programs tends to lead to a preoccupation with the wrong things and a rigidity in the way programs are approached. They work well in environments that are linear, straightforward, and in situations where funders need simple, rapid overviews of programs. But as Hummelbrunner says:
Logframes fail in messy environments
The reason is often that people make assumptions of simplicity when really such programs are complicated or complex. Patricia Rogers illustrated ways of conceptualizing programs using the traditional box models, but showing how different program outcomes could emerge from one program, or that there may be the need to have multiple programs working simultaneously to achieve a particular outcome.
What Rogers emphasized was the need for logic models to have a sense of beauty to it.
Logic models need to be beautiful, to energize people. It’s can’t just be the equivalent of a wiring diagram for a program.
According to Rogers, the process of developing a logic model is most effective when it maintains harmony between the program and the people within it. Too often such model development processes are dispiriting events rather than exciting ones.
Bob Williams concluded the session by furthering the discussion of beauty, truth and justice, by expanding the definitions of these terms within the context of logic models. Beauty is the essence of relationships, which is what logic models show. Truth is about providing opportunities for multiple perspectives on a program. And a boundary critique is a an opportunity for ethical decision making.
On that last point, Williams made some important arguments about how, in systems related research and evaluation, the act of choosing a boundary is a profound ethical decision. Who is in, who is out, what counts and what does not are all critical questions to the issue of justice.
To conclude, Williams also challenged us to look at models in new ways, asking:
Why should models be the servant of data, rather than have data serve the models?
In this last point, Williams highlights the current debates within the knowledge management community, which is dealing with a decade where trillions of points of data have been generated to make policy and programming decisions, yet better decisions still elude us. Is more data, better?
The session was a wonderful puctuation to the day and really advanced the discussion on something so fundamental as logic models, yet took us to a new set of places by considering them as things of artful design, beauty, ethical decision making tools, and vehicles for exploring the truths that we live. Pretty profound stuff for a session on something seemingly benign as a planning tool.
The session ended with a great question from Bob Williams to the audience that speaks to why systems are also about the people within them and emplored evaluators to consider:
Why don’t we start with the people first instead of the intervention, rather than the other way around like we normally do?
American Evaluation Association Conference
Posted: November 9, 2010 Filed under: evaluation, systems thinking | Tags: developmental evaluation, evaluation, research, systems thinking Leave a comment »Over the next few days I’ll be attending the American Evaluation Association conference in San Antonio, Texas. The conference, the biggest gathering of evaluators in the world. Depending on the Internet connections, I will try to do some live tweeting from my @cdnorman and some blogging reflections along the way, so do follow along if you’re interested. In addition to presenting some of the work that I’ve been engaged in on team science with my colleagues at the University of British Columbia and Texas Tech University, I will be looking to connect more with those groups and individuals doing work on systems evaluation and developmental evaluation with an eye to spotting the trends and developments (no pun intended) in those fields.
Evaluation is an interesting area to be a part of. It has no disciplinary home, a set of common practices, but much diversity as well and brings together a fascinating blend of people from all walks of professional life.
Stay tuned.
Developmental Evaluation: Problems and Opportunities with a Complex Concept
Posted: July 31, 2010 Filed under: complexity, evaluation, research, social systems, Systems science, systems thinking | Tags: complexity, developmental evaluation, eHealth, evaluation, knowledge integration, knowledge translation, public health, systems thinking 11 Comments »
Everyone's Talking About Developmental Evaluation
“When it rains, it pours” so says the aphorism about how things tend to cluster. Albert Lazlo-Barabasi has found that pattern to be indicative of a larger complex phenomenon that he calls ‘bursts‘, something worth discussing in another post.
This week, that ‘thing’ seems to be developmental evaluation. I’ve had more conversations, emails and information nuggets placed in my consciousness this week than I have in a long time. It must be worth a post.
Developmental evaluation is a concept widely attributed to Michael Quinn Patton, a true leader in the field of evaluation and its influence on program development and planning. Patton first wrote about the concept in the early 1990′s, although the concept didn’t really take off until recently in parallel with the growing popularity of complexity science and systems thinking approaches to understanding health and human services.
At its root, Developmental Evaluation (DE) is about evaluating a program in ‘real time’ by looking at programs as evolving, complex adaptive systems operating in ecologies that share this same set of organizing principles. This means that there is no definitive manner to assess program impact in concrete terms, nor is any process that is documented through evaluation likely to reveal absolute truths about the manner in which a program will operate in the future or in another context. To traditional evaluators or scientists, this is pure folly, madness or both. When your business is coming up with the answer to a problem, any method that fails to give you ‘the’ answer is problematic.
But as American literary critic H.L. Mencken noted:
“There is always an easy solution to every human problem — neat, plausible and wrong”
Traditional evaluation methods work when problems are simple or even complicated, but rarely do they provide the insight necessary for programs with complex interactions. Most community-based social services fall into this realm as does much of the work done in public health, eHealth, and education. The reason is that there are few ways to standardize programs that are designed to adapt to changing contexts or operate in an environment where there is no stable benchmark to compare.
Public health operates well within the former situation. Disaster management, disease outbreaks, or wide-scale shifts in lifestyle patterns all produce contexts that shift — sometimes radically — so that the practice that works best today, might not be the one that works best tomorrow. We can see this problem demonstrated in the difficulty with ‘best practice’ models of public health and health promotion, which don’t really look like ‘best’ practices, but rather provide some examples of things that worked well in a complex environment. (It is for this reason that I don’t favour or use the term ‘best practice’ in public health, because I simply view too much of it as operating in the realm of the complex, which is something for which the term is not suited.)
eHealth provides an example of the latter. The idea that we can expect to develop, test and implement successful eHealth interventions and tools in a manner that fits with the normal research and evaluation cycle is impractical at best and dangerous at the worst. Three years ago Twitter didn’t exist except in the minds of a few thousand and now has a user population bigger than a large chunk of Europe. Geo-location services like Foursquare, Gowalla and Google Latitude are becoming popular and morphing so quickly that it is impossible to develop a clear standard to follow.
And that is OK, because that is the way things are, not the way evaluators want them to be.
DE seeks to bring some rigour, method and understanding to these problems by creating opportunities to learn from this constant change and use the science of systems to help make sense of what has happened, what is going on now, and to anticipate possible futures for a program. While it is impossible to fully predict what will happen in a complex system due to the myriad interacting variables, we can develop an understanding of a program in a manner that accounts for this complexity and creates useful means of understanding opportunities. This only really works if you embrace complexity rather than try and pretend that things are simple.
For example, evaluation in a complex system considers the program ecology as interactive, relationship-based (and often networked) and dynamic. Many of the traditional evaluation methods seek to understand programs as if they were static. That is, that the lessons of the past can predict the future. What isn’t mentioned, is that we evaluators can ‘game the system’ by developing strategies that can generate data that can fit well into a model, but if the questions are not suited to a dynamic context, the least important parts of the program will be highlighted and thus, the true impact of a program might be missed in the service of developing an acceptable evaluation. It is, what Russell Ackoff called: doing the wrong things righter.
DE also takes evaluation one step further and fits it with Patton’s Utlization-focused evaluation approach., which frames evaluation in a manner that focuses on actionable results. This approach to evaluation integrates the process of problem framing,data collection, analysis, interpretation and use together akin to the concept of knowledge integration. Knowledge integration is the process by which knowledge is generated and applied together, rather than independently, and reflects a systems-oriented approach for knowledge-to-action activities in health and other sciences, with an emphasis on communication.
So hopefully these conversations will continue and that DE will no longer be something that peaks on certain weeks, but rather infuses my colleagues conversations about evaluation and knowledge translation on a regular basis.
Design Thinking or Design Thinking + Action?
Posted: April 8, 2010 Filed under: design thinking, Education & Learning, evaluation, innovation, research, systems thinking | Tags: design, design thinking, education, evaluation, innovation, research methods, university Leave a comment »
There is a fine line between being genuinely creative, innovative and forward thinking and just being trendy.
The issue is not a trivial one because good ideas can get buried when they become trendy, not because they are no longer any good, but because the original meaning behind the term and its very integrity get warped by the influx of products that poorly adhere to the spirit, meaning and intent of the original concepts. This is no more evident than in the troika of concepts that fit at the centre of this blog: systems thinking, design thinking and knowledge translation. (eHealth seems to have lost some its lustre).
This issue was brought to light in a recent blog post by Tim Brown, CEO of the design and innovation firm IDEO. In the post, Brown responds to another post on the design blog Core77 by Kevin McCullagh that spoke to the need to re-think the concept of design thinking and whether it’s popularity has outstripped its usefulness. It is this popularity which is killing the true discipline of design by unleashing a wave of half-baked applications of design thinking on the world and passing it off as good practice.
There’s something odd going on when business and political leaders flatter design with potentially holding the key to such big and pressing problems, and the design community looks the other way.
McCullagh goes on to add that the term design thinking is growing out of favour with designers themselves:
Today, as business and governments start to take design thinking seriously, many designers and design experts are distancing themselves from the term.While I have often been dubbed a design thinker, and I’ve certainly dedicated my career to winning a more strategic role for design. But I was uncomfortable with the concept of design thinking from the outset. I was not the only member of the design community to have misgivings. The term was poorly defined, its proponents often implied that designers were merely unthinking doers, and it allowed smart talkers with little design talent to claim to represent the industry. Others worried about ‘overstretch’—the gap between design thinkers’ claims, and their knowledge, capabilities and ability to deliver on those promises.
This last point is worth noting and it speaks to the problem of ‘trendiness’. As the concept of design thinking has become commonplace, the rigor in which it was initially applied and the methods used to develop it seem to have been cast aside, or at least politely ignored, in favour of something more trendy so that everyone and anyone can be a design thinker. And whether this is a good thing or not is up for debate.
Tim Brown agrees, but only partially, adding:
I support much of what (McCullagh) has to say. Design thinking has to show impact if it is to be taken seriously. Designing is as much about doing as it is about thinking. Designers have much to learn from others who are more rigorous and analytical in their methodologies.
What I struggle with is the assertion that the economic downturn has taken the wind out of the sails of design thinking. My observation is just the opposite. I see organizations, corporate or otherwise, asking broader, more strategic, more interesting questions of designers than ever before. Whether as designers we are equipped to answer these questions may be another matter.
And here in lies the rub. Design thinking as a method of thinking has taken off, while design thinking methodologies (or rather, their study and evaluation) has languished. Yet, for design thinking to be effective in producing real change (as opposed to just new ways of thinking) its methods need to be either improved, or implemented better and evaluated. In short: design thinking must also include action.
I would surmise that it is up to designers, but also academic researchers to take on this challenge and create opportunities to develop design thinking as a disciplinary focus within applied research faculties. Places like the University of Toronto’s Rotman School of Business and the Ontario College of Art and Design’s Strategic Innovation Lab are places to start, but so should schools of public health, social work and education. Only when the methods improve and the research behind it will design thinking escape the “trendy” label and endure as a field of sustained innovation.
When Change Potential is Embedded in Bigger Systems
Posted: March 13, 2010 Filed under: behaviour change, complexity, evaluation, health promotion, psychology, social systems, Systems science, systems thinking | Tags: Barack Obama, behaviour change, complexity, ecological model, embedded systems, empowerment, evaluation, heath promotion, lock-in, motivation, policy, politics, psychology, structure, systems thinking Leave a comment »
Yesterday I was part of an examination committee for a student discussing issues of health promotion, policy change and advocacy for a population that has been widely viewed as marginalized. The challenge that this student was wrestling with was balancing issues of collective and individual empowerment and where the appropriate action needs to take place (and then determining how to evaluate the impact of such action). Drawing on the work of Isaac Prilleltensky and his brilliant work on empowerment theory, the student’s project hopes to foster change that fits somewhere between the individual and community. But how to evaluate the impact?
An empowerment approach, as conceived by Prilleltensky, involves both personal and societal shifts simultaneously to be most effective. If individuals are motivated to change, yet the system is not prepared to adapt to these changes, the value of empowerment is diminished and so is the effect on society. The question shifts to looking at a place to start or determining what the is chicken and what is the egg. This question is less useful than one that considers ways to understand the embedded nature of change agents and change itself within systems shaped both by structure and time.
Barack Obama was elected in a manner that greatly changed the way we look at politics. While he made enormous strides in shaping an electorate, his success at governing has been more muted. Obama’s potential to do well in governing is embedded in the policies and practices that came before him, whether he likes it or not. This is illustrated to full comic effect in a recent Ron Howard ‘Presidential Reunion’ short on Funny or Die. George W. Bush built his policy agenda in a manner that was positioned with Bill Clinton’s, which was positioned with George H.B. Bush’s and so on. Yes, there are some clear departures based on incidents of massive, abrupt change such as September 11th attacks which led to major reactive shifts in policy like the creation of the U.S. Patriot Act , creation of new governmental bodies, and the initiation of two wars abroad. But these are the extremes. A closer look at most non-revolutionary government shifts shows that policy evolves and gets tweaked, but rarely exhibits radical change from administration to administration. Even though the rhetoric around health care reform in the U.S. has spoken of ‘radical change’, the bottom line is that no matter what policy emerges, it will bear closer resemblance to what came before than it will differ.
The embedded structure of social systems is akin to Russian Matryoshka dolls. Our ability to change hinges upon where in stack of dolls we lay and how tightly those dolls are stuck together. I would argue that Obama’s electoral success had a lot to do with a system where the fit of the dolls was loose. There was a clear process to getting nominated (e.g., primaries), but the manner by which interest gets generated and people get out to vote was loose at the time of his campaign. Obama succeeded primarily because he got people to vote who had never come out before, the population that most had given up on trying to reach. In government, the fit is much tighter. Everything has a protocol, a history, and receives an intense scrutiny in that even the smallest shift is noticed, dissected and critiqued.
That leads to a lot of information and feedback, much of it contradictory. Hence, the inertia. With more information than ever at our disposal, the risk that this inertia will persist is high. Jaron Lanier, who I wrote about in my last post, migh attribute this to ‘lock in’ : the dominant way of doing things. Obama succeed because he found a new model of campaigning, captured nicely in three recent books (Harfoush / Plouffe / Sabato). We don’t yet have a new way of governing.
From an evaluation perspective it becomes critical that we understand both these structures and the fit between these variables, or the degree to which the dominant design or ‘lock in’ plays in mediating the impact of change if we are to understand the impact that our efforts to create change are having.
The student who just defended her comprehensive exam, her challenge in using health promotion to instill change will depend on how locked in our society is in its attitudes towards vulnerable populations and the fit between the individual and community with regards to empowerment. I hope, like Obama campaigning in 2008, that fit is loose.
Benchmarking Success in Times of Change
Posted: November 30, 2009 Filed under: behaviour change, complexity, design thinking, eHealth, public health, Science & technology | Tags: complexity, complexity science, design thinking, eHealth, evaluation, innovation, public eHealth, public health, social networks, systems thinking, technology, Twitter, wave Leave a comment »
Successful evaluators know the power of benchmark. The Oxford English Dictionary describes the act ‘to benchmark’ as “evaluate or check (something) by comparison with a standard. The Wikipedia definition of Benchmarking is:
“Benchmarking is the process of comparing the business processes and performance metrics including cost, cycle time, productivity, or quality to another that is widely considered to be an industry standard benchmark or best practice. Essentially, benchmarking provides a snapshot of the performance of your business and helps you understand where you are in relation to a particular standard.”
From an evaluation standpoint, a benchmark provides us with a comparator to help assess how well (or poorly) a particular program is doing. From corporate leaders to university presidents to healthcare administrators benchmarking serves as the referent and focus for programming activities and the foundation for ‘best practice’. But what if best practice isn’t good enough? Or put another way, what if following the leader means going the wrong way?
In the world of consumer or behavioural eHealth much of what we use as our benchmarks are derived from a type of healthcare model that is institution and often technology-centred rather than patient-centred. It is more often something tied to medical treatment of specific problems and technology focused using a highly linear approach to treatment.
Yet in the age of Google Wave, these linear models don’t look to fare well. The future of healthcare, as Frog Design recently opined, is social. What are the benchmarks when your eHealth intervention is not a single technology, but a suite of interacting tools that are online, collaborative and mobile in different measures at different times within a diverse context of treatment and preventive behaviour? How do we measure success? What happens when the ‘effect’ of an intervention is social in nature and supported by multiple tools working in different combinations each time?
In evaluation, we often look for the most likely cause of a particular effect. Yet, what is the effect of any one wave in an ocean of influence? While it is impossible to deconstruct the influence of that wave, it is possible to anticipate what a wave might do under certain conditions and, if the timing is right, it might be possible to get on top of that wave and surf it to shore.
What if we took a wave model and, like surfers, read the seas to determine the appropriate time to dive in, acknowledging that the break will occur differently, the velocity might vary, the height of can’t be predicted, but through activity and practice we can enhance our anticipatory guidance systems to better select waves that might lead to some fine surfing? My research team at the University of Toronto has begun working on these models and methods because as anyone in public health can tell you, the tide is high and with complex problems like chronic disease, the waves are getting big. Twitter, Facebook, blogs, iPhone apps big and small are all collectively influencing people’s behaviour in subtle ways and through acknowledging that these collective tools are the cause and consequence of change can we begin to develop evaluation models to make sense of their impact on the world around us.
