Social Media and Health: Leaders(hip) and Followers(hip)
Posted: May 18, 2012 Filed under: behaviour change, eHealth, health promotion, innovation, public health, Social media, systems thinking | Tags: complexity, creativity, design thinking, developmental design, developmental evaluation, eHealth, evaluation, Facebook, health, health promotion, healthcare, innovation, medicine, organizational change, organizational design, public health, Social media, systems thinking Leave a comment »
Social media is finally catching on with healthcare, public health, and health promotion. With a few recent articles published in the academic literature to rest on, academic health sciences has finally (and I might argue, begrudgingly) conceded that 900+ million users and $100B valuations (Facebook), and thousands of messages exchanged every milisecond (microblogs like Twitter and Sina Wiebo) might have some value for the public beyond entertainment.
If you note how long it took the health sector to start using the telephone as a serious means of engaging their patients or the public, this is lightning-quick adoption. Still, the barriers to adoption are high and the approach to using the technology is scattered. Indeed, just like the start of Internet-delivered telehealth (or cybermedicine (PDF), which has now evolved into eHealth), there is a mad rush to get liked, followed or some other metrics that most health professionals barely understand.
And that is part of the problem.
Meaningful Social Media Metrics
What is a meaningful metric for social media and health? A recently published article in Health Promotion Practice suggested four metrics that are taken from social marketing and applied to social media. These Key Performance Indicators (KPI’s) are:
- Insights (consumer feedback)
- Exposure (media impressions, visits, views, etc..)
- Reach (# people who connect to the social media application)
- Engagement (level of interaction with the content)
These are reasonable, but to to the uninitiated I would suggest a few words of caution and commentary to this list.
Firstly, the insights suggested by Neiger and colleagues “can be derived from practices such as sentiment analysis or data mining that uses algorithms to extract consumer attitudes and other perspectives on a particular topic” (p.162). While not incorrect, this makes the job sound relatively simple and it is not. Qualitative analysis + quantitative metrics such as those derived from data mining are key. Context counts immeasurably in social media use. It’s only in situations where social media is used as a broadcasting tool that gross measures of likes and sentiment analysis work with little qualification.
Even that is problematic. Counts of ‘likes’, ‘visits’, ‘follows’ and such are highly problematic and can be easily gamed. I am ‘followed’ on Twitter by people who have tens of thousands of followers, yet virtually no presence online. Most often they are from marketing fields where the standard practice is to always follow back those who follow you. Do this enough and pretty quickly you, too can have 23,000 followers and follow 20,000 more. This is meaningless from the perspective of developing relationships.
Engagement is the most meaningful of these metrics and the hardest to fully apply. This category gets us to consider the difference between “OMG! AWESOME!” and “That last post made me think of this situation [described here] and I suggest you read [reference] here for more” as comments. Without understanding the context in which these are made within the post, between posts (temporally and sequentially), and in relation to a larger social and informational context, simple text analysis won’t do.
Social Media Evidence: Problems and More Problems
One of the objections to the use of social media by some is that it is not evidence-based. To that extent I would largely agree that this is the case, but then we’ve been jumping out of airplanes with parachutes despite any randomized controlled trial to prove their worth.
Another article in Health Promotion Practice in 2011 highlights potential applications for social media and behaviour change without drawing on specific examples from the literature, but rather on theoretical and rhetorical arguments. An article published in the latest issue of Perspectives on Psychological Science highlights the current state of research on Facebook, which is timely given that its IPO is set for today. That review by Wilson and colleagues illustrates the largely descriptive nature of the field and offers some insight on to the motivation of Facebook users and their online activities, but rather little in what Facebook does to promote active change in individuals and communities when they leave the platform.
The answer to whether social media like platforms such as Facebook ‘work’ as methods of promoting change is simply: we don’t know.
Does social media provide support to people? Yes. Does it inform them? Yes to that too. Does that information produce something other than passive activity on the topic? We don’t know.
In order to answer these questions, health sciences professionals, evaluators, and tech developers need to consider not just followership, but leadership. In this respect, it means creating changes to the way we gather evidence, the tools and methods we use to analyse data, and the organizational structures necessary to support the kind of real-time, rapid cycle evaluation and developmental design work necessary to make programs and evidence relevant to a changing context.
As Facebook launches into its new role as a public company it is almost assured to be introducing new innovations at a rapid pace to ensure that investor expectations (which are enormous) are met. This means that today’s Facebook will not be next month’s. Having funding mechanisms, review and approval mechanisms, a staff trained and oriented to rapid response research, and an overall organizational support system for innovation is the key.
Right now, we are a long way from that. Hospitals are very large, risk averse organizations; public health units are not much different. They both operate in a command-and-control environment suited for complicated, not complex informational and social environments. Social media is largely within the latter.
Systems thinking, design thinking, developmental evaluation, creativity, networks and innovation: these are the keywords for health in the coming years. They are as author Eric Topol calls the dawning of the creative destruction of medicine.
The public is already using social media for health and now the time has come for health (care, promotion and protection) systems to get on board and make the changes necessary to join them.
Like this:
The Complexity of Planning and Design in Social Innovation
Posted: January 7, 2012 Filed under: complexity, design thinking, evaluation, Systems science, systems thinking | Tags: complexity, design, developmental design, developmental evaluation, evaluation, health, human services, Social media, systems thinking 2 Comments »
The Architecture of Complex Plans
Planning works well for linear systems, but often runs into difficulty when we encounter complexity. How do we make use of plans without putting too much faith in their anticipated outcome and still design for change and can developmental design and developmental evaluation be a solution?
It’s that time of year when most people are starting to feel the first pushback to their New Year’s Resolutions. That strict budget, the workout plan, the make-time-for-old-friends commitments are most likely encountering their first test. Part of the reasons is that most of us plan for linear activities, yet in reality most of these activities are complex and non-linear.
A couple interesting quotes about planning for complex environments:
No battle plan survives contact with the enemy – Colin Powell
In preparing for battle I have always found that plans are useless, but planning is indispensable – Dwight D. Eisenhower
Combat might be the quintessential complex system and both Gens Powell and Eisenhower knew about how to plan for it and what kind of limits planning had, yet it didn’t dissuade them from planning, acting and reacting. In war, the end result is what matters not whether the plan for battle went as outlined (although the costs and actions taken are not without scrutiny or concern). In human services, there is a disproportionate amount of concern about ‘getting it right’ and holding ourselves to account for how we got to our destination relative what happens at the destination itself.
Planning presents myriad challenges for those dealing with complex environments. Most of us, when we plan, expect things to go according to what we’ve set up. We develop programs to fit with this plan, set up evaluation models to assess the impact of this plan, and envisage entire strategies to support the delivery and full realization of this plan into action. For those working in social innovation, what is often realized falls short of what was outlined, which inevitably causes problems with funders and sponsors who expect a certain outcome.
Part of the problem is the mindset that shapes the planning process in the first place. Planning is designed largely around the cognitive rational approach to decision making (PDF), which is based on reductionist science and philosophy. Like the image above, a plan is often seen as a blueprint for laying out how a program or service is to unfold over time. Such models of outlining a strategy is quite suitable for building a physical structure like an office where everything from the materials to the machines used to put them together can be counted, measured and bound. This is much less relevant for services that involve interactions between autonomous agents who’s actions have influence on the outcome of that service and that result might vary from context to context as a consequence.
For evaluators, this is problematic because it reduces the control (and increases variance and ‘noise’) into models that are designed to reveal specific outcomes using particular tools. For program implementers, it is troublesome because rigid planning can drive actions away from where people are and for them into activities that might not be contextually appropriate due to some change in the system.
For this reason the twin concepts of developmental evaluation and developmental design require some attention. Developmental evaluation is a complexity-oriented approach to feedback generation and strategic learning that is intended for programs where there is a high degree of novelty and innovation. Programs where the evidence is low or non-existent, the context is shifting, and there are numerable strong and diverse influences are those where developmental evaluations are not only appropriate, but perhaps one of the only viable models of data collection and monitoring available.
Developmental design is a concept I’ve been working on as a reference to the need to incorporate ongoing design and re-design into programs even after they have been initially launched. Thus, a program evolves over time drawing in information from feedback gained through processes like evaluation to tweak its components to meet changing circumstances and needs. Rather than have a static program, a developmental design is one that systematically incorporates design thinking into the evolutionary fabric of the activities and decision making involved.
Both developmental design and evaluation work together to provide data required to allow program planners to constantly adapt their offerings to meet changing conditions, thus avoiding the problem of having outcomes becoming decoupled from program activities and working with complexity rather than against it. For example, developmental evaluation can determine what are the key attractors shaping program activities while developmental design can work with those attractors to amplify them or dampen them depending on the level of beneficial coherence they offer a program. In two joined processes we can acknowledge complexity while creating more realistic and responsive plans.
Such approaches to design and evaluation are not without contention to traditional practitioners, leaving questions about the integrity of the finished product (for design) and the robustness of the evaluation methods, but without alternative models that take complexity into account, we are simply left with bad planning instead of making it like Eisenhower wanted it to be: indispensable .
A Call to Evaluation Bloggers: Building A Better KT System
Posted: December 9, 2011 Filed under: evaluation, knowledge translation | Tags: American Evaluation Association, Andrea Yip, blogging, CENSE Research + Design, consulting, data visualization, developmental evaluation, evaluation, knowledge translation, Michael Quinn Patton, Social media, tips 1 Comment »
Time To Get Online...
Are you an evaluator and do you blog? If so, the American Evaluation Association wants to hear from you. This CENSEMaking post features an appeal to those who evaluate, blog and want to share their tips and tricks for helping create a better, stronger KT system.
Build a better moustrap and the world will beat a path to your door — Attributed to Ralph Waldo Emerson
Knowledge translation in 2011 is a lot different than it was before we had social media, the Internet and direct-to-consumer publishing tools. We now have the opportunity to communicate directly to an audience and share our insights in ways that go beyond just technical reports and peer-reviewed publications, but closer to sharing our tacit knowledge. Blogs have become a powerful medium for doing this.
I’ve been blogging for a couple of years and quite enjoy it. As an evaluator, designer, researcher and health promoter I find it allows me to take different ideas and explore them in ways that more established media do not. I don’t need to have the idea perfect, or fully formed, or relevant to a narrow audience. I don’t need to worry about what my peers think or my editor, because I serve as the peer review, editor and publisher all at the same time.
I originally started blogging to share ideas with students and colleagues — just small things about the strange blend of topics I engage in that many don’t know about or understand or wanted to know more of. Concepts like complexity, design thinking, developmental evaluation, and health promotion can get kind of fuzzy or opaque for those outside of those various fields.
Blogs enable us to reach directly to an audience and provide a means of adaptive feedback on ideas that are novel. Using the comments, visit statistics, and direct messages sent to me from readers, I can gain some sense of what ideas are being taken up with people and which one’s resonate. That enables me to tailor my messages and amplify those parts that are of greater utility to a reader, thus increasing the likelihood that a message will be taken up. For CENSEMaking, the purpose is more self-motivated writing rather than trying to assess the “best” messages for the audience, however I have a series of other blogs that I use for projects as a KT tool. These are, in many cases, secured and by invitation only to the project team and stakeholders, but still look and feel like any normal blog.
WordPress (this site) and Posterous are my favorite blogging platforms.
As a KT tool, blogs are becoming more widely used. Sites like Research Blogging are large aggregations of blogs on research topics. Others, like this one, are designed for certain audiences and topics — even KT itself, like the KTExchange from the Research Into Action Action initiative at the University of Texas and MobilizeThis! from the Research Impact Knowledge Mobilization group at York University.
The American Evaluation Association has an interesting blog initiative led by AEA’s Executive Director Susan Kistler called AEA365, which is a tip-a-day blog for evaluators looking to learn more about who and what is happening in their field. A couple of years ago I contributed a post on using information technology and evaluation and was delighted at the response it received. So it reaches people. It’s for this reason that AEA is calling out to evaluation bloggers to contribute to the AEA365 blog with recommendations and examples for how blogging can be used for communications and KT. AEA365 aims to create small-bite pockets of information that are easily digestible by its audience.
If you are interested in contributing, the template for the blog is below, with my upcoming contribution to the AEA365 blog posted below that.
By embracing social media and the power to share ideas directly (and done so responsibly), we have a chance to come closer to realizing the KT dream of putting more effective, useful knowledge into the hands of those that can use it faster and engage those who are most interested and able to use that information more efficiently and humanely.
Interested in submitting a post to the AEA365 blog? Contact the AEA365 curators at aea365@eval.org.
Template for aea365 Blogger Posts (see below for an example)
[Introduce yourself by name, where you work, and the name of your blog]
Rad Resource – [your blog name here]: [describe your blog, explain its focus including the extent to which it is related to evaluation, and tell about how often new content is posted]
Hot Tips – favorite posts: [identify 3-5 posts that you believe highlighting your blogging, giving a direct link and a bit of detail for each (see example)]
- [post 1]
- [post 2]
- Etc.
Lessons Learned – why I blog: [explain why you blog – what you find useful about it and the purpose for your blog and blogging. In particular, are you trying to inform stakeholders or clients? Get new clients? Provide a public service? Help students?]
Lessons Learned: [share at least one thing you have learned about blogging since you started]
Remember – stay under 450 words total please!
–
My potential contribution (with a title I just made up): Cameron Norman on Making Sense of Complexity, Design, Systems and Evaluation: CENSEMaking
Rad Resource – [CENSEMaking]: CENSEMaking is a play on the name of my research and design studio consultancy and on the concept of sensemaking, something evaluators help with all the time. CENSEMaking focuses on the interplay of systems and design thinking, health promotion and evaluation and weaves together ideas I find in current social issues, reflections on my practice as well as the evidence used to inform it. I aspire to post on CENSEMaking 2-3 times per week, although because it is done in a short-essay format, find the time can be a challenge.
Hot Tips – favorite posts:
- What is Developmental Evaluation? This post came from a meeting of working group with Michael Quinn Patton and was fun to write because the original exercise that led to the content (described in the post) was so fun to do. It also provided an answer to a question I get asked all the time.
- Visualizing Evaluation and Feedback. I believe that the better we can visualize complexity the more feedback we provide, the greater the opportunities we have for engaging others, and more evaluations will be utilized. This post was designed to provoke thinking about visualization and illustrate how its been creatively used to present complex data in interesting and accessible ways. My colleague and CENSE partner Andrea Yip has tried to do this with a visually oriented blog on health promoting design, which provides some other creative examples of ways to make ideas more appealing and data feel simpler.
- Developmental Design and Human Services. Creating this post has sparked an entire line of inquiry for me on bridging DE and design that has since become a major focus for my work. This post became the first step in a larger journey.
Lessons Learned – why I blog: CENSEMaking originally served as an informal means of sharing my practice reflections with students and colleagues, but has since grown to serve as a tool for knowledge translation to a broader professional and lay audience. I aim to bridge the sometimes foggy world that things like evaluation inhabit – particularly developmental evaluation – and the lived world of people whom evaluation serves.
Lessons Learned: Blogging is a fun way to explore your own thinking about evaluation and make friends along the way. I never expected to meet so many interesting people because they reached out after reading a blog post of mine or made a link to something I wrote. This has also led me to learn about so many other great bloggers, too. Give a little, get a lot in return and don’t try and make it perfect. Make it fun and authentic and that will do.
___
** Photo by digitalrob70 used under Creative Commons License from Flickr
Like this:
What is Developmental Evaluation?
Posted: November 19, 2011 Filed under: complexity, evaluation, innovation, systems thinking | Tags: complexity, developmental evaluation, emergence, Michael Quinn Patton, program evaluation, road trip, strategic learning, systems thinking 5 Comments »
Developmental evaluation (DE) is a problematic concept because it deals with a complex set of conditions and potential outcomes that differ from and challenge the orthodoxy in much of mainstream research and evaluation and makes it difficult to communicate. At a recent gathering of DE practitioners in Toronto, we were charged with coming up with an elevator pitch to describe DE to someone who wasn’t familiar with it; this is what I came up with.
Developmental evaluation is an approach to understanding the activities of a program operating in dynamic, novel environments with complex interactions. It focuses on innovation and strategic learning rather than standard outcomes and is as much a way of thinking about programs-in-context and the feedback they produce. The concept is an extension of Michael Quinn Patton’s original concept of Utilization Focused Evaluation with concepts gleaned from complexity science to account for the dynamism and novelty. While Utilization Focused Evaluation has a series of steps to follow (PDF), Developmental Evaluation is less prescriptive, which is both its strength and its challenge for describing it to people (things I’ve discussed in earlier posts).
So with that in mind, our group was charged with coming up with a way to explain DE to someone who is not familiar with it using anything we’d like — song, poetry, dance, slides, stories and beyond. While my colleague Dan chose to lead us all in song, I opted to go with a simple analogy by comparing DE to a hybrid of Trip Advisor and the classic Road Trip (due to lack of good vocalizing skills).
Trip Advisor has emerged as one of the most popular tools for travellers seeking advice on everything from hotel rooms to airlines to resorts and all the destinations along the way. Trip Advisor is averaging more than 13 million unique visitors per month and, unlike its competitors, focuses on user-generated content to support its service. Thus, your fellow travellers are the source of the recommendations not some professional travel agent or journalist. At its heart are stories of varies tones, detail and quality. People upload various accounts of their stay, chronicling even the most minute detail through photos, links to their blogs, video, and narrative. If you want to get the inside details on what a hotel is really like, check Trip Advisor and you’ll likely find it.
However, like any self-organizing set of ideas, the quality of the content will vary along with the level of reportage and the conclusions will be different depending on the context and experience of the person doing the reporting. For example, if you are a North American who is used to having even the most basic hotel chain offer a room with full-service linens, a bathroom, closet, desk and separate shower, you’ll have a hard time adjusting to something like EasyHotel in Europe.
The Road Trip part (capitalization intended here to denote something different than a regular trip by road), denotes the experience that comes from a journey with a desired destination, but not a pre-determined route and only a generalized timeline. A Road Trip is something that is more than just traveling from Point A to Point B, which is usually accomplished by taking the shortest route, the fastest route or a combination of the two; rather it is a journey. Movies like National Lampoon’s Vacation (and, European Vacation), Thelma and Louise, Harold & Kumar Go to White Castle, and (surprise!) Road Trip all capture this spirit to some effect. I suppose one might even find a more grim example of a Road Trip in the Lord of the Rings Trilogy or The Road.
Road Trips have a long history and are not just a North American phenomenon as this article from the Indian Newspaper, The Hindu reports in some detail:
“Road trips are fun when they are not planned point-to-point. As long as you have accommodation booked, that is enough. Its better not to have agendas; get as spontaneous and adventurous as you can. My friends and I went on a road trip to Goa last year. It was loads of fun as it was the first time we took off on our own without parents. To me, it was more than just a trip with friends. It showed that I could take care of myself and that I was now a grown-up, free to do what I wanted,” says Siddharth, who is doing his engineering.
Chance favours the prepared mind
Road Trip Advisor for Developmental Evaluation
“Road-tripping is a great way to bond with the people you are travelling with and I would strongly recommend it to people. It not only makes you appreciate yourself as an individual but is an amazing experience as you get to meet new people, know different cultures and sample different cuisines. I can never forget biking on sleet, riding though torrential rains, gobbling hot rotis at dhabas, the beautiful snow-capped mountains and guy talk with friends on the trip,” says Dheeraj, who recently went to Ladakh.
all is not hunky dory during these trips. You have to be way about accidents and mishaps. And, realise that freedom comes with responsibility. Says Arjun: “I had borrowed my friend’s bike for the trip, and though it looked good, it gave problems on the foothills of Kodaikanal and we couldn’t do the climb. Being a weekend, there were no mechanics. It helps to know your machine. A passion for road-tripping is not enough. You need to be equipped to take care of yourself also.”
Here, the story parallel is about being prepared. Know evaluation methods, know how to build and sustain relationships and to deal with conflict. A high tolerance for ambiguity and the flexibility to adapt is also important. Knowing a little about systems thinking and complexity doesn’t hurt either. Developmental evaluation is not healthy for those who need a high degree of predictability, are not flexible in their approach, and adhere to rigid timelines. Complex systems collapse under rigid boundary conditions and do evaluators working with such restrictions in developmental contexts.
So why do people do it? “Well, my memories of my favourite road trip were an injured leg, chocolates, beautiful photographs and a great sense of fulfilment,” recalls Arjun.
It is youngsters like these who have transformed road-tripping from just a hobby to an art.
After all, friendship and travel is a potent combination that you can’t say no to.
Systems Thinking and the Design of Empathy
Posted: September 22, 2011 Filed under: complexity, design thinking, systems thinking | Tags: Charles Leadbeater, design, developmental design, developmental evaluation, empathy, organizational change, systems thinking Leave a comment »
Leadbeater's Systems Thinking - Empathy Grid
Scalability is an issue that faces practitioners in systems and design. How do we design systems at scale and if so, what might they look like.
Charles Leadbeater has been on a mission to find ways to make large organizations — particularly those in the social sector – more innovative. Leadbeater, like many social innovators, is hard to pin down to a single title or role. He is at once a researcher, a designer, a systems thinker, and a urbanist. Like most innovators, all and none of these descriptors truly fit.
Leadbeater was in Toronto earlier this week to speak on the issue of innovation in cross-sector collaboration for public good at the MaRS Discovery District. If you’ve seen Leadbeater speak (consider the talks on TED here or elsewehere), you’ll know that you’re in for some English-style self-depricating humour alongside of much about the manner in which people engage in change actions within a system. You’ll also get a lesson in social design, the kind that Victor Papanek advocated for.
To my delight, Leadbeater did not disappoint. Unlike other talks, the value came less from a focused “take home message” and more in a way of conceiving social systems through the combined lens of systems and design thinking (my terms, not his). At the heart of his talk was the challenge we face with building systems and empathy at scale. When things interact, eventually they become understood within a set of boundary conditions and interact, thus making a system. The system in turn begins to establish rules (or rather, the rules determine the system). These emergent properties thus shape the way the system operates or, in social situations, governs or guides the actions of those within it.
The problem is that at certain scales the very factors that create positive social relations, that is those that yield tangible emotional, resource, or informational benefits for one or more parties, get warped under the changes in that scaling. Thus, we have the countless stories of a beloved small business grown in a confined community that becomes a multinational corporation and, in doing so, loses the intimacy and connection to its customers in the process. Companies do this, government organizations show this, and so do cities.
The more one designs for the humans within the system in ways that create meaningful engagement, the greater the empathy. Yet deep empathy is often founded upon intimacy, which is something that is difficult to scale. Leadbeater illustrates the various ways in which firms and cities have addressed this on the graph above. In each case, there are examples that fit. In business for example, there is Ryanair, which embodies a highly structured system with low empathy (top left corner). Opposed to that is the local farmer’s market where one gets to know their grower, experience high mutual empathy, but in a manner that is unique, idiosyncratic and non-systematic in most ways. The challenge is how to design organizations at scale from the cosy-ness of the Farmer’s Market without becoming a Ryanair.
It struck me that the food service industry might be one of the areas where the scalability can be achieved. For example, Starbucks is a gigantic corporation with shops worldwide, yet it still manages to create a very homey, local feel at each one. In the mornings I go to the gym I stop by a location to get a smoothie and my server always remembers my order. At the location near where I take classes, they gave me a free drink because they couldn’t get the computer to take off my 10 cent reusable cup discount. In each location, the benefits were not just in customer service, but in the chit-chat and relationships that I develop with the staff. It’s not like I am speaking to owner-operators at some of the great independent coffee shops around my city, but it is close.
The Starbucks experience was thought through, intentional and thus, by design. We can do this with other systems. The key is whether or not the systems themselves are aware enough to know when they have, indeed, become systems. Starbucks today could not be empathic in exactly the same way as it was when it was a one-shop place at Pikes Market in Seattle. But it can create something similar, which is parallel to Simon’s notion that design is about the science of the artificial.
I’ve been developing and advocating for an approach to creating scale — in time and scope — that I call developmental design. A developmental design approach means shifting and changing over time and designing things in a manner that adjust to the complexities associated with dynamic systems. It brings together complexity, systems, design and the detailed feedback mechanism that comes through developmental evaluation. Leadbeater’s grid helps add to this concept by giving a focus to the development, from one level of empathy to another and one systemic scale to another.
Through thinking in systems and acting through design, perhaps then we can create the kinds of services and organizations that respond to the challenges we face.
And designing for empathy will help us know when we’ve achieved it.
Like this:
Developmental Thinking and Evaluation
Posted: June 20, 2011 Filed under: evaluation, innovation, research | Tags: design, developmental design, developmental evaluation, evaluation, systems thinking 1 Comment »
Think developmentally before evaluating developmentally
Developmental evaluation is difficult to initiate, largely because the thinking behind it is so foreign to normal program planning and reporting. It appears that developmental thinking needs to be in place before one can hope to implement a DE project successfully.
Over the next few days I will be meeting with colleagues working with the Social Innovation Generation Group, Michael Quinn Patton and others who share an interest and wrestle with developmental evaluation (DE) in practice.
Over the course of the last year we have been meeting monthly to discuss our experiences, challenges and learning on the issue of developmental evaluation. Although our group members come from diverse fields — government, academia, non-profit and others — and are focused on projects that range in scope, we all share one common experience: frustration with implementing DE.
Reading through a case study the other night I couldn’t help be see something I’d seen before: the principal barrier to the implementation of DE is that the program, its partners, or the stakeholders associated with the program didn’t individually or collectively function in a manner that supported DE. Whether they actually bought into DE in the first place is also not known, but it seems to me that the two are related.
Developmental thinking about social issues has shown itself in my work to be a linchpin for any progress on developmental evaluation. Commiserating with colleagues in this area, it seems evident to me that assessing for DE is a critical step in the pre-work that needs to come before any evaluation takes place. Without developmental thinking, developmental actions and evaluation is hard to reasonably achieve.
If you do not see your program as one that evolves, but rather just gets bigger, better, stronger, weaker etc.., having real-time evaluation tools will be less useful or perhaps even harmful in the absence of a thinking framework to make sense of the data. Real-time, consultative evaluation, and its utilization-focused actions makes DE stand apart from other approaches to evaluation, even if the methods and tools are similar.
The implications for this assertion in practice are enormous. It means that a DE practitioner cannot be just an evaluator or at least must find others that can work with a program to educate, inspire and contemplate collaboratively about developmental thinking and what it means for a program. It also brings the evaluation function far closer to program planning than evaluators (and program planners) might be used to. It also means holding a willingness to think different, not just implement different thinking. To that end, knowledge of motivation and some sense of how one provokes or creates space for change is also important.
Taken together, we have ourselves a real challenge. The “core competencies” for DE already include qualities like people skills, knowledge of complexity, and communication skills (in addition to fundamental skills in evaluation methods and process implementation), but now we are adding additional ones. Systems thinking, behaviour change, program planning and design are all reasonable skills that would assist an evaluator in doing this work. Nice in theory, but how about in practice? Can we reasonably expect that there are enough people out there with these skills to do it well? Or is this a call for more of a team-science (or rather, team evaluation) approach to evaluation?
Thinking Developmentally About Social Issues
Posted: June 4, 2011 Filed under: Education & Learning, innovation, systems thinking | Tags: complexity, design, developmental design, developmental evaluation, education, feedback, innovation, learning, systems thinking 1 Comment »
Thinking About Development or Being Consumed By Linear Progress?
An often unstated assumption in efforts to provoke change in complex, developmental environments is that people are primed to think in those terms. That might be a false assumption and the reason why concepts like developmental evaluation are so hard to take root.
Difference is hard to grasp. So too, is development. Add the two together and you have a real problem. This is an opinion I’ve formed through my work in complexity science, education and health promotion.
We humans are great at categorizing things. Our eyes are in the front of our head and our bodies are designed for forward movement so we are biologically positioned to look forward. Over the last few centuries, forward has often been linked with progress. Forward imposes a directionality to it and progress imposes an evaluative standard. But what if what we were dealing with in social issues had neither of these assumptions proven right?
Romantic relationships provide an example. Classic literature to pop culture typically present relationship narratives as linear (e.g, characters meet, date, fall in love, get married, buy a home, have kids, grow old together…), which has the effect of imposing an enormous burden of expectation on society that seems out of sync with the manner in which we live as human beings. Why can’t relationships come into being, intensify, draw back, morph, fade and grow simultaneously?
There are many healthy relationships out there and the one described above might only be one example rather than the standard. The problem is that we seek to create standards — best practice – and impose these standards when they might be ill-fitting to the circumstances or context. They don’t take into account development or contextual differences, nor do they appreciate complexity.
We do this with education too, assuming that people all learn the same way. Consider the absurdity of lumping all kids together in grades based on age. Is it reasonable to assume that because you and I were born the same year that we will learn content and evolve our knowledge base about the world in the same way? We put kids (and adults — even graduate students) in rows and talk at them for hours hoping that they will all absorb knowledge and do the same thing with it. That might explain why many students struggle and teachers get frustrated.
Thinking developmentally means attenuating oneself to nuance, punctuated learning, ongoing feedback, and inconsistent behaviour. I don’t blame people for wanting to impose a simple cause-and-effect narrative on the world, but doing so doesn’t mean its useful. As I’ve argued elsewhere, unless we consider changing our thinking we may continue to spend time devising ways to do what systems thinker and management leader Russell Ackoff called “the wrong things, righter”.
It is one thing to complain about this, but another to do it. And it is here that the lens needs to be turned back on us systems thinkers and developmental evaluators or designers. Perhaps it is time to stop assuming that people think this way and shift it towards assuming the converse, yet adding that people have the capability to think this way. Of course many will surprise us by already thinking in terms consistent with development and find it very comforting, but that exception will delight us rather than inspire frustration at the thought of “why doesn’t everyone else think like this?”
With so many social narratives that point towards linear thinking about the world we should not be surprised when we find something akin to psychologist Abraham Maslow‘s often paraphrased sentiment:
When the only tool you own is a hammer, pretty soon every problem resembles a nail
A developmental perspective on things, supported through concepts like design thinking, systems thinking, and creative education and learning is something that isn’t standard in our work, but perhaps should be. Building these muscles, much like a good personal trainer does with his or her client, requires attuning oneself to where a person is at and what kind of space they are in to work.
As this 30-day blogging challenge exploring social innovation, design and creativity continues so too will this discussion. It will be, developmental.
** Photo titled dimensional change by alasis used under Creative Commons license from Flickr
Developmental Design and Human Services
Posted: March 2, 2011 Filed under: behaviour change, complexity, design thinking, evaluation, innovation, public health, research, Social media | Tags: climate change, complexity, design, design thinking, developmental design, developmental evaluation, innovation, program planning, Social media, strategic foresight 5 Comments »
The ever-changing landscape
One of the principal challenges for program evaluators and researchers is overcoming design limitations imposed by programs that fail to account for time and development. What might it look like if we took this path and what does it mean to engage in developmental design?
If you are like me, climate change scares you. I live in Canada, so in some ways (particularly these cold winter days), the thought of having the season be a little warmer has some appeal. But complex systems don’t work quite that way as there are intense costs projected with the privilege of having more days of the year wearing a light jacket or shorts than an overcoat or parka. What makes climate change an interesting example from the perspective of service programming, design and evaluation is that it provides a look at a real-world way to conceive of development over concepts like improvement and takes change in a whole new direction.
Humans are rather paradoxical creatures in that we are both attuned to moving forward (consider the design of the body: everything is oriented to one direction) and a perfect example of a developmental system.
A developmental system is one that evolves and adapts to changing inputs and transforms itself over time as more information is added to it (i.e., it is a complex adaptive system). From a programming standpoint, it means things don’t “get better” or “improve” per se — those are value judgements places by us — but rather, they build adaptive capacity.
Concepts like developmental evaluation, introduced and discussed in this space before, are ways to respond to this from an evaluation standpoint. DE provides a method of feedback generation that can enable programs to adapt and evolve by using the principles of complexity science with program evaluation methods to create a platform to detect and monitor emergent conditions and support innovation. And while there are some questions to ask of a program to see if it is suited to a developmental evaluation, we often forget to ask whether the program was designed to develop in the first place. What if we placed that at the centre of our discussion and started with development in mind?
My previous post looked at designing for time and space, but designing for development takes this one step further. Social media and technology-delivered program spaces provide an example of an environment where development is most obvious. Facebook was designed to expand and evolve, although one might challenge how well they’ve really developed. If you consider how effective, long-lasting software and services survive, they develop over time. In some cases, this development was designed into the process. Many open source software platforms are designed with this in mind — the Firefox browser and even Google’s Chrome are examples of tools that were built to be developed on. The originators designed the basic foundation with the idea that they would evolve into something else.
This doesn’t happen very often with human services. There are few programs that are designed with development in mind. When it is acknowledged that things will change, it is done so reluctantly. Program in this context refers to any organized effort to change behaviour and produce products for human need. In public health, the further irony is that programs aimed at changing behaviour — whether it is supporting healthy eating, smoking cessation, mental health promotion or others — are often designed with rigid controls built in. We develop manuals, create ‘best practice guidelines’, amass evidence and create toolkits that can be applied to any circumstance, without attention to context or adaptation.
Indeed, when you relax these controls, many get concerned.
Having conducted a few social media trainings and presentations over the years, the most consistent question I am asked by those in public health is: how do I control the message. The answer is: you don’t. This can lead to questions about evaluation, which gets into problems of research design and trusting the findings, because research typically applies rigid controls for quality assurance.
With social media, what can be done is to use a process of developmental design by engaging with the audience/client/public in an authentic manner with the explicit thought that the program that launches today will not be the one that people engage with in a year, or a month or sooner. Support this evolution through developmental evaluation (which I would include as a part of the developmental design process) and you’ll have a feedback mechanism that encourages shifts over time.
Developmental design takes into account the complexity of the environment in which a product or service takes place and enlists a continued process of engagement with stakeholders over time — a true relationship (which is why social media can serve as a good example). Rather than take a static design brief, a living design brief would be used and constantly revisited and tweaked over time. Paying attention to changes in the brief over time would also enable program developers to detect weak signals that could precede large shifts in behaviour and potentially support strategic foresight and planning. Developmental design, as I’ve conceived of here, is attuned to complexity and innovation in human systems and designs for it and adapts with it, rather than assuming the opposite.
Applying developmental design may get us past the inevitable square-peg-round-hole problem that many evaluators, program planners and policy makers find themselves in as they seek to get greater value from their programs and demand more return on their investments. Evaluation and research is sought as the means to do it and with programs designed for evolution from the start, perhaps we won’t be surprised when the metaphorical ice sheets start to fall apart (as seen above) and see it as a developmental step to a new reality.
** Photo Nature Antarctica 17 by Christian Revival Network used under Creative Commons Licence.
Grasping Developmental Evaluation
Posted: February 8, 2011 Filed under: complexity, Education & Learning, evaluation, innovation | Tags: complexity, developmental evaluation, evaluation, innovation 2 Comments »
Is this what you ordered?
Developmental evaluation is gaining interest in the evaluation, service and policy communities, but do these newfound fans know what they are getting when they ask for it? Grasping the fundamentals of development as opposed to growth is one of the foundations to its utility.
I recently concluded four days of meetings with a peer group — a community of practice – on Developmental Evaluation. The CoP brings together a diverse group of professionals who are seeking to learn more about DE, apply it to their work, and to advance thinking and scholarship on the idea. We are the true-believers. At the same time, we are evaluators, policy makers, and program leaders who are pragmatic and see that DE isn’t for every situation and indeed, there are many situations where DE is not appropriate.
In his book on the subject, Michael Quinn Patton outlines a few questions that can help frame questions to ask when considering whether DE is a suitable approach — particularly with respect to when you hope to use the results:
- What is being developed now?
- What do the results of what has been developed mean now?
- What are the next steps now? What is the next stage of development now?
These simple questions can help users of DE determine whether it is appropriate. In particular, the bigger question is: are you able to use information gleaned from DE now? If not, then it might be you are not wrestling with a developmental issue, but one more suited to formative or summative evaluation. You may want to know whether a program worked or how it worked, not necessarily how it is developing.
One of the analogies that I find fits DE is that of purchasing something on the Internet. Yes, you can read all about the thing, consider the specs on its dimensions, and see a picture, but when it arrives to your door there is often a surprise when you open the box and are actually holding it , seeing it before you. Sometimes that is a pleasant surprise, sometimes not, or maybe its just different, but quite often it is not the same as what you thought it was when you clicked “buy” online.
DE is truly suited to situations where you seek to develop a program, evolve it, and adapt it to changing circumstances. It requires a different mindframe around the problem context and the goals of the program than the ordinary one that is focused on growth, change over time and improvement. A development is an adaptation, while an improvement is a judgement based on pre-established criteria. Because DE is suited to programs operating in great complexity, establishing the context in advance (i.e., creating controls in traditional research parlance) is not possible.
For those looking to innovate, to evolve their programs operating in complex conditions, and to learn-as-you-go in a manner that enables on-the-fly decision making, then DE is for you. If you want evaluators to go off, bring the data, and give you results, you might want to try something else.
** Photo package! by uzbeckistan used under a Creative Commons License from Flickr
Systems Thinking and Design: A Case for Egypt?
Posted: January 30, 2011 Filed under: art & design, complexity, design thinking, evaluation, Systems science, systems thinking | Tags: design, design thinking, developmental evaluation, Egypt, evaluation, politics, systems thinking 1 Comment »Politics provides a great analogy for why systems thinking and design fit together and how effective “design” and systems thinking work so closely together. It’s time that our politicians and policy makers start considering the role of design and systems thinking a little more and Egypt provides a great example of what happens when those areas come together.
Designing (Building) a New Egypt?
Over the past month I’ve done a lot of reading on the role of design and the culture of designers. The reasons are many, but mostly because I see the challenges that we face as a society as ones of poor design and an inability to see systems, think about them clearly, and translate that into action. This last part is really where design comes into play.
Take the current situation in Egypt, that is in many ways a design problem. Many years ago, Egyptians were comfortable — if not always happy — to accept a government designed to be under the heavy influence of one person. Despite the flaws of that leadership, there was acceptance and general, if somewhat muted, support for that model for the last part of the 20th century and the early start of this century. The recent events in Tunisia showed Egyptians that there were alternatives to their current model of politics and that the people could design a new leadership. The past few days have seen a remarkable chain of events that represent the culmination of desire of Egypt’s people for change.
Design thinking provides a lens for viewing problems and developing contextual solutions — or new situations. The exploration of the problem and developing new solutions using design thinking involves a number of steps. This usually includes such steps as:
- Define the scope, scale and context of the problem at hand;
- Research the problem and determine causes, consequences, alternatives and opportunities related to that problem;
- Start working on developing possible options — ideate — in ways that include the wild and outlandish to ensure that there is sufficient opportunity to build on ideas that might encompass the fullest possible perspective on the issue, even if such ideas may seem impractical;
- Prototype some new options. In the case of political systems, perhaps trying some new ways to organize parts of the government, shift the leadership structure or conduct local experiments to try new models of governance in relatively safe environments;
- Evaluate the implementation of the prototype and incorporate the findings into successive models and then re-implement them in the form of new prototypes. This rapid-cycle prototyping on small scale experiments enable a safe-fail culture to form rather than aim for the impractical fail-safe models that almost never work in complex systems;
- Implement and repeat. Take the lessons learned from taking a utilization-focused evaluation (PDF) approach or a developmental evaluation approach (or a combination), implement the necessary changes and repeat.
The manner by which the outcomes and implementation of the new model are assessed can be viewed by taking a systems thinking perspective, largely because what is being designed is indeed a system. By considering the boundaries, the interconnections between civil society and government, and by articulating intention in guiding the change, it is possible to design a new political system for one of the world’s oldest societies in a manner that honours the past and creates a compelling, healthy future.
Change isn’t usually so straightforward, but that’s often because there isn’t the planning and process in place for change. Now is a perfect time to bring that about as Egyptians struggle along with their Northern African neighbours in Tunisia to find ways to bring their intention to bear on the way their country is governed and, in doing so, create one of the most significant opportunities for design and systems thinking in either of those nations’ histories.
