Evaluating Health Promotion Social Media Strategies for Public Health Impact
Posted: April 11, 2013 Filed under: eHealth, evaluation, social media, systems thinking | Tags: developmental evaluation, evaluation, health promotion, public health, social media, strategy Leave a comment »
How is social media stacking up?
I recently spoke at an interactive workshop presentation at the 2013 Ontario Public Health Convention (TOPHC) looking at social media use in public health and the strategies available for evaluating those strategies in practice. The talk was focused on the tools, methods and approaches and the inherent challenges in dealing with a dynamic social communication environment.
Here are the slides from that presentation.
Evaluating Health Promotion Social Media Strategies for Public Health Impact
Image: Shutterstock (used under licence)
The Importance of Journalism to Public Health: 10 Years After SARS How Are We Doing?
Posted: April 4, 2013 Filed under: public health, social media, systems thinking | Tags: developmental design, developmental evaluation, evidence, health communications, health reporting, Helen Branswell, journalism, Julie Leask, media, public health, social media, systems thinking, Wayne MacPhail 2 Comments »
Risk communication in public health with Julie Leask
If a health scare manifested itself in the world and there were no journalists to cover the story, what would the impact on the public be?
That is a question that lingered with me throughout the start of the 2013 Ontario Public Health Convention (TOPHC) which began with a morning dedicated to improving public health communication. Opening up the conference was a series of linked keynote presentations from a risk communications researcher (Julie Leask); a former newspaper editor, journalism professor and social media advocate (Wayne MacPhail), and one of Canada’s leading health specialist reporters (Helen Branswell).
The Academic’s Perspective
Keynote speaker Julie Leask (pictured above) and her colleague Dr. Claire Hooker (a good friend of mine) have been looking at the ways journalists engage in risk communication with the public on matters of public health from immunization to SARS to understanding the health priorities of professionals. In 2010 they published a paper looking at how the media covers health topics and argued that the health professions need to be aware of how stories are made, communicated and to be an active partner with reporters if they are to have positive impact in moments of health scares.
“It’s too late when the crisis comes up” – Julie Leask speaking on the need for public health to get engaged with the public using social media
In a previous post I wrote about how journalism is the fourth estate of medicine and public health. Journalists are the storytellers that the public listen to and are charged with looking at a problem from many perspectives to develop that coherent narrative that speaks to their audience. These are qualities that most scientists and public health professionals don’t bring to their jobs, nor are they always expected to or even should. As such, journalists play an important role for this very reason.
Nonetheless, the health sector has an uneasy relationship with journalism. Health professionals – particularly researchers — poorly understand the world of journalists and sometimes view the profession with suspicion. Julie Leask and her colleagues have found this to be the case, but argue that it is no reason to shy away from engaging the public using the tools that are comfortable to journalists. She spoke to the invaluable role of specialist health journalists in acting not only as producers of high quality health content in the news, but also guardians against low quality content making into press. In speaking to her research, she pointed out that specialist health journalists help educate their peers and editors on health issues, which are often complex and require more than a passing understanding of context to communicate well, as key gatekeepers for quality in the health landscape.
The Editor’s Perspective
To this end, Wayne MacPhail, a former editor of the Hamilton Spectator, argued that public health has a near ethical imperative (my choice of term) to be in the social media space to not only promote good health, but counter and challenge myths and misinformation. This isn’t some naive pronouncement that we’ll eliminate the snake oil sales or quackery that proliferates in the public sphere and media, but rather a simple observation that we have no chance of making impact if we are not even engaged in the space at all.
Like Leask, MacPhail says that it’s too late to engage the public when a health crisis comes up and that public health needs to be in the conversation stream before that happens.
The Reporter’s Perspective
Reporting through SARS to today: Helen Branswell
Helen Branswell, a reporter from The Canadian Press, rounded out the panel and spoke frankly about the dwindling resources and rapidly changing landscape in journalism. She was on the front lines of reporting the 2003 SARS outbreak and showed a picture taken during that time of an empty newsroom and remarked how that the scene is the same now only for different reasons (limited budgets due to decreased ad revenue and the related shift to digital information on the web being two such reasons, among others).
Branswell paints a bleak picture of the present and future in many areas of health journalism. Stories are increasingly being covered by general reporters who may treat the story the same as they would a traffic incident, political story, or crime; journalists who are unlikely to know the context and details that are critical to communicating the nuances present in health matters. Interns are replacing some full time or veteran reporters in the newsroom and there are only a handful of specialists in practice.
Pressures from time, budget and competing interests in the newsroom are all contributing to an environment where quality health reporting is threatened.
What Next?
I asked the panel what they thought public health should do to ensure that the healthy stories are reported well and there was little answers. Helen Branswell said, truthfully and somewhat cheekily: “buy newspapers”. She reminded us that we should be paying for the quality content and supporting good journalism in practice if we want it to survive, which is hard to argue against.
But that alone will not do all the work needed to preserve good journalism. I spoke to another conference attendee, a formally trained journalist who is now working with a research firm, about the ways in which journalists have helped other organizations craft their messages and engaging the public citing the Calgary Police Service’s social media team as an example. This pointed to ways in which journalists can make a difference in matters of public health and social services.
Yet, what about investigative journalism? What about the potential conflicts that come from being paid to report on issues that might be critical of the organization who does the paying (e.g., Ministries of Health, Departments of Public Health, Universities and colleges etc..)? This model doesn’t solve that, but it is at least another option.
Yet, the examples from public health taking this challenge of working with journalists up are few. Many still believe that social media is another means of broadcasting, which misses the mark. Others still view social media, journalism, engaging with the public through the media, with suspicion on the grounds that much of the work out there is not evidence based.
But what evidence did we have when SARS hit us 10 years ago? We had lots of epidemiological data on infectious disease, but that was only part of the story. Many of the leading health scientists were adapting their models, creating new ones and only after the disease left did we really have a full sense of what happened. We learned as we went.
This is what social media is all about, too. The lessons from major health events — disasters, outbreaks, and pandemics — parallel social media. It is innovation space at its clearest and thus there is an imperative to view it as innovation space with the tools and lenses that best support movement within complex adaptive system. From a communications standpoint, social media and the tools of modern journalism (and the style of communication they employ) are one thing to consider. Developmental design and evaluation are also among these tools combined with systems thinking.
Linear thinking and action will not work in a complex system and as this panel pointed out, there is much reason to be concerned if we are not prepared to communicate and support those that communicate well in such times when — not if — they come back.
Ten years after SARS how better off are we? And if we are better, how are we communicating that to the public?
The Job Market Metric In Education
Posted: December 30, 2012 Filed under: design thinking, education & learning, evaluation, innovation | Tags: complex systems, complexity, contemplative inquiry, continuing medical education, design thinking, developmental evaluation, disruption, education, educational design, evaluation, foresight, jobs, journalism, learning, learning organization, mindfulness, path dependence, professional education, resilience, teaching, university 2 Comments »
Post-secondary and continuing education is continuing to be rationalized in ways that are transforming the very foundation of the enterprise. Funding is a major driver of change in this field: how much is available, when it flows, where it comes from, what is funded, and who gets the funding are questions on the minds of those running the academy.
At the centre of the focus of this funding issue is the job market. Training qualified professionals for the job market in various forms has been one of the roles a university has played for more than a century. Now that role has become central.
Let’s consider what that means and what it could do in shaping the various possible futures of the university. This second in a series looking at the post-secondary and continuing education focuses on the metrics of jobs.
“What are all these people going do?”
The employability of graduates is now the holy grail of education industry statistics. Earlier this year I was sitting on the stage at an academic convocation with a senior colleague staring out at a sea of soon-to-be-graduates when he leaned over and asked the question quoted above. Staring at a sea of masters and doctoral graduates numbered in the hundreds and knowing that this ceremony was held twice per year, the question stuck and remains without an answer.
Maybe there were enough jobs for that cohort, but this process gets repeated twice each year at universities around the world and each year that I’ve been a professor those numbers (of graduates) seem to go up. Some of our programs in the health sciences are admitting three times the number of students than they were just ten years ago. There is much demand for education (as judged by departmental applications), but are there jobs demanding this kind of education in its current form?
Yes, the Baby Boom is moving into an age of retirement and increasing needs for health services, but do we need to graduate 80+ Physical or Occupational Therapists to meet this need this year? Do we need a few dozen more epidemiologists or health promotion specialists to add to the pool? How about psychologists or social workers: how many of those do we need? The answer from my colleagues in these fields is: We don’t know.
Chasing the Wind
Jobs are a red herring. It’s one thing to have a job, but is it the job that you trained for? (And is having that job even a reasonable goal?) Being employed is not the same as building a career. What if you were trained perfectly for a job that no longer existed? Imagine a Blacksmith in the 20th century or a Bloodletter. These questions are not asked, nor is much asked about quality of education relative to the pressures of recruitment, cost-cutting and educational rationalization. Most of us don’t know what quality education is in real terms because we are measuring it (if we are measuring anything at all besides jobs) by standards set for the jobs of the past, not the future (or even the present?).
“Skate where the puck is going, not where it’s been.” – Wayne Gretzky
Jobs are living things and very few in 2013 will resemble what they did even 10 years ago. The citizens of the developing world are entering this rapidly changing job market ready for change (See also McKinsey Global Institute report on future of work in advanced economies) because they don’t have the old ways to rely on. They are primed for change and if professional education is to meet the needs of a changing world, it needs to change too. It means getting serious about learning.
If education is rationalizing itself to focus more on jobs, then it also needs to get serious about clarifying what jobs mean, defining what ‘success’ looks like for a graduate, and whether those jobs are designed for where the proverbial puck is now or for where it is going.
Disruptive Learning / Disturbed Education
“The Only Thing That Is Constant Is Change -” ― Heraclitus
I’ve pointed out that learners have an uneasy relationship with learning principally because it means disrupting things. This is a topic I’ll be covering in greater depth in a future post, but if one considers how our social, economic, and environmental systems are changing it is not unreasonable to call this the age of disruption .
Change in complex systems is often logarithmic, not linear. It may be massively punctuated like a Lévy Flight or it could be closer to a random walk. In environments with a change coefficient that is large the level of attention must be more fine-grained than 5-year reviews. It requires developmental evaluation methods and learning organizations, not just conventional approaches to generating and assessing feedback. It requires mindful attention and contemplative inquiry to guide a regular reflective practice if one is to pay attention to the subtleties in change that could have enormous impact.
For example, if journalists and news media waited every five years to assess the state of their profession, they would have missed out on Twitter and come late to blogging, two of their (now) powerful sources of competition and tools of the trade. Some have waited, which is why they are no longer around. Metrics for journalism education today might consider the amount of exposure and proficiency in social media use, digital photography, use of handheld tools for communication, and real-time reporting skills. Metrics of the past might focus on newspapers and radio broadcasting. Which mindset, skillset and toolset would you rather be trained in today?
Questions for educators, learners (and evaluators):
Whether health sciences, journalism, human services or any field, what might some questions be that can help determine the role of job training in professional education? Here are five starters:
1. What is the state of your profession right now and are you training people for existing in this state? Are you preparing people for the next evolution?
2. Where is your field of practice going? What are the possible futures for your profession in the next 5, 10, and 20 years? Will it still exist? Are you a blacksmith looking for more horses in the automobile age or Steve Jobs waiting to attract people to a new graphical user interface?
3. Is your mindset, skillset or toolset in need of re-consideration? Does it still do the job you’ve hired it to do?
4. What do people need that your skills can help with? What unfilled needs and expectations are there in the world that your mindset, skillset and toolset could solve?
5. What would happen if your field of practice disappeared? How else could you apply what you know to making the contribution you wish to make and earn a living? What other skills, tools and ways of thinking would you need to adapt?
Design thinking can greatly help shape the way that one conceives of a problem, works through possible options, and develops prototypes to address the needs of the present and the future. Foresight methods help lay additional context for design and systems thinking by providing ways to anticipate possible futures for any given field. Lastly, knowing what the state of things are now and how they got to where they are now can help determine the path dependencies that education may have fallen into.
We can’t change what we don’t see and better foresight, hindsight and present sight is critical to better ensuring that education outcomes are not imagined, but based on something that can actually improve learning.
Evaluating Social Innovation For Social Impact
Posted: November 15, 2012 Filed under: complexity, design thinking, education & learning, emergence, evaluation, knowledge translation, systems thinking | Tags: design, design thinking, developmental design, developmental evaluation, evaluation, evidence, Helsinki Design Lab, knowledge exchange, knowledge translation, research design, research methods, social impact, social innovation 1 Comment »
How do the innovation letters line up?
Earlier this week I has the pleasure of attending talks from Bryan Boyer from the Helsinki Design Lab and learning about the remarkable work they are doing in applying design to government and community life in Finland. While the focus of the audience for the talks was on their application of design thinking, I found myself drawn to the issue of evaluation and the discussion around that when it came up.
One of the points raised was that design teams are often working with constraints that emphasize the designed product, rather than its extended outcome, making evaluation a challenge to adequately resource. Evaluation is not a term that frequents discussion on design, but as the moderator of one talk suggested, maybe it should.
I can’t agree more.
Design and Evaluation: A Natural Partnership
It has puzzled me to no end that we have these emergent fields of practice aimed at social good – social finance and social impact investing, social innovation, social benefit (PDF)– that have little built into their culture to assess what kind of influence they are having beyond the basics. Yet, social innovation is rarely about simple basics, it’s influence is likely far larger, for better or worse.
What is the impact being invested in? What is the new thing being created of value? and what is the benefit and for whom? What else happened because we intervened?
Evaluation is often the last thing to go into a program budget (along with knowledge translation and exchange activities) and the first thing to get cut (along with the aforementioned KTE work) when things go wrong or budgets get tightened. Regrettably, our desire to act supersedes our desire to understand the implication of those actions. It is based on a fundamental idea that we know what we are doing and can predict its outcomes.
Yet, with social innovation, we are often doing things for the first time, or combining known elements into an unknown corpus, or repurposing existing knowledge/skills/tools into new settings and situations. This is the innovation part. Novelty is pervasive and with that comes opportunities for learning as well as the potential for us to good as well as harm.
An Ethical Imperative?
There are reasons beyond product quality and accountability that one should take evaluation and strategic design for social innovation seriously.
Design thinking involves embracing failure (e.g, fail often to succeed sooner is the mantra espoused by product design firm IDEO) as a means of testing ideas and prototyping possible outcomes to generate an ideal fit. This is ideal for ideas and products that can be isolated from their environment safely to measure the variables associated with outcomes, if considered. This works well with benign issues, but can get more problematic when such interventions are aimed at the social sphere.
Unlike technological failures in the lab, innovations involving people do have costs. Clinical intervention trials go through a series of phases — preclinical through five stages to post-testing — to test their impact, gradually and cautiously scaling up with detailed data collection and analysis accompanying each step and its still not perfect. Medical reporter Julia Belluz and I recently discussed this issue with students at the University of Toronto as part of a workshop on evidence and noted that as complexity increases with the subject matter, the ability to rely on controlled studies decreases.
Complexity is typically the space where much of social innovation inhabits.
As the social realm — our communities, organizations and even global enterprises — is our lab, our interventions impact people ‘out of the gate’ and because this occurs in an inherently a complex environment, I argue that the imperative to evaluate and share what is known about what we produce is critical if we are to innovate safely as well as effectively. Alas, we are far from that in social innovation.
Barriers and Opportunities for Evaluation-powered Social Innovation
There are a series of issues that permeate through the social innovation sector in its current form that require addressing if we are to better understand our impact.
- Becoming more than “the ideas people”: I heard this phrased used at Bryan Boyer’s talk hosted by the Social Innovation Generation group at MaRS. The moderator for the talk commented on how she had wished she’d taken more interest in statistics in university because they would have helped in assessing some of the impact fo the work done in social innovation. There is a strong push for ideas in social innovation, but perhaps we should also include those that know how to make sense and evaluate those ideas in our stable of talent and required skillsets for design teams.
- Guiding Theories & Methods: Having good ideas is one thing, implementing them is another. But tying them both together is the role of theory and models. Theories are hypotheses about the way things happen based on evidence, experience, and imagination. Strategic designers and social innovators rarely refer to theory in their presentations or work. I have little doubt that there are some theories being used by these designers, but they are implicit, not explicit, thus remaining unevaluable and untestable or challenged by others. Some, like Frances Westley, have made theories guiding her work explicit, but this is a rarity. Social theory, behaviour change models and theories of discovery beyond just use of Rogers’ Diffusion of Innovation theory must be introduced to our work if we are to make better judgements about social innovation programs and assess their impact. Indeed, we need the kind of scholarship that applies theory and builds it as part of the culture of social innovation.
- Problem scope and methodological challenges with it. Scoping social innovation is immensely wide and complicated task requiring methods and tools that go beyond simple regression models or observational techniques. Evaluators working social innovation require a high-level understanding of diverse methods and I would argue cannot be comfortable in only one tradition of methods unless they are part of a diverse team of evaluation professionals, something that is costly and resource intensive. Those working in social innovation need to live the very credo of constant innovation in methods, tools and mindsets if they are to be effective at managing the changing conditions in social innovation and strategic design. This is not a field for the methodologically disinterested.
- Low attendance to rigor and documentation. When social innovators and strategic designers do assess impact, too often there is a low attention to methodological rigor. Ethnographies are presented with little attention to sampling and selection or data combination, statistics are used sparingly, and connections to theory or historical precedent are absent. Of course, there are exceptions, but this is hardly the rule. Building a culture of innovation within the field relies on the ability to take quality information from one context and apply it to another critically and if that information is absent, incomplete or of poor quality the possibility for effective communication between projects and settings diminishes.
- Knowledge translation in social innovation. There are few fora to share what we know in the kind of depth that is necessary to advance deep understanding of social innovation, regularly. There are a lot of one-off events, but few regular conferences or societies where social innovation is discussed and shared systematically. Design conferences tend towards the ‘sage on the stage’ model that favours high profile speakers and agencies, while academic conferences favour research that is less applied or action-oriented. Couple that with the problem of client-consultant work that is common in social innovation areas and we get knowledge that is protected, privileged or often there is little incentive to add a KT component to the budget.
- Poor cataloguing of research. To the last point, we have no formalized methods of determining the state-of-the-art in social innovation as research and practice is not catalogued. Groups like the Helsinki Design Lab and Social Innovation Generation with their vigorous attention to dissemination are the exception, not the rule. Complicating matters is the interdisciplinary nature of social innovation. Where does one search for social innovation knowledge? What are the keywords? Innovation is not a good one (too general), yet neither is the more specialized disciplinary terms like economics, psychology, geography, engineering, finance, enterprise, or health. Without a shared nomenclature and networks to develop such a project the knowledge that is made public is often left to the realm of unknown unknowns.
Moving forward, the challenge for social innovation is to find ways to make what it does more accessible to those beyond its current field of practice. Evaluation is one way to do this, but in pursuing such a course, the field needs to create space for evaluation to take place. Interestingly, FSG and the Center for Evaluation Innovation in the U.S. recently delivered a webinar on evaluating social innovation with the principle focus being on developmental evaluation, something I’ve written about at length.
Developmental evaluation is one approach, but as noted in the webinar : an organization needs to be a learning organization for this approach to work.
The question that I am left with is: is social innovation serious about social impact? If it is, how will it know it achieved it without evaluation?
And to echo my previous post: if we believe learning is essential to strategic design we must ask: How serious are we about learning?
Tough questions, but the answers might illuminate the way forward to understanding social impact in social innovation.
* Photo credit from Deviant Art innovation_by_genlau.jpg used under Creative Commons Licence.
The Forward Orientation Problem With Complexity
Posted: June 26, 2012 Filed under: complexity, evaluation, social systems, systems science | Tags: complexity, contemplative inquiry, developmental design, developmental evaluation, mindfulness, mindset, perception, program planning, science, social innovation, time 6 Comments »
The human body is oriented towards forward motion and so too are our institutions, yet while this helps us move linearly and efficiently from place to place, it may obscure opportunities and challenges that come from other directions such as those posed by complexity. Thinking about and re-orienting our perceptions of who we are and where we are going might be the key to understanding and dealing with complexity now and in the future.
When heading out into the turbulent waters that face us we humans tend to look straight ahead and press forward. Our entire physical being and that of all mammals is aimed at facing forward. We look forward, walk forward and this often means thinking forward.
Doing this predisposes us to seeing problems ahead of us or behind us, but is less useful when what challenges us is positioned elsewhere. For this reason, fish and birds, with their eyes on the side of their head, are able to adapt to challenges from nearly any direction quickly. It also allows them to fly/swim in flocks/swarms/schools and operate with high degrees of coordination on a large scale.
These are skills that are useful for handling the social problems that are complex in nature and require mass action to address. But, we don’t have eyes on the side of our head and we tend to look forward or backward to orient ourselves and our activities.
One way this expresses itself in our perceptions of time. Thor Muller, writing in Psychology Today online, highlighted how our perceptions of time influence the way we handle appointments and punctuality with modern technology. Citing the work of anthropologist Edward T. Hall (although mistakenly referring to Manhattan Project contributor Edward Teller), Muller points to the differences in perceived time across cultures and the way that plays out in our treatment of time and technology used to “manage” it and the complexity of everyday life. Monochronistic and polychronistic time orientations matter to whether you see time as a linear, quantifiable phenomenon or a more non-linear, contextual one. One allows you to “bank” time while the other perception deals more with the present moment, less dependent on forward-backward thinking.
Western society and the technologies developed within it are oriented primarily towards dealing with a monochronistic form of time. This works well when patterns, problems and situations have a linear, ordered set of circumstances to them. The cause-and-effect world of normal science fits within this worldview.
Complexity is non-linear and not easily defined in cause-and-effect terms and conditions. Two-dimensional space doesn’t capture complexity the way it can for linear situations. It also means thinking solely in forward and back terms is problematic.
An example of where this comes to conflict is in program planning and evaluation. Traditional evaluation methods and metrics are set up for looking at programs that are planned to start and end with impacts developed and detected in between. This implies a certain level of consistency in the conditions in which that program operates. This control and measure aspect of evaluation is part of the hallmark features of scientific inquiry.
For programs operating in environments of great change and flux, this is a faulty proposition. We cannot hold constant the environment for starters. Secondly, feedback gained from learning about the program as it proceeds is critical to ensuring adaptation and promoting resilience in the face of changing conditions. In these cases, failure to act and adapt on the go may result in a program failing catastrophically.
This is where developmental evaluation comes in. Developmental evaluation works with these conditions to generate data in a manner that programs can make sense of and use to facilitate strategic adaptation rather than simply reacting to changes. As the name suggests, it promotes development rather than improvement.Developmental design is the incorporation of this feedback into an ongoing program development and design process.
Both developmental design and evaluation require ways of seeing the world beyond forward/backward. This seeing comes from understanding where one’s position is in the first place and that requires methods of centring that take us into the world of polychronistic time. One example of a strategy that suits this approach is mindfulness programming. Mindfulness-based programs have shown remarkable efficacy in healing and health interventions aimed at stress reduction across conditions. Mindfulness techniques ranging from meditation to contemplative inquiry (video) brings focus to the present moment away from an orientation towards linear trajectories of time, thought and attention.
Some forms of martial arts promote attentive awareness to the present moment by training practitioners in strategies that are focused on simple rules of engagement, rather than just learning techniques for defence.
These approaches combine inward reflection — reflective practice — with an openness to the data that comes in around them without imposing an order on it a priori. The orientation is to the data and the lessons that come from it rather than its directionality or imposing values on what the data might mean at the start. It means slowing down, contemplating things, and acting on reflection not reacting based on protocol. This is a fundamental shift for many of our activities, but may be the most necessary thing we can focus on if we are to have any hope of understanding, dealing with, and adapting to complexity.
All the methods and tools at our disposal will not help if we cannot change our mindset and orientation — even in the temporary — to this reality when looking at complexity in our work. One of complexity’s biggest challenges right now is that it is seductive in accounting for the massive, dynamic sets of conditions we face every day, yet it lacks methods beyond evaluation to do things with it. The irony of mindfulness and contemplative approaches is that they are less about acting differently and more about seeing things in new ways, yet it is that orientation that is the key to making real change from talking about change. It is the design doing that comes with design thinking and the systems change from systems thinking.
Evaluation and Design For Changing Conditions
Posted: June 5, 2012 Filed under: complexity, design thinking, emergence, evaluation, innovation, systems thinking | Tags: complexity, developmental design, developmental evaluation, evaluation, IDEO, measurement, Michael Quinn Patton, social innovation, Tim Brown, utilization-focused evaluation 3 Comments »
Growth and Development
The days of creating programs, products and services and setting them loose on the world are coming to a close posing challenges to the models we use for designing and evaluation. Adding the term ‘developmental’ to both of these concepts with an accompanying shift in mindset can provide options moving forward in these times of great complexity.
We’re at the tail end of a revolution in product and service design that has generated some remarkable benefits for society (and its share of problems), creating the very objects that often define our work (e.g., computers). However, we are in an age of interconnectedness and ever-expanding complexity. Our disciplinary structures are modifying themselves, “wicked problems” are less rare
Developmental Thinking
At the root of the problem is the concept of developmental thought. A critical mistake made in comparative analysis — whether through data or rhetoric — is one that mistakenly views static things to moving things through the same lens. Take for example a tree and a table. Both are made of wood (maybe the same type of wood), yet their developmental trajectories are enormously different.
Wood > Tree
Wood > Table
Tables are relatively static. They may get scratched, painted, re-finished, or modified slightly, but their inherent form, structure and content is likely to remain constant over time. The tree is also made of wood, but will grow larger, may lose branches and gain others; it will interact with the environment providing homes for animals, hiding spaces or swings for small children; bear fruit (or pollen); change leaves; grow around things, yet also maintain some structural integrity that would allow a person to come back after 10 years and recognize that the tree looks similar.
It changes and it interacts with its environment. If it is a banyan tree or an oak, this interaction might take place very slowly, however if it is bamboo that same interaction might take place over a shorter time frame.
If you were to take the antique table shown above, take its measurements and record its qualities and come back 20 years later, you will likely see an object that looks remarkably similar to the one you lefty. The time of initial observation was minimally relevant to the when the second observation was made. The manner by which the table was used will have some effect on these observations, but to a matter of degree the fundamental look and structure is likely to remain consistent.
However, if we were to do the same with the tree, things could look wildly different. If the tree was a sapling, coming back 20 years might find an object that is 2,3,4 times larger in size. If the tree was 120 years old, the differences might be minimal. It’s species, growing conditions and context matters a great deal.
Design for Development / Developmental Design
In social systems and particularly ones operating with great complexity, models of creating programs, policies and products that simply release into the world like a table are becoming anachronistic. Tables work for simple tasks and sometimes complicated ones, but not complex ones (at least, consistently). It is in those areas that we need to consider the tree as a more appropriate model. However, in human systems these “trees” are designed — we create the social world, the policies, the programs and the products, thus design thinking is relevant and appropriate for those seeking to influence our world.
Yet, we need to go even further. Designing tables means creating a product and setting it loose. Designing for trees means constantly adapting and changing along the way. It is what I call developmental design. Tim Brown, the CEO of IDEO and one of the leading proponents of design thinking, has started to consider the role of design and complexity as well. Writing in the current issue of Rotman Magazine, Brown argues that designers should consider adapting their practice towards complexity. He poses six challenges:
- We should give up on the idea of designing objects and think instead about designing behaviours;
- We need to think more about how information flows;
- We must recognize that faster evolution is based on faster iteration;
- We must embrace selective emergence;
- We need to focus on fitness;
- We must accept the fact that design is never done.
Developmental Evaluation
Bringing Design and Evaluation Together
Image (Header) Growth by Rougeux
Image (Tree) Arbre en fleur by zigazou76
Image (Table) Table à ouvrage art nouveau (Musée des Beaux-Arts de Lyon) by dalbera
All used under licence.
Social Media and Health: Leaders(hip) and Followers(hip)
Posted: May 18, 2012 Filed under: behaviour change, eHealth, health promotion, innovation, public health, social media, systems thinking | Tags: complexity, creativity, design thinking, developmental design, developmental evaluation, eHealth, evaluation, Facebook, health, health promotion, healthcare, innovation, medicine, organizational change, organizational design, public health, social media, systems thinking 2 Comments »
Social media is finally catching on with healthcare, public health, and health promotion. With a few recent articles published in the academic literature to rest on, academic health sciences has finally (and I might argue, begrudgingly) conceded that 900+ million users and $100B valuations (Facebook), and thousands of messages exchanged every milisecond (microblogs like Twitter and Sina Wiebo) might have some value for the public beyond entertainment.
If you note how long it took the health sector to start using the telephone as a serious means of engaging their patients or the public, this is lightning-quick adoption. Still, the barriers to adoption are high and the approach to using the technology is scattered. Indeed, just like the start of Internet-delivered telehealth (or cybermedicine (PDF), which has now evolved into eHealth), there is a mad rush to get liked, followed or some other metrics that most health professionals barely understand.
And that is part of the problem.
Meaningful Social Media Metrics
What is a meaningful metric for social media and health? A recently published article in Health Promotion Practice suggested four metrics that are taken from social marketing and applied to social media. These Key Performance Indicators (KPI’s) are:
- Insights (consumer feedback)
- Exposure (media impressions, visits, views, etc..)
- Reach (# people who connect to the social media application)
- Engagement (level of interaction with the content)
These are reasonable, but to to the uninitiated I would suggest a few words of caution and commentary to this list.
Firstly, the insights suggested by Neiger and colleagues “can be derived from practices such as sentiment analysis or data mining that uses algorithms to extract consumer attitudes and other perspectives on a particular topic” (p.162). While not incorrect, this makes the job sound relatively simple and it is not. Qualitative analysis + quantitative metrics such as those derived from data mining are key. Context counts immeasurably in social media use. It’s only in situations where social media is used as a broadcasting tool that gross measures of likes and sentiment analysis work with little qualification.
Even that is problematic. Counts of ‘likes’, ‘visits’, ‘follows’ and such are highly problematic and can be easily gamed. I am ‘followed’ on Twitter by people who have tens of thousands of followers, yet virtually no presence online. Most often they are from marketing fields where the standard practice is to always follow back those who follow you. Do this enough and pretty quickly you, too can have 23,000 followers and follow 20,000 more. This is meaningless from the perspective of developing relationships.
Engagement is the most meaningful of these metrics and the hardest to fully apply. This category gets us to consider the difference between “OMG! AWESOME!” and “That last post made me think of this situation [described here] and I suggest you read [reference] here for more” as comments. Without understanding the context in which these are made within the post, between posts (temporally and sequentially), and in relation to a larger social and informational context, simple text analysis won’t do.
Social Media Evidence: Problems and More Problems
One of the objections to the use of social media by some is that it is not evidence-based. To that extent I would largely agree that this is the case, but then we’ve been jumping out of airplanes with parachutes despite any randomized controlled trial to prove their worth.
Another article in Health Promotion Practice in 2011 highlights potential applications for social media and behaviour change without drawing on specific examples from the literature, but rather on theoretical and rhetorical arguments. An article published in the latest issue of Perspectives on Psychological Science highlights the current state of research on Facebook, which is timely given that its IPO is set for today. That review by Wilson and colleagues illustrates the largely descriptive nature of the field and offers some insight on to the motivation of Facebook users and their online activities, but rather little in what Facebook does to promote active change in individuals and communities when they leave the platform.
The answer to whether social media like platforms such as Facebook ‘work’ as methods of promoting change is simply: we don’t know.
Does social media provide support to people? Yes. Does it inform them? Yes to that too. Does that information produce something other than passive activity on the topic? We don’t know.
In order to answer these questions, health sciences professionals, evaluators, and tech developers need to consider not just followership, but leadership. In this respect, it means creating changes to the way we gather evidence, the tools and methods we use to analyse data, and the organizational structures necessary to support the kind of real-time, rapid cycle evaluation and developmental design work necessary to make programs and evidence relevant to a changing context.
As Facebook launches into its new role as a public company it is almost assured to be introducing new innovations at a rapid pace to ensure that investor expectations (which are enormous) are met. This means that today’s Facebook will not be next month’s. Having funding mechanisms, review and approval mechanisms, a staff trained and oriented to rapid response research, and an overall organizational support system for innovation is the key.
Right now, we are a long way from that. Hospitals are very large, risk averse organizations; public health units are not much different. They both operate in a command-and-control environment suited for complicated, not complex informational and social environments. Social media is largely within the latter.
Systems thinking, design thinking, developmental evaluation, creativity, networks and innovation: these are the keywords for health in the coming years. They are as author Eric Topol calls the dawning of the creative destruction of medicine.
The public is already using social media for health and now the time has come for health (care, promotion and protection) systems to get on board and make the changes necessary to join them.
The Complexity of Planning and Design in Social Innovation
Posted: January 7, 2012 Filed under: complexity, design thinking, evaluation, systems science, systems thinking | Tags: complexity, design, developmental design, developmental evaluation, evaluation, health, human services, social media, systems thinking 2 Comments »
The Architecture of Complex Plans
Planning works well for linear systems, but often runs into difficulty when we encounter complexity. How do we make use of plans without putting too much faith in their anticipated outcome and still design for change and can developmental design and developmental evaluation be a solution?
It’s that time of year when most people are starting to feel the first pushback to their New Year’s Resolutions. That strict budget, the workout plan, the make-time-for-old-friends commitments are most likely encountering their first test. Part of the reasons is that most of us plan for linear activities, yet in reality most of these activities are complex and non-linear.
A couple interesting quotes about planning for complex environments:
No battle plan survives contact with the enemy – Colin Powell
In preparing for battle I have always found that plans are useless, but planning is indispensable – Dwight D. Eisenhower
Combat might be the quintessential complex system and both Gens Powell and Eisenhower knew about how to plan for it and what kind of limits planning had, yet it didn’t dissuade them from planning, acting and reacting. In war, the end result is what matters not whether the plan for battle went as outlined (although the costs and actions taken are not without scrutiny or concern). In human services, there is a disproportionate amount of concern about ‘getting it right’ and holding ourselves to account for how we got to our destination relative what happens at the destination itself.
Planning presents myriad challenges for those dealing with complex environments. Most of us, when we plan, expect things to go according to what we’ve set up. We develop programs to fit with this plan, set up evaluation models to assess the impact of this plan, and envisage entire strategies to support the delivery and full realization of this plan into action. For those working in social innovation, what is often realized falls short of what was outlined, which inevitably causes problems with funders and sponsors who expect a certain outcome.
Part of the problem is the mindset that shapes the planning process in the first place. Planning is designed largely around the cognitive rational approach to decision making (PDF), which is based on reductionist science and philosophy. Like the image above, a plan is often seen as a blueprint for laying out how a program or service is to unfold over time. Such models of outlining a strategy is quite suitable for building a physical structure like an office where everything from the materials to the machines used to put them together can be counted, measured and bound. This is much less relevant for services that involve interactions between autonomous agents who’s actions have influence on the outcome of that service and that result might vary from context to context as a consequence.
For evaluators, this is problematic because it reduces the control (and increases variance and ‘noise’) into models that are designed to reveal specific outcomes using particular tools. For program implementers, it is troublesome because rigid planning can drive actions away from where people are and for them into activities that might not be contextually appropriate due to some change in the system.
For this reason the twin concepts of developmental evaluation and developmental design require some attention. Developmental evaluation is a complexity-oriented approach to feedback generation and strategic learning that is intended for programs where there is a high degree of novelty and innovation. Programs where the evidence is low or non-existent, the context is shifting, and there are numerable strong and diverse influences are those where developmental evaluations are not only appropriate, but perhaps one of the only viable models of data collection and monitoring available.
Developmental design is a concept I’ve been working on as a reference to the need to incorporate ongoing design and re-design into programs even after they have been initially launched. Thus, a program evolves over time drawing in information from feedback gained through processes like evaluation to tweak its components to meet changing circumstances and needs. Rather than have a static program, a developmental design is one that systematically incorporates design thinking into the evolutionary fabric of the activities and decision making involved.
Both developmental design and evaluation work together to provide data required to allow program planners to constantly adapt their offerings to meet changing conditions, thus avoiding the problem of having outcomes becoming decoupled from program activities and working with complexity rather than against it. For example, developmental evaluation can determine what are the key attractors shaping program activities while developmental design can work with those attractors to amplify them or dampen them depending on the level of beneficial coherence they offer a program. In two joined processes we can acknowledge complexity while creating more realistic and responsive plans.
Such approaches to design and evaluation are not without contention to traditional practitioners, leaving questions about the integrity of the finished product (for design) and the robustness of the evaluation methods, but without alternative models that take complexity into account, we are simply left with bad planning instead of making it like Eisenhower wanted it to be: indispensable .
A Call to Evaluation Bloggers: Building A Better KT System
Posted: December 9, 2011 Filed under: evaluation, knowledge translation | Tags: American Evaluation Association, Andrea Yip, blogging, CENSE Research + Design, consulting, data visualization, developmental evaluation, evaluation, knowledge translation, Michael Quinn Patton, social media, tips 1 Comment »
Time To Get Online...
Are you an evaluator and do you blog? If so, the American Evaluation Association wants to hear from you. This CENSEMaking post features an appeal to those who evaluate, blog and want to share their tips and tricks for helping create a better, stronger KT system.
Build a better moustrap and the world will beat a path to your door — Attributed to Ralph Waldo Emerson
Knowledge translation in 2011 is a lot different than it was before we had social media, the Internet and direct-to-consumer publishing tools. We now have the opportunity to communicate directly to an audience and share our insights in ways that go beyond just technical reports and peer-reviewed publications, but closer to sharing our tacit knowledge. Blogs have become a powerful medium for doing this.
I’ve been blogging for a couple of years and quite enjoy it. As an evaluator, designer, researcher and health promoter I find it allows me to take different ideas and explore them in ways that more established media do not. I don’t need to have the idea perfect, or fully formed, or relevant to a narrow audience. I don’t need to worry about what my peers think or my editor, because I serve as the peer review, editor and publisher all at the same time.
I originally started blogging to share ideas with students and colleagues — just small things about the strange blend of topics I engage in that many don’t know about or understand or wanted to know more of. Concepts like complexity, design thinking, developmental evaluation, and health promotion can get kind of fuzzy or opaque for those outside of those various fields.
Blogs enable us to reach directly to an audience and provide a means of adaptive feedback on ideas that are novel. Using the comments, visit statistics, and direct messages sent to me from readers, I can gain some sense of what ideas are being taken up with people and which one’s resonate. That enables me to tailor my messages and amplify those parts that are of greater utility to a reader, thus increasing the likelihood that a message will be taken up. For CENSEMaking, the purpose is more self-motivated writing rather than trying to assess the “best” messages for the audience, however I have a series of other blogs that I use for projects as a KT tool. These are, in many cases, secured and by invitation only to the project team and stakeholders, but still look and feel like any normal blog.
WordPress (this site) and Posterous are my favorite blogging platforms.
As a KT tool, blogs are becoming more widely used. Sites like Research Blogging are large aggregations of blogs on research topics. Others, like this one, are designed for certain audiences and topics — even KT itself, like the KTExchange from the Research Into Action Action initiative at the University of Texas and MobilizeThis! from the Research Impact Knowledge Mobilization group at York University.
The American Evaluation Association has an interesting blog initiative led by AEA’s Executive Director Susan Kistler called AEA365, which is a tip-a-day blog for evaluators looking to learn more about who and what is happening in their field. A couple of years ago I contributed a post on using information technology and evaluation and was delighted at the response it received. So it reaches people. It’s for this reason that AEA is calling out to evaluation bloggers to contribute to the AEA365 blog with recommendations and examples for how blogging can be used for communications and KT. AEA365 aims to create small-bite pockets of information that are easily digestible by its audience.
If you are interested in contributing, the template for the blog is below, with my upcoming contribution to the AEA365 blog posted below that.
By embracing social media and the power to share ideas directly (and done so responsibly), we have a chance to come closer to realizing the KT dream of putting more effective, useful knowledge into the hands of those that can use it faster and engage those who are most interested and able to use that information more efficiently and humanely.
Interested in submitting a post to the AEA365 blog? Contact the AEA365 curators at aea365@eval.org.
Template for aea365 Blogger Posts (see below for an example)
[Introduce yourself by name, where you work, and the name of your blog]
Rad Resource – [your blog name here]: [describe your blog, explain its focus including the extent to which it is related to evaluation, and tell about how often new content is posted]
Hot Tips – favorite posts: [identify 3-5 posts that you believe highlighting your blogging, giving a direct link and a bit of detail for each (see example)]
- [post 1]
- [post 2]
- Etc.
Lessons Learned – why I blog: [explain why you blog – what you find useful about it and the purpose for your blog and blogging. In particular, are you trying to inform stakeholders or clients? Get new clients? Provide a public service? Help students?]
Lessons Learned: [share at least one thing you have learned about blogging since you started]
Remember – stay under 450 words total please!
–
My potential contribution (with a title I just made up): Cameron Norman on Making Sense of Complexity, Design, Systems and Evaluation: CENSEMaking
Rad Resource – [CENSEMaking]: CENSEMaking is a play on the name of my research and design studio consultancy and on the concept of sensemaking, something evaluators help with all the time. CENSEMaking focuses on the interplay of systems and design thinking, health promotion and evaluation and weaves together ideas I find in current social issues, reflections on my practice as well as the evidence used to inform it. I aspire to post on CENSEMaking 2-3 times per week, although because it is done in a short-essay format, find the time can be a challenge.
Hot Tips – favorite posts:
- What is Developmental Evaluation? This post came from a meeting of working group with Michael Quinn Patton and was fun to write because the original exercise that led to the content (described in the post) was so fun to do. It also provided an answer to a question I get asked all the time.
- Visualizing Evaluation and Feedback. I believe that the better we can visualize complexity the more feedback we provide, the greater the opportunities we have for engaging others, and more evaluations will be utilized. This post was designed to provoke thinking about visualization and illustrate how its been creatively used to present complex data in interesting and accessible ways. My colleague and CENSE partner Andrea Yip has tried to do this with a visually oriented blog on health promoting design, which provides some other creative examples of ways to make ideas more appealing and data feel simpler.
- Developmental Design and Human Services. Creating this post has sparked an entire line of inquiry for me on bridging DE and design that has since become a major focus for my work. This post became the first step in a larger journey.
Lessons Learned – why I blog: CENSEMaking originally served as an informal means of sharing my practice reflections with students and colleagues, but has since grown to serve as a tool for knowledge translation to a broader professional and lay audience. I aim to bridge the sometimes foggy world that things like evaluation inhabit – particularly developmental evaluation – and the lived world of people whom evaluation serves.
Lessons Learned: Blogging is a fun way to explore your own thinking about evaluation and make friends along the way. I never expected to meet so many interesting people because they reached out after reading a blog post of mine or made a link to something I wrote. This has also led me to learn about so many other great bloggers, too. Give a little, get a lot in return and don’t try and make it perfect. Make it fun and authentic and that will do.
___
** Photo by digitalrob70 used under Creative Commons License from Flickr
What is Developmental Evaluation?
Posted: November 19, 2011 Filed under: complexity, evaluation, innovation, systems thinking | Tags: complexity, developmental evaluation, emergence, Michael Quinn Patton, program evaluation, road trip, strategic learning, systems thinking 8 Comments »
Developmental evaluation (DE) is a problematic concept because it deals with a complex set of conditions and potential outcomes that differ from and challenge the orthodoxy in much of mainstream research and evaluation and makes it difficult to communicate. At a recent gathering of DE practitioners in Toronto, we were charged with coming up with an elevator pitch to describe DE to someone who wasn’t familiar with it; this is what I came up with.
Developmental evaluation is an approach to understanding the activities of a program operating in dynamic, novel environments with complex interactions. It focuses on innovation and strategic learning rather than standard outcomes and is as much a way of thinking about programs-in-context and the feedback they produce. The concept is an extension of Michael Quinn Patton’s original concept of Utilization Focused Evaluation with concepts gleaned from complexity science to account for the dynamism and novelty. While Utilization Focused Evaluation has a series of steps to follow (PDF), Developmental Evaluation is less prescriptive, which is both its strength and its challenge for describing it to people (things I’ve discussed in earlier posts).
So with that in mind, our group was charged with coming up with a way to explain DE to someone who is not familiar with it using anything we’d like — song, poetry, dance, slides, stories and beyond. While my colleague Dan chose to lead us all in song, I opted to go with a simple analogy by comparing DE to a hybrid of Trip Advisor and the classic Road Trip (due to lack of good vocalizing skills).
Trip Advisor has emerged as one of the most popular tools for travellers seeking advice on everything from hotel rooms to airlines to resorts and all the destinations along the way. Trip Advisor is averaging more than 13 million unique visitors per month and, unlike its competitors, focuses on user-generated content to support its service. Thus, your fellow travellers are the source of the recommendations not some professional travel agent or journalist. At its heart are stories of varies tones, detail and quality. People upload various accounts of their stay, chronicling even the most minute detail through photos, links to their blogs, video, and narrative. If you want to get the inside details on what a hotel is really like, check Trip Advisor and you’ll likely find it.
However, like any self-organizing set of ideas, the quality of the content will vary along with the level of reportage and the conclusions will be different depending on the context and experience of the person doing the reporting. For example, if you are a North American who is used to having even the most basic hotel chain offer a room with full-service linens, a bathroom, closet, desk and separate shower, you’ll have a hard time adjusting to something like EasyHotel in Europe.
The Road Trip part (capitalization intended here to denote something different than a regular trip by road), denotes the experience that comes from a journey with a desired destination, but not a pre-determined route and only a generalized timeline. A Road Trip is something that is more than just traveling from Point A to Point B, which is usually accomplished by taking the shortest route, the fastest route or a combination of the two; rather it is a journey. Movies like National Lampoon’s Vacation (and, European Vacation), Thelma and Louise, Harold & Kumar Go to White Castle, and (surprise!) Road Trip all capture this spirit to some effect. I suppose one might even find a more grim example of a Road Trip in the Lord of the Rings Trilogy or The Road.
Road Trips have a long history and are not just a North American phenomenon as this article from the Indian Newspaper, The Hindu reports in some detail:
“Road trips are fun when they are not planned point-to-point. As long as you have accommodation booked, that is enough. Its better not to have agendas; get as spontaneous and adventurous as you can. My friends and I went on a road trip to Goa last year. It was loads of fun as it was the first time we took off on our own without parents. To me, it was more than just a trip with friends. It showed that I could take care of myself and that I was now a grown-up, free to do what I wanted,” says Siddharth, who is doing his engineering.
Chance favours the prepared mind
Road Trip Advisor for Developmental Evaluation
“Road-tripping is a great way to bond with the people you are travelling with and I would strongly recommend it to people. It not only makes you appreciate yourself as an individual but is an amazing experience as you get to meet new people, know different cultures and sample different cuisines. I can never forget biking on sleet, riding though torrential rains, gobbling hot rotis at dhabas, the beautiful snow-capped mountains and guy talk with friends on the trip,” says Dheeraj, who recently went to Ladakh.
all is not hunky dory during these trips. You have to be way about accidents and mishaps. And, realise that freedom comes with responsibility. Says Arjun: “I had borrowed my friend’s bike for the trip, and though it looked good, it gave problems on the foothills of Kodaikanal and we couldn’t do the climb. Being a weekend, there were no mechanics. It helps to know your machine. A passion for road-tripping is not enough. You need to be equipped to take care of yourself also.”
Here, the story parallel is about being prepared. Know evaluation methods, know how to build and sustain relationships and to deal with conflict. A high tolerance for ambiguity and the flexibility to adapt is also important. Knowing a little about systems thinking and complexity doesn’t hurt either. Developmental evaluation is not healthy for those who need a high degree of predictability, are not flexible in their approach, and adhere to rigid timelines. Complex systems collapse under rigid boundary conditions and do evaluators working with such restrictions in developmental contexts.
So why do people do it? “Well, my memories of my favourite road trip were an injured leg, chocolates, beautiful photographs and a great sense of fulfilment,” recalls Arjun.
It is youngsters like these who have transformed road-tripping from just a hobby to an art.
After all, friendship and travel is a potent combination that you can’t say no to.
