Category: knowledge translation

complexityemergencejournalismknowledge translationsocial media

Shaking the System of Knowledge Translation and Journalism

Media covering the media talking about the media #riptide #media #harvard #journalism

Leveraging systems change comes when you are willing to examine the system itself, not just the component parts. News media is struggling to remain financially viable in a time when readership / viewership is high and revenues low by considering ways to adapt to an online world and the way it thinks about the problem will go a long way to whether it can solve it.  

Last night The Joan Shorenstein Center  at Harvard University hosted an event launching the public face of an initiative called Riptide, which sought to create an oral history of journalism as it transmophizes from independent media like paper, television and radio into what I would say is transmedia and social mediaThe Riptide Project has already been criticized for its lack of diversity of its subject matter to the point of being called “The History of Internet News, as Told by Rich, White Men” , although for its many faults it does bring together individuals who have shaped the landscape of the English-language news. That story is still worth listening to and learning from.

The event was organized around a panel featuring AOL Chairman Tim Armstrong, Caroline Little – head of the Newspaper Association of America, and New York Times publisher Arthur Sulzberger Jr. The one hour event featured some wide-ranging discussion on how mainstream media has responded to digital challenges and is seeking to promote quality journalism amidst all these threats (A summary of key points are summarized in a Storify  (click link)).  Among the points that stood out was one NOT discussed and that was around the news systems themselves. While AOL, local newspapers and international publishers like the NY Times were exploring different media vehicles for news — such as AOL’s Huffington Post and recently scaled back local news network Patch — the way journalism was to be done was basically the same, except for journalists this means more work.

There was much handwringing over the threats to the system of journalism and publishing without seeing it as a system that itself requires adaptation at a fundamental level.

Seeing the system

While the event was focused on news and journalism, it could have easily been a parallel lecture in the world of health and scientific publishing and knowledge translation or knowledge mobilization. The leaders were speaking about how they were adding video, using social media and pointed to the well-known (and critiqued) ‘Snowfall‘ journalistic endeavour tried at the New York Times as an example of doing things differently. Snowfall is a multi-media story that brings video, text, and audio together under a NY Times digital umbrella and was intended to show how old and new media could work together. Yet, there are many critics who point out that the apparent success of this new multi-media, long form journalism was really just window dressing and that the numbers — 3 million visits — actually obscured a harder truth that indicated that very few of those readers went through it all. Most skimmed. Few got the whole story

The parallels with academic publishing are startling. For all the talk of high-impact scientific publications, the truth is that getting an article included in a top-flight academic journal is — if it is very well received — is likely to garner less than a few dozen citations. Yet the amount of energy and resources that go into these publications is enormous.

Academic journals are seeking to respond to this challenge by using open-access and web-based publishing, but the same fundamental challenge exists: adapting to new media while keeping the old. The publishing model is not developing, it is adapting to threats and not necessarily in a way that is resilient.

A developmental challenge

Developmental evaluation and design is about transforming the system as you move it along. It means being willing to examine or re-examine commonly held assumptions and working with changing conditions as they change, not just upon past reflection as we saw last night. It also means considering what developing a program is all about, not just improving it. Slide number 17 of the presentation below illustrates how this might look in practice. Developmental evaluation is not about program improvements, it is about developing them further to adapt and respond to changing conditions. The resulting program response might be something that is more effective at achieving goals, but that is not the primary focus.

For journalism the risk is that they will add all these additional layers to their product without questioning the assumptions behind what it means to do good journalism. Are journalist going to be videographers, photographers and web coders as well? The point was raised that the Huffington Post has a climate where journalists sit next to engineers. While creative and useful for looking at innovation, it doesn’t help if journalists, editors and publishers are still also doing all of what they used to do and now need to add on additional activities. At some point it all suffers. Yet, the panelists also argued that strong brands like the NY Times will do well when quality markers fail in the sea of low-brow content. How can this be if the resources to do good reporting aren’t there? You can’t act like a budget outfit, but claim to be bespoke.

Academics and scientists are in the same situation. They are being pushed to deliver high quality science and teaching in an age of diminishing resources, with few good metrics to assess outcome,  TED-worthy presentations, Tweet, blog and get into the community to speak to end-users. It is a lot and might even be possible if the system changed to support it. Instead, fewer resources are given, less support for excellence provided and the expectations rise.

Without quality knowledge translation — whether it be great science journalists or outstanding health scientist or clinical communicators — our entire system will collapse. There is too much information to sift through, it is too complex of a system to operate in, and there are far too many actors to navigate it well. Journalists and their institutions can provide common touch points for many across the system and the woes, challenges and systems issues they face are ones we face in health sciences. Learning from what they did and didn’t do in the realm of communication is worthy as is watching where they go as we seek to question if other areas of health communication need to follow.

Audience seeking direction on the future of #journalism by hearing from leaders of the past #riptide

design thinkingknowledge translationpublic healthsystems thinking

Design (re)Thinking Health Systems

3140253711_c2229c7f98

How might we design health systems to promote health and wellbeing and not just treat illness and disease and manage infirmary and chronic conditions? What if health systems were about health?

If we were to apply design thinking to health systems, what might be do?

In a previous post, I suggested that knowledge translation is too important to be trusted solely to health professionals, partly because they  have largely failed to take up the charge. Taking a step back — a systems thinking perspective — one realizes that to design better knowledge translation, we need to design better health systems.

Julio Frenk, Dean of the School of Public Health at Harvard, believes this too. In a 2010 paper published in PLOS Medicine, Frenk comments on the state of health systems and examines how we might re-think them in light of global health challenges.

Health systems are the main instrumentality to close the knowledge–action gap. To realize this potential, it will be necessary to mobilize the power of evidence to promote change. Yet all too often reform efforts are not evaluated adequately. Each innovation in health systems constitutes a learning opportunity.

Frenk’s article is an invitation to engage in systems and design thinking about health. Both approaches invite pause to consider what the problem is in the first place. For design thinkers, problem scoping is the first step.

For systems thinkers this is akin to setting the boundaries around the problem.

Once we set the boundaries and find the appropriate problem, we then frame it appropriately for design. Problem definition is something often over-looked or under appreciated, but is the core of effective problem solving and design.

If I had an hour to solve a problem I’d spend 55 minutes thinking about the problem and 5 minutes thinking about solutions – Albert Einstein

Health systems are typically defined in light of professional services and policies aimed at making the sick well. They are essentially illness and disease (sick care) systems.  This conceptualization, still dominant in the professional and policy discourse in many Western countries, places medicine at the centre of health services with the allied disciplines working alongside, but rarely ventures its gaze beyond the institutions of care or the conditions such institutions are designed to treat.

Frenk, writing in PLOS Medicine, suggests its time to expand our view of what makes a health system if we are to truly promote and sustain global health and see three key points as provoking such re-thinking:

First, health has been increasingly recognized as a key element of sustainable economic development [1], global security, effective governance, and human rights promotion [2]. Second, due to the growing perceived importance of health, unprecedented—albeit still insufficient—sums of funds are flowing into this sector [3]. Third, there is a burst of new initiatives coming forth to strengthen national health systems as the core of the global health system and a fundamental strategy to achieve the health-related Millennium Development Goals.

In order to realize the opportunities offered by the conjunction of these unique circumstances, it is essential to have a clear conception of national health systems that may guide further progress in global health.

Frenk offers some suggestions:

Part of the problem with the health systems debate is that too often it has adopted a reductionist perspective that ignores important aspects. Developing a more comprehensive view requires that we expand our thinking in four main directions.

First, we should think of the health system not only in terms of its component elements (like human resources, financing, hospitals, clinics, technologies, etc.) but most importantly in terms of their interrelations. Second, we should include not only the institutional or supply side of the health system, but also the population. In a dynamic view, the population is not an external beneficiary of the system; it is an essential part of it.

It’s important to note the mention of the role of the population and its dynamical impact on the system. As populations change dramatically in their composition and form of residency within countries, including a greater movement to urbanization, so too will the myriad factors that influence health systems. The people are the system and thus it will change as populations change. While Frenk lists this as one point of many, it is a radical departure for reductionists or those who see health systems as being about care, not people.

A third expansion of our understanding of systems refers to their goals. Typically, we have limited the discussion to the goal of improving health. This is, indeed, the defining goal of a health system. However, we must look not only at the level of health, but also at its distribution, which gives equity a central place in assessing a health system. In addition, we must also include other goals that are intrinsically valued beyond the improvement of health. One of those goals is to enhance the responsiveness of the health system to the legitimate expectations of the population for care that respects the dignity of persons and promotes their satisfaction. The other goal is fair financing, so that the burden of supporting the system is distributed in an equitable manner and families are protected from the financial consequences of disease.

Frenk’s third challenge is to affirm the very point of health systems at all.

While not explicitly speaking of systems thinking or design thinking, there is much that both fields have in common with Frenk’s argument. Design thinkers might ask: What have we hired our health system to do?

Frenk argues that our health systems must go well beyond just making gains in measured health outcomes towards dignity, respect and social justice.

Finally, we should expand our view with respect to the functions that a health system must perform. Most global initiatives have been concerned mainly with one of those functions, namely, the direct provision of services, whether they are medical or public health services. This is, of course, an essential function, but for it to happen at all, health systems must perform other enabling functions, such as stewardship, financing, and resource generation, including what is probably the most complex of all challenges, the health workforce.

Frenk did not identify specific solutions, but did pose some key questions for health systems design.

If we were to take this challenge up as designers and systems thinkers, what might we do? Here are some suggestions for inquiry:

  • Consider new definitions of health like the one posed in the British Medical Journal that emphasizes looking at the social and environmental influences on health beyond just the absence of physical symptoms. Further inclusion of a psychology of human flourishing might add to this definition.
  • Map out a new system visually with people at the centre, not professionals or institutions. What does that look like? Tools like a Gigamap might provide the kind of multi-media, multi-sensory visual way to conceive of the interrelationships that make up health system. System dynamic models can help this out as well.
  • Engage people across this system to validate this map and co-create possible future models that could serve to shape discussion at multiple levels and  mobilize civil society to support healthy environments.
  • Create small scale, safe-fail / fail-forward, prototypes of small-scale innovations that can be tested, developmentally designed, and rapidly re-developed as needed to start shifting the system as a whole.

Designing health requires designing health systems. Applying new thinking and envisioning a system that is dynamic, comprised of people and just institutions is a start.

Photo: Bartolomeo Eustachi: Peripheral Nervous System, c. 1722 shared by brain_blogger used under Creative Commons Licence

education & learninginnovationjournalismknowledge translation

Is Knowledge Translation In Health Too Important to Leave to Health Professionals?

Storytelling

Knowledge translation — and its affiliated terms knowledge exchange, knowledge integration and knowledge mobilization — was coined to describe a process of taking what is known into what is done in health across the spectrum of science, practice, policy and  the public’s health. As health issues become more complex due to the intertwining of demographics, technology, science, and cultural transformations the need to better understand evidence and its impact on health has never been higher. Questions remain: has demand met supply? How are the health professions dealing with this equation?

Translating knowledge

The Canadian Institutes of Health Research (CIHR), one of the earliest champions of the concept of knowledge translation in research, define it as:

a dynamic and iterative process that includes synthesisdisseminationexchange and ethically-sound application of knowledge to improve the health of Canadians (sic), provide more effective health services and products and strengthen the health care system.

These ideas are expanded below:

Synthesis – Synthesis, in this context, means the contextualization and integration of research findings of individual research studies within the larger body of knowledge on the topic. A synthesis must be reproducible and transparent in its methods, using quantitative and/or qualitative methods. It could take the form of a systematic review, follow the methods developed by the Cochrane Collaboration, result from a consensus conference or expert panel or synthesize qualitative or quantitative results. Realist syntheses, narrative syntheses, meta-analyses, meta-syntheses and practice guidelines are all forms of synthesis. Resources related to synthesis are available.

Dissemination – Dissemination involves identifying the appropriate audience and tailoring the message and medium to the audience. Dissemination activities can include such things as summaries for / briefings to stakeholders, educational sessions with patients, practitioners and/or policy makers, engaging knowledge users in developing and executing dissemination/implementation plan, tools creation, and media engagement.

Exchange – The exchange of knowledge refers to the interaction between the knowledge user and the researcher, resulting in mutual learning. According to the Canadian Health Services Research Foundation (CHSRF), the definition of knowledge exchange is “collaborative problem-solving between researchers and decision makers that happens through linkage and exchange. Effective knowledge exchange involves interaction between knowledge users and researchers and results in mutual learning through the process of planning, producing, disseminating, and applying existing or new research in decision-making.”

Ethically-sound application of knowledge – Ethically-sound KT activities for improved health are those that are consistent with ethical principles and norms, social values, as well as legal and other regulatory frameworks – while keeping in mind that principles, values and laws can compete among and between each other at any given point in time. The term application is used to refer to the iterative process by which knowledge is put into practice.

In short, knowledge translation is about taking what we learn and know from evidence, sharing that knowledge with others and assisting them to make useful health choices in practice and policy through KT.

This often involves communicating across contexts, disciplines, and roles between and from scientists, clinicians, policy makers and to the public alike. In a health environment that is increasingly becoming complex, the ability to communicate across boundaries is no longer an advantage, it’s an essential skill. While we may not always have the right language, we can translate meaning through stories.

But if stories are to be effective they need to be valued.

The value of storytelling

I’ve seen health professionals — scientists and clinicians — roll their eyes when you mention storytelling in a work context. It is as if the only legitimate role for stories is to communicate with children (which University of Alberta researchers are exploring as a tool for sharing health knowledge with parents). Yet, it is through stories that most people share what they know in every other context; why would it be different in health?

Perhaps it is the connotation that stories are ‘made up’ like children’s bedtime tales, but one need only look to journalism to find that we’ve been making ‘stories’ a central part of our life every day. We listen to drive-time radio for stories about the traffic conditions, we watch, download and listen to news stories filed by professional journalists and citizen bloggers alike on mainstream media, Twitter, YouTube, Facebook along with myriad sources across the web. Last week we were glued to various sources to learn storiessome of them false — and create stories about the events of the Boston Marathon bombings.

Stories are what conveys multiple information threads and puts it in a coherent context.

Stories are coherence engines.

Valuing knowledge translation

If knowledge translation is important then it should be reflected in research priorities and evidence for its impact on the system across different disciplines. Dr Shannon Scott and her U of A team recently conducted a systematic review of knowledge translation strategies in the allied health professions and found that the field was full of low quality studies that made it impossible to make firm statements on which methods were best among them . That team has recently proposed a systematic review looking at how the arts and visual methods can further contribute to KT in practice, although it likely the same issue with methodological quality might come into play here, too.

What she and her team are doing is looking at the process of sharing stories and, from a research perspective, sharing stories appears to not have been worth investing in scientifically. At least, not enough to generate a lot of studies and good evidence.

One could argue that knowledge translation is still new and that it takes time to generate such evidence. That is partly true, but it is also an easy prop for those who want to avoid the messiness that comes with communication (and its problematic research context), learning from others, and creating more equitable information spaces, which is what knowledge translation ultimately does. Knowledge translation has also been in use for almost 20 years so in that time — even with the most dismal assessment of the length of time it takes to put knowledge into practice — we should be seeing some decent research published.

KT is fundamentally about sharing. Journalists’ are rewarded for sharing — the more they share and the more people who they share with (as measured by readers, listeners, viewers etc..) the more successful they are in their work. Teachers are rewarded for sharing because that means that they are teaching people. Librarians are rewarded for sharing because that means people are checking out books and using the resources in their library.

We don’t apply the same standard to academic research, even though we have some crude metrics to measure reach and impact,  and there is roughly no metric for the degree to which clinicians share among themselves. Maybe this needs to change.

I have scientific colleagues who are fierce in the face of their most strident academic critics and have delivered keynotes to auditoriums filled with researchers that are nearly paralyzed in the face of speaking to the public. This is not fear of public speaking, its fear of speaking to the public.

Should they be? I don’t think speaking to the public should be expected to be enjoyable for everyone, but neither are doing statistical calculations, completing ethics applications, or presenting posters at conferences, but we still expect scientists to do that. We still expect nurses, doctors, psychologists, medical technicians and social workers to traverse complex social problems to talk to their patients in an open and honest way.

Why is it when scientists are speaking to policy makers, clinicians to scientists, policy makers to the public, or any professional to another from another discipline, speciality or division we decide its not critical for them to make the effort?

Why don’t we do the research to support it? 

Why is it OK not to do KT because its uncomfortable, awkward, difficult or confusing?

Declining interest, rising demand

It is perhaps for reasons like this that knowledge translation is so poorly understood and taken up as a focus for research. Looking at Google NGram data (which tracks mention of specific topics in books and publications) we see a steady rise in citations until about 2003 followed by a levelling off. Keep in mind that the leveling begins before social media became known. In the years after Twitter, Facebook and YouTube — arguably the most powerful communications media we have for doing knowledge translation widely (but perhaps not deeply) — there is roughly no sharp increase.

Below are the citations for the terms knowledge translation, knowledge exchange, and knowledge integration  from 1996 (when the Web first started gaining wide use beyond academia and the military) and 2008, the latest year for which there is available data. Note that the numbers reflect general mentions as a percentage of overall terms, so they are relative, not absolute values.

Figure 1: Google NGram Data for KT, KE & KI: 1996-2008

Knowledge Translation, Exchange & Integration NGram

Is there so much other stuff to talk about in 2013 that the relative importance of knowledge translation is diminished?

A look at Google Trend data using the same terms finds that not only are these concepts not growing, their mention is actually shrinking.

Looking at the three terms we see that all three concepts have declined over time. During these years — 2004-2013 — we saw not only the birth of social media, but the rise of Internet-enabled handheld devices to allow knowledge to be shared anywhere there is a data signal. We now have apps and nearly all of the Internets resources in our pockets and we are seeing a decline in the use of these terms.

Figure 2: Google Trend Data for KT, KE & KI: 1996-2013

Knowledge Term Trends

Where to?

So to review: We have a body of evidence in KT that is problematic and incomplete at the same time we have a decrease in use of the terms, while at the very same time we have a sharp rise in available tools and technologies to share information quickly and a continued, steady demand for more information to make decisions for health providers, patients, policy makers and insurers.

Yes, the data presented here are not perfect. But does it not make sense that there should at least be some trend upward if knowledge translation is valued? Should we not see some shift to more research, better research evidence, and greater interest given the tools and scope of communications we have through social media?

This begs the question: is knowledge translation in health too important to leave to health professionals? 

In future posts this question will be looked at in greater depth. Stay tuned.

* Blog has been updated since original post

knowledge translationsocial media

Twitter shows how the news is made, and it’s not pretty — but it’s better that we see it

With the tragic events surrounding the Boston Marathon bombings today, the strength and weaknesses of Twitter and the new media for journalism gets brought out for everyone to see. The news is changing and the importance of traditional journalism and citizen witness reporting all comes together. Much to consider as we reflect on the ways of the world and try to make it a better place while others seek otherwise.

knowledge translationsocial media

Social Media For Researchers

Social Media For Researchers

I recently sat down and chatted with Armine Yalnizyan, a journalist and board member of the Canadian Institutes for Health Research (CIHR) Institute of Public and Population Health (IPPH) to chat about social media for the IPPH about how social tools can assist researchers to do their work, share their learnings, and improve knowledge translation to the community .

Armine kindly referred me to a “rock star social media communicator” but I think we all can play some pretty interesting metaphorical music in our use of social media to assist us with engaging the public. Here is the link to that webinar conversation for those of you interested in understanding more about what social media is and how it works to support the goals of health research more broadly.

knowledge translationpublic healthscience & technologysocial media

The Fourth Estate of Health and Medicine

Who Will Hold Evidence To Account?

Journalists occupy an important, yet often unacknowledged, role in the health system by providing a dispassionate account of the system’s strengths, weaknesses, and opportunities to the public. It is through journalists that much of the research we scientists and practitioners produce gets communicated to the audiences likely to use them. This fourth estate is also a  place where hard questions can be asked and answered, holding governments, business and the health system itself to account because journalists operate apart from this space, unlike scientists and clinicians. We are at risk of losing this and it’s time to consider what that means for our collective health and wellbeing.

Disrupted Media

The news business is going through a massive upheaval, part of a larger overall disruption in media. Many newspapers are reducing the size of their print offerings, publishing less frequently or ceasing operations altogether.

This reduction in the capacity and size of the fourth estate begs two simple questions: Who will hold health scientists, clinicians, pharmaceutical companies, health product manufacturers, and policy makers to account? and who will tell the stories of science, health and medicine in public?

While we have some activist academics doing great work on influencing broader audiences like policy makers, they are exceptions not the norm. Stanton Glantz, a major tobacco control champion from UCSF who taken to blogging as a means of communicating to professionals and the public directly, is one of these such people. But Stan is atypical and holds a tenured position at a major university, something he’s acknowledged protected him when pursuing issues of evidence withholding from the tobacco companies in the 1990’s and beyond. Many faculty (particularly younger ones) are not this secure and even fewer independently funded scientists are. Academia is changing and not in ways that favour security and stability, which has implications for the kind of stories that get told.

Journalists have traditionally relied on protection from their publisher or producer under the name of journalistic freedom (the fourth estate) as a key pillar of their profession. It’s hard to imagine the Watergate scandal coming to light had Bob Woodward, Carl Bernstein and the other reporters working for the Washington Post, Time Magazine and New York Times not had the resources, stability and support provided by their newspapers . But what happens when these resources are no longer available or there are no institutions to support journalists in serving as watchdogs to hold people or institutions to account for what they do and don’t do?

Are ‘Monkeys in Coats’ A Healthy Story?

It’s been suggested that the Internet will take care of this. Citizen journalists, armed with camera-laden handsets connected to social media will fill the news gap. For example, it was citizens, not journalists, who first captured the story of Darwin the monkey, dressed in a shearling coat, walking around an Ikea parking lot in Toronto that went viral on a global scale on December 10, 2012. This is great for those interested in simian fashions and retail adventures, but the reason it was captured was because the story was obvious and in the face (or at the ankles) of those who told it. (For those of you not familiar with Toronto, coat-wearing monkeys are not typically seen at shopping centres or anywhere around town for that matter.)

Health and medicine is not the same as monkeys wearing coats (no matter what kind of joke you want to make). There is nuance, debate and reason that requires sustained attention and focus that someone with an iPhone and Twitter account is less likely to convey. Reasoned arguments for citizen journalism’s potential suggest it can complement the work of traditional journalism, not replace it. Yet, is this belief in one form (citizen journalism) undermining support for the other (traditional journalism) and serving as a fix that ultimately fails? If free-and-easy content is available, how likely are publishers willing to pay for professional work? Particularly if the choice of stories of one group (e.g., monkeys in coats) are more likely to garner the kind of attention that drives advertising than that of another (e.g., health care financing). Only one of these stories will impact our collective health.

Why does this matter? Trained journalists are required to be good communicators to a broad audience, scientists are not. Clinicians are slightly better, but decades of research has shown it is still highly problematic across areas of practice. This will not be solved overnight, if at all. Scientists and clinicians have told me they are already burdened with enough job expectations and adding knowledge translation skills to that list is asking too much.

As I have argued previously, there is a valued place for synthetics in research: those are who are good at taking ideas and weaving them together into an accessible narrative. Journalists are ideally suited to play or support this role. They do the job that many scientists can’t or won’t do and have better to tools, skills and strategies to do it. They write in a style that is suited to broad audiences in a way that suit those audiences’ needs, not what funders, disciplinary traditions, universities, or scientific peers demand (without evidence that those methods of communication are effective). There are reasons why journalists assess the reach of their work in the thousands and social scientists in the dozens (by citations in their field of practice).

Going Deeper to See Clearer

Although we have more information about health available to us than ever before, this may not be healthy for patients. The potential for those uninformed about medical diagnostics, evidence, and the nature of health itself to make poor choices based on incomplete, incorrect or overwhelming information is high. Further, without the kind of dispassionate examination of evidence in a synthetic manner that is tied to the way in which that evidence is expressed in the world through public opinion, policy making and healthcare practices, we lose a major accountability mechanism and means of informing public discourse.

In October I co-delivered a workshop on health evidence for students at the University of Toronto with the 2012 Hancock Lecturer and journalist Julia Belluz. Julia writes the Science-ish blog for Macleans Magazine and is an Associate Editor with the Medical Post. Julia`s lecture was on the role that social media plays in our health system and how its power to leverage the attention of the masses — for good and ill — is shaping the public understanding of health and medicine often in the absence of evidence for effects of conditions, processes, and practice. The lecture is summarized online on Science-ish beginning here.

Reading through the lecture notes one sees a depth of study that would be unlikely to be found anywhere within the formal health system. The reasons are that it blends evidence with commentary, observation with carefully selected sources, and takes a perspective that seeks to inform a wide, not narrow audience in both practical and intellectually stimulating ways. Taken together, this is a collection of activities that are not within the scope of practice for scientists and practitioners. There are reasons why the greatest contributors to public discourse on many scientific issues has come from journalists, not the scientists who generate the research. They tell the story better.

Malcolm Gladwell, Steven Johnson, Mitch Waldrop, Julia Belluz, Andre Picard and others are a big part of the reasons most of the those who vote to support funding of science, who donate to research-related causes, and fight for policies to keep us healthy know of the research that backs those ideas up.

Imperfect as journalism is, it serves the public when done with integrity. It’s worth spending some time considering what can be done to support the fourth estate so it supports us.

Photo credit: DBduo Photography on Flickr used under Creative Commons Licence.

complexitydesign thinkingeducation & learningemergenceevaluation

Evaluating Social Innovation For Social Impact

How do the innovation letters line up?

Earlier this week I has the pleasure of attending talks from Bryan Boyer from the Helsinki Design Lab and learning about the remarkable work they are doing in applying design to government and community life in Finland. While the focus of the audience for the talks was on their application of design thinking, I found myself drawn to the issue of evaluation and the discussion around that when it came up.

One of the points raised was that design teams are often working with constraints that emphasize the designed product, rather than its extended outcome, making evaluation a challenge to adequately resource. Evaluation is not a term that frequents discussion on design, but as the moderator of one talk suggested, maybe it should.

I can’t agree more.

Design and Evaluation: A Natural Partnership

It has puzzled me to no end that we have these emergent fields of practice aimed at social good — social finance and social impact investing, social innovation, social benefit (PDF)– that have little built into their culture to assess what kind of influence they are having beyond the basics. Yet, social innovation is rarely about simple basics, it’s influence is likely far larger, for better or worse.

What is the impact being invested in? What is the new thing being created of value? and what is the benefit and for whom? What else happened because we intervened?

Evaluation is often the last thing to go into a program budget (along with knowledge translation and exchange activities) and the first thing to get cut (along with the aforementioned KTE work) when things go wrong or budgets get tightened. Regrettably, our desire to act supersedes our desire to understand the implication of those actions. It is based on a fundamental idea that we know what we are doing and can predict its outcomes.

Yet, with social innovation, we are often doing things for the first time, or combining known elements into an unknown corpus, or repurposing existing knowledge/skills/tools into new settings and situations. This is the innovation part. Novelty is pervasive and with that comes opportunities for learning as well as the potential for us to good as well as harm.

An Ethical Imperative?

There are reasons beyond product quality and accountability that one should take evaluation and strategic design for social innovation seriously.

Design thinking involves embracing failure (e.g,  fail often to succeed sooner is the mantra espoused by product design firm IDEO) as a means of testing ideas and prototyping possible outcomes to generate an ideal fit. This is ideal for ideas and products that can be isolated from their environment safely to measure the variables associated with outcomes, if considered. This works well with benign issues, but can get more problematic when such interventions are aimed at the social sphere.

Unlike technological failures in the lab, innovations involving people do have costs. Clinical intervention trials go through a series of phases — preclinical through five stages to post-testing — to test their impact, gradually and cautiously scaling up with detailed data collection and analysis accompanying each step and its still not perfect. Medical reporter Julia Belluz and I recently discussed this issue with students at the University of Toronto as part of a workshop on evidence and noted that as complexity increases with the subject matter, the ability to rely on controlled studies decreases.

Complexity is typically the space where much of social innovation inhabits.

As the social realm — our communities, organizations and even global enterprises — is our lab, our interventions impact people ‘out of the gate’ and because this occurs in an inherently a complex environment, I argue that the imperative to evaluate and share what is known about what we produce is critical if we are to innovate safely as well as effectively. Alas, we are far from that in social innovation.

Barriers and Opportunities for Evaluation-powered Social Innovation

There are a series of issues that permeate through the social innovation sector in its current form that require addressing if we are to better understand our impact.

  1. Becoming more than “the ideas people”: I heard this phrased used at Bryan Boyer’s talk hosted by the Social Innovation Generation group at MaRS. The moderator for the talk commented on how she had wished she’d taken more interest in statistics in university because they would have helped in assessing some of the impact fo the work done in social innovation. There is a strong push for ideas in social innovation, but perhaps we should also include those that know how to make sense and evaluate those ideas in our stable of talent and required skillsets for design teams.
  2. Guiding Theories & Methods: Having good ideas is one thing, implementing them is another. But tying them both together is the role of theory and models. Theories are hypotheses about the way things happen based on evidence, experience, and imagination. Strategic designers and social innovators rarely refer to theory in their presentations or work. I have little doubt that there are some theories being used by these designers, but they are implicit, not explicit, thus remaining unevaluable and untestable or challenged by others. Some, like Frances Westley, have made theories guiding her work explicit, but this is a rarity. Social theory, behaviour change models and theories of discovery beyond just use of Rogers’ Diffusion of Innovation theory must be introduced to our work if we are to make better judgements about social innovation programs and assess their impact. Indeed, we need the kind of scholarship that applies theory and builds it as part of the culture of social innovation.
  3. Problem scope and methodological challenges with it. Scoping social innovation is immensely wide and complicated task requiring methods and tools that go beyond simple regression models or observational techniques. Evaluators working social innovation require a high-level understanding of diverse methods and I would argue cannot be comfortable in only one tradition of methods unless they are part of a diverse team of evaluation professionals, something that is costly and resource intensive. Those working in social innovation need to live the very credo of constant innovation in methods, tools and mindsets if they are to be effective at managing the changing conditions in social innovation and strategic design. This is not a field for the methodologically disinterested.
  4. Low attendance to rigor and documentation. When social innovators and strategic designers do assess impact, too often there is a low attention to methodological rigor. Ethnographies are presented with little attention to sampling and selection or data combination, statistics are used sparingly, and connections to theory or historical precedent are absent. Of course, there are exceptions, but this is hardly the rule. Building a culture of innovation within the field relies on the ability to take quality information from one context and apply it to another critically and if that information is absent, incomplete or of poor quality the possibility for effective communication between projects and settings diminishes.
  5. Knowledge translation in social innovation. There are few fora to share what we know in the kind of depth that is necessary to advance deep understanding of social innovation, regularly. There are a lot of one-off events, but few regular conferences or societies where social innovation is discussed and shared systematically. Design conferences tend towards the ‘sage on the stage’ model that favours high profile speakers and agencies, while academic conferences favour research that is less applied or action-oriented. Couple that with the problem of client-consultant work that is common in social innovation areas and we get knowledge that is protected, privileged or often there is little incentive to add a KT component to the budget.
  6. Poor cataloguing of research. To the last point, we have no formalized methods of determining the state-of-the-art in social innovation as research and practice is not catalogued. Groups like the Helsinki Design Lab and Social Innovation Generation with their vigorous attention to dissemination are the exception, not the rule. Complicating matters is the interdisciplinary nature of social innovation. Where does one search for social innovation knowledge? What are the keywords? Innovation is not a good one (too general), yet neither is the more specialized disciplinary terms like economics, psychology, geography, engineering, finance, enterprise, or health. Without a shared nomenclature and networks to develop such a project the knowledge that is made public is often left to the realm of unknown unknowns.

Moving forward, the challenge for social innovation is to find ways to make what it does more accessible to those beyond its current field of practice. Evaluation is one way to do this, but in pursuing such a course, the field needs to create space for evaluation to take place. Interestingly, FSG and the Center for Evaluation Innovation in the U.S. recently delivered a webinar on evaluating social innovation with the principle focus being on developmental evaluation, something I’ve written about at length.

Developmental evaluation is one approach, but as noted in the webinar : an organization needs to be a learning organization for this approach to work.

The question that I am left with is: is social innovation serious about social impact? If it is, how will it know it achieved it without evaluation?

And to echo my previous post: if we believe learning is essential to strategic design we must ask: How serious are we about learning? 

Tough questions, but the answers might illuminate the way forward to understanding social impact in social innovation.

* Photo credit from Deviant Art innovation_by_genlau.jpg used under Creative Commons Licence.