Tag: knowledge translation

evaluationsocial innovation

Benchmarking change

on April 4, 2016

The quest for excellence within social programs relies on knowing what excellence means and how programs compare against others. Benchmarks can enable us to compare one program to another if we have quality comparators and an evaluation culture to generate them – something we currently lack. 

5797496800_7876624709_b

A benchmark is something used by surveyors to provide a means of holding a levelling rod to determine some consistency in elevation measurement of a particular place that could be compared over time. A benchmark represents a fixed point for measurement to allow comparisons over time.

The term benchmark is often used in evaluation as a means of providing comparison between programs or practices, often taking one well-understood and high performing program as the ‘benchmark’ to which others are compared. Benchmarks in evaluation can be the standard to which other measures compare.

In a 2010 article for the World Bank (PDF), evaluators Azevedo, Newman and Pungilupp, articulate the value of benchmarking and provide examples for how it contributes to the understanding of both absolute and relative performance of development programs. Writing about the need for benchmarking, the authors conclude:

In most benchmarking exercises, it is useful to consider not only the nature of the changes in the indicator of interest but also the level. Focusing only on the relative performance in the change can cause the researcher to be overly optimistic. A district, state or country may be advancing comparatively rapidly, but it may have very far to go. Focusing only on the relative performance on the level can cause the researcher to be overly pessimistic, as it may not be sufficiently sensitive to pick up recent changes in efforts to improve.

Compared to what?

One of the challenges with benchmarking exercises is finding a comparator. This is easier for programs operating with relatively simple program systems and structures and less so for more complex ones. For example, in the service sector wait times are a common benchmark. In the province of Ontario in Canada, the government provides regularly updated wait times for Emergency Room visits via a website. In the case of healthcare, benchmarks are used in multiple ways. There is a target that is used as the benchmark, although, depending on the condition, this target might be on a combination of aspiration, evidence, as well as what the health system believes is reasonable, what the public demands (or expects) and what the hospital desires.

Part of the problem with benchmarks set in this manner is that they are easy to manipulate and thus raise the question of whether they are true benchmarks in the first place or just goals.

If I want to set a personal benchmark for good dietary behaviour of eating three meals a day, I might find myself performing exceptionally well as I’ve managed to do this nearly every day within the last three months. If the benchmark is consuming 2790 calories as is recommended for someone of my age, sex, activity levels, fitness goals and such that’s different. Add on that, within that range of calories, the aim is to have about 50% of those come from carbohydrates, 30% from fat and 20% from protein, and we a very different set of issues to consider when contemplating how performance relates to a standard.

One reason we can benchmark diet targets is that the data set we have to set that benchmark is enormous. Tools like MyFitnessPal and others operate to use benchmarks to provide personal data to its users to allow them to do fitness tracking using these exact benchmarks that are gleaned from having 10’s of thousands of users and hundreds of scientific articles and reports on diet and exercise from the past 50 years. From this it’s possible to generate reasonably appropriate recommendations for a specific age group and sex.

These benchmarks are also possible because we have internationally standardized the term calorie. We have further internationally recognized, but slightly less precise, measures for what it means to be a certain age and sex. Activity level gets a little more fuzzy, but we still have benchmarks for it. As the cluster of activities that define fitness and diet goals get clustered together we start to realize that it is a jumble of highly precise and somewhat loosely defined benchmarks.

The bigger challenge comes when we don’t have a scientifically validated standard or even a clear sense of what is being compared and that is what we have with social innovation.

Creating an evaluation culture within social innovation

Social innovation has a variety of definitions, however the common thread of these is that its about a social program aimed at address social problems using ideas, tools, policies and practices that differ from the status quo. Given the complexity of the environments that many social programs are operating, it’s safe to assume that social innovation** is happening all over the world because the contexts are so varied. The irony is that many in this sector are not learning from one another as much as they could, further complicating any initiative to build benchmarks for social programs.

Some groups like the Social Innovation Exchange (SIX) are trying to change that. However, they and others like them, face an uphill battle. Part of the reason is that social innovation has not established a culture of evaluation within it. There remains little in the way of common language, frameworks, or spaces to share and distribute knowledge about programs — both in description and evaluation — in a manner that is transparent and accessible to others.

Competition for funding, the desire to paint programs in a positive light, lack of expertise, not enough resources available for dissemination and translation, absence of a dedicated space for sharing results, and distrust or isolation from academia among certain sectors are some reasons that might contribute to this. For example, the Stanford Social Innovation Review is among the few venues dedicated to scholarship in social innovation aimed at a wide audience. It’s also a venue focused largely on international development and what I might call ‘big’ social innovation: the kind of works that attract large philanthropic resources. There’s lot of other types of social innovation and they don’t all fit into the model that SSIR promotes.

From my experiences, many small organizations or initiatives struggle to fund evaluation efforts sufficiently, let alone the dissemination of the work once it’s finished. Without good quality evaluations and the means to share their results — whether or not they cast a program in positive light or not — it’s difficult to build a culture where the sector can learn from one another. Without a culture of evaluation, we also don’t get the volume of data and access to comparators — appropriate comparators, not just the only things we can find — to develop true, useful benchmarks.

Culture’s feast on strategy

Building on the adage attributed to Peter Drucker that culture eats strategy for breakfast (or lunch) it might be time that we use that feasting to generate some energy for change. If the strategy is to be more evidence based, to learn more about what is happening in the social sector, and to compare across programs to aid that learning there needs to be a culture shift.

This requires some acknowledgement that evaluation, a disciplined means of providing structured feedback and monitoring of programs, is not something adjunct to social innovation, but a key part of it. This is not just in the sense that evaluation provides some of the raw materials (data) to make informed choices that can shape strategy, but that it is as much a part of the raw material for social change as enthusiasm, creativity, focus, and dissatisfaction with the status quo on any particular condition.

We are seeing a culture of shared ownership and collective impact forming, now it’s time to take that further and shape a culture of evaluation that builds on this so we can truly start sharing, building capacity and developing the real benchmarks to show how well social innovation is performing. In doing so, we make social innovation more respectable, more transparent, more comparable and more impactful.

Only by knowing what we are doing and have done can we really sense just how far we can go.

** For this article, I’m using the term social innovation broadly, which might encompass many types of social service programs, government or policy initiatives, and social entrepreneurship ventures that might not always be considered social innovation.

Photo credit: Redwood Benchmark by Hitchster used under Creative Commons License from Flickr.

About the author: Cameron Norman is the Principal of Cense Research + Design and works at assisting organizations and networks in supporting learning and innovation in human services through design, program evaluation, behavioural science and system thinking. He is based in Toronto, Canada.

complexityinnovationsocial innovation

The Ecology of Innovation: Part 2 – Language

on October 2, 2015
Idea Factories or ecologies of innovation?

Idea Factories or ecologies of innovation?

Although Innovation is about producing value through doing something new or different than before, the concept is far from simple when applied in practice by individuals and institutions. This second in a series of articles on innovation ecology looks at the way we speak of innovation and how what we talk about new ideas and discovery shapes what we do about it. 

“Language can be a way of hiding your thoughts and preventing communication” – Abraham Maslow

Innovation is one of the few concepts that offers little benefit contemplated in the abstract. We innovate on specific things with an eye to application, maybe even scaling that idea broadly. Humans innovate because the status quo is no longer satisfying, is unacceptable or has changed so we strive to come up with new ways of doing things, novel processes and tools to make the current situation a preferred one.

Thus, we are designers seeking our client, customer and creation through innovation and we do this through our words and actions — our language. Indeed, if one agrees with Marty Neumeier‘s assertion that design is the discipline of innovation and Greg Van Alystne & Bob Logan’s definition of design as “creation for reproduction” then our language of innovation is critical to ensuring that we design products and services that have the potential to reproduce beyond an idea.

Language matters in innovation.

To illustrate, lets look at how language manifests itself in the communication of ideas using an example from public health. In a paper entitled Knowledge integration: Conceptualizing communications in cancer control systems I co-authored with my colleagues Allan Best and Bob Hiatt, we looked at the way language was used within a deep and broad field like cancer control in shaping communications. This was not merely an academic exercise, but served to illustrate the values, practices and structures that are put in place to support communicating concepts and serves to illustrate how innovations are communicated.

Innovation as product

What we found was that there are three generations of cancer communications defined by their language and the practices and policies that are manifested in or representative of that language. The first generation of terms were traced up to the 1990’s and were characterized by viewing knowledge as a product. Indeed, the term knowledge products can be traced back to this period. Other key characteristics of this period include:

  • The terminology used to describe communications included the terms diffusion, dissemination, knowledge transfer, and knowledge uptake.
  • Focus on the handoff between knowledge ‘producers’ and knowledge (or research) ‘users’. These two groups were distinct and separate from one another
  • The degree of use is a function of effective packaging and presentation presuming the content is of high quality.

The language of this first generation makes the assumption that the ideas are independent of the context in which they are to be used or where they were generated. The communication represented in this generation of models relies on expertise and recognition of this. But what happens when expertise is not recognized? Or where expertise isn’t even possible? This is a situation we are increasingly seeing as we face new, complex challenges that require mass collaboration and innovation, something the Drucker Forum suggests represents the end of expertise.

Innovation as a contextual process

From the early and mid-1990’s through to the present we’ve seen a major shift from viewing knowledge or innovation as a product to that of a dynamic process where expertise resides in multiple places and sources and networks are valued as much as institutions or individuals. Some of the characteristics of this generation are:

  • Knowledge and good ideas come from multiple sources, not just recognized experts or leaders
  • Social relationships media what is generated and how it is communicated (and to whom)
  • Innovation is highly context-dependent
  • The degree of use of ideas or knowledge is a function of having strong, effective relationships and processes.

What happens when the context is changing consistently? What happens when the networks are dynamic and often unknown?

Systems-embedded innovation

What the paper argues is that we are seeing a shift toward more systems-oriented approaches to communication and that is represented in the term knowledge integration. A systems-oriented model views the design of knowledge structures as an integral to the support of effective innovation by embedding the activities of innovation — learning, discovery, and communication — within systems like institutions, networks, cultures and policies. This model also recognizes the following:

  • Both explicit and implicit knowledge is recognized and must be made visible and woven into policy making and practice decisions
  • Relationships are mediated through a cycle of innovation and must be understood as a system
  • The degree of integration of policies, practices and processes within a system is what determines the degree of use of an idea or innovation.

The language of integration suggests there is some systems-level plan to take the diverse aspects within a set of activities and connect, coordinate and, to some degree, manage to ensure that knowledge is effectively used.

Talking innovation

What makes language such a critical key to understanding innovation ecologies is that the way in which we speak about something is an indication of what we believe about something and how we act. As the quote from psychologist Abraham Maslow suggests above, language can also be used to hide things.

One example of this is in the realm of social innovation, where ideas are meant to be generated through social means for social benefit. This process can be organized many different ways, but it is almost never exclusively top-down, expert-driven. Yet, when we look at the language used to discuss social innovation, we see terms like dissemination regularly used. Examples from research, practice and connecting the two to inform policy all illustrate that the language of one generation continues to be used as new ones dawn.  This is to be expected as the changes in language of one generation never fully supplants that of previous generations — at least not initially. Because of that, we need to be careful about what we say and how we say it to ensure that our intentions are reflected in our practice and our language. Without conscious awareness of what we say and what those words mean there is a risk that our quest to create true innovation ecosystems, ones where innovation is truly systems-embedded and knowledge is integrated we unwittingly create expectations and practices rooted in other models.

If we wish to walk the walk of innovation at a systems level, we need to talk the talk.

Tips and Tricks

Organizational mindfulness is a key quality and practice that embeds reflective practice and sensemaking into the organization. By cultivating practices that regularly check-in and examine the language and actions of an organization in reference to its goals, processes and outcomes. A recent article by Vogus and Sutcliffe (2012) (PDF) provides some guidance on how this can be understood.

Develop your sensemaking capacity by introducing space at regular meetings that bring together actors from different areas within an organization or network to introduce ideas, insights and observations and process what these mean with respect to what’s happened, what is happening and where its taking the group.

Some key references include: 

Best, A., Hiatt, R. A., & Norman, C. D. (2008). Knowledge integration: Conceptualizing communications in cancer control systems. Patient Education and Counseling, 71(3), 319–327. http://doi.org/10.1016/j.pec.2008.02.013

Best, A., Terpstra, J. L., Moor, G., Riley, B., Norman, C. D., & Glasgow, R. E. (2009). Building knowledge integration systems for evidence‐informed decisions. Journal of Health Organization and Management, 23(6), 627–641. http://doi.org/10.1108/14777260911001644

Vogus, T. J., & Sutcliffe, K. M. (2012). Organizational Mindfulness and Mindful Organizing: A Reconciliation and Path Forward. Academy of Management Learning & Education, 11(4), 722–735. http://doi.org/10.5465/amle.2011.0002C

Weick, K. E., Sutcliffe, K. M., & Obstfeld, D. (2005). Organizing and the Process of Sensemaking. Organization Science, 16(4), 409–421. http://doi.org/10.1287/orsc.1050.0133

*** If you’re interested in applying these principles to your organization and want assistance in designing a process to support that activity, contact Cense Research + Design.

journalismknowledge translationpublic healthscience & technology

The Power, Peril and Promise of Health Journalism

on February 14, 2015

Online Prescription Concept

The Toronto Star, Canada’s most widely read newspaper known for its investigative reporting gifted anti-vaccination audiences armament by using poor science to point to a spurious connection between an HPV vaccine and illness. The issue points to journalism’s power to shape the discourse of health issues and it points to the power, promise and peril associated with good (and not so good) science reporting. 

With great power comes great responsibility – Uncle Ben, Spiderman

It started with a story

On Thursday February 5th, 2015 the Toronto Star, Canada’s most widely read newspaper that has a reputation for solid investigative journalism, published an story that connected the experience of young girls and negative health effects with the receiving the Gardasil HPV vaccine. The story was immediately and widely criticized by experienced science journalists and health professionals alike, who argued that it was based on terribly flawed science.

The Toronto Star’s reaction was to defend itself, arguing in many different fora that they indeed mentioned that there was little scientific evidence that supported the link between the vaccine and the negative health effects being discussed in the article. The problem is that these links are buried deep in the article and certainly are not its focus: the hypothesized harms are.

Two days later, the Star published a follow-up op-ed letter which was authored by two health professionals and co-signed / supported by dozens of Toronto’s leading physicians condemning the original article. However, by that time the damage is likely to have been done and one more bit contribution to the fictitious ‘evidence’ for vaccine harms had been added to the anti-vaccine movement’s war chest.

Perpetuating harm

This matter of poor reporting is not a trivial issue. The fraudulent science performed by Andrew Wakefield linking autism to vaccines helped spur an evidence-thin anti-vaccination movement. Today, we are seeing the resurgence of diseases once thought to be eliminated in North America (like measles) because so many people are not having their children vaccinated. Jenny McCarthy is among the celebrities who have taken up the cause of anti-vaccination and has written about and spoken at length about what she sees as the connection between autism and vaccines, using her son’s experience as an ‘example’. Oprah Winfrey, perhaps unwittingly, gave McCarthy a platform to speak about her beliefs on her show offering wider possible credibility to something that has been thoroughly discredited in the scientific literature (PDF).

For the Toronto Star, it was bad enough that the story was published — and is now online, likely for all time in various forms thanks to the Web — but what made it worse was that the Star was so vigorous in its defence of it, unwilling or unable to recognize their role in public health. Medical evidence champion, author, physician and columnist Ben Goldacre was among the many who counter-attacked, pointing to what he called The Star’s ‘smear campaign‘ against the story’s critics.

For an interesting discussion of the issue of just how the Star got it wrong, listen to Vox health reporter Julia Belluz, interviewed on the CBC’s radio show The Current. Belluz, a past MIT Knight Journalism Fellow, is one of a dwindling number of journalists who understand the practice of reporting, science, and medicine and wrote a stellar critique of the Toronto Star article, but as importantly makes the case for why there is a need for specialized, trained, supported journalists out there doing this kind of work.

…and health

I’ve argued in the past that journalism is very much a pillar of public health. When it fails, so does public health. Journalism is not and should not be an arm of public health for the very independence that good, professional journalism strives to maintain is a reason it’s often called the fourth estate, keeping governments and other forces in check to ensure they are not abusive. Yet, that distance is also what makes it a part of public health. Public health is better for journalism and journalism certainly can benefit from health stories as they continue to be popular and sought after by readers.

As a group, scientists and many clinicians are not great at communicating what they do, why their research is important to others outside their field, and what the implications of their findings are for the public and science as a whole. Some are, most are not. It’s for this reason that the entire sub-field of health sciences focused on knowledge translation, exchange and mobilization has emerged. Just as we value the ability of a graphic designer to make visuals come alive, so too have we learned to value those with the skills to communicate information well and that is what journalists are trained and paid to do. They are a big part of this process, or at least should be.

Healthy journalism, healthy science, healthy people

Science journalism is too important to be ignored. There is much skepticism of journalists by scientists and clinicians and indeed, as the Toronto Star shows, journalists sometimes get things wrong. But its one thing to get it wrong through errors of judgement or interpretation it’s quite another to get things wrong by design. The Toronto Star has some good health reporters, but they weren’t the ones on this story. Nor did they bring in the health reporters to consult on this or other health professionals prior to publication– at least as far as one can tell.

The importance to the public’s health of good reporting requires that health and science journalists have more than a rudimentary knowledge of the topics they are covering. What’s strange is how we understand this with our sports reporting, weather forecasts and foreign correspondents. You wouldn’t watch someone who has little understanding of a sport covering it in depth, would you? It’s one thing to read scores, it’s another to provide investigative and deep coverage of a game if you don’t know the players, the rules, the criteria for quality and success and so forth.

Why do we do this with health journalism and science?

Yet, journalism is under pressure and no doubt the Toronto Star, for whatever genuine contrition they experience from what happened, have to like that they are being talked about. The reason is that journalism is under threat for market reasons, the Internet and the changing ways we get our news. It is, as Jürgen Krönig wrote way back in 2004, “A crisis of the Fourth Estate”. That crisis is only getting worse.

As anyone interested in public health, we need to take actions to ensure that the fourth estate is protected, supported and not ignored. Our health might just depend on it.

Image: iStockphoto, used under licence.

education & learningknowledge translationpsychologysystems thinking

Bullying, the market for education and the damaged quest for learning

on September 11, 2014
Dark classroom, light minds

Dark classroom, light minds

A recent study found looked into the experience of cyberbullying by university professors at the hands of their students. This disturbing phenomenon points to much larger issues beyond mental health promotion and calls into question many of the assumptions we have about the systems we’ve designed to foster education and what it means to be a learner at university. 

The university is one of our oldest cultural institutions and its instructors are considered to have among societies most respected jobs, even if not always well compensated. In the past, students often approached their professors with a mixed sense of wonder, respect, curiosity and fear and that, in healthy situations, was reciprocated by faculty to create a space where people could explore ideas, learn, and challenge themselves and others to grow. That relationship has started to change as evidenced by the rise of cyberbullying in the classroom.

A recent article in Macleans Magazine looked at the changing state of the post-secondary classroom and the role of cyberbullying. Only this was not about student victims, but students as the perpetrators against their professors. The effects of cyberbullying are crippling and professors are bearing the burden of having hundreds of eyes watching them, writing about them and writing ‘consumer reviews’ about them in anonymous and sometimes unflattering, inflammatory and questionable terms on sites like RateMyProfessor.com .

Researchers at the University of California, Riverside found that as students age the incidence of face-to-face bullying decreases and cyberbullying increases, which might partly explain why we’re seeing this in university settings when face-to-face bullying goes subterranean. Yet, the notion that professors that are getting bullied by their students belies some other issues that require further investigation, namely those related to the nature of education and the role of students-as-consumers.

Consuming knowledge, producing expectations

If you pay for something, should you not expected to get something rather specific for that experience or product? Aside from some rare experiences of profane/profound personal challenge/punishment like Tough Mudder and its peers or dental work, there are few things we willingly pay for that we don’t derive pleasure from or achieve a very specific (anticipated) outcome.

Education is problematic because we might not know what we’ll get from it going in, what kind of experiences or ideas will emerge, and how our relationship to those experiences will change us. That is its great gift.

Many of us have had profound life changes because of something we experienced through our education and writing as one who has completed four different degree programs and a post-doc I can confidently say that I didn’t receive a lot of what I expected in any of those programs and I am a better person for it. Indeed, if I go to a specific learning event (aside from those focused on a specific technique or technology) I am disappointed if I actually come away with exactly what I expected.

That is part of the point. We don’t know what we don’t know.

But when you start viewing education as a thing that resembles any other market-driven product or services, you begin to focus on learning as a consumable good and your students as customers. In following this line of thought, it makes some sense to focus the delivery of this product on the desires of the consumer.

Increasingly, teachers (of various stripes) are being asked to consider a range of student-related variables in their education. Things like learning styles and preferences are now being woven into classroom instruction and students have come to learn to expect and are increasingly demanding to be taught in ways that match their unique learning preferences and styles. While there is reason to imagine that this approach is useful in stimulating engagement of students in the lessons, there is increasing evidence much of it does little to enhance actual learning. Many of the life lessons we’ve gained that shape what we do and who we are were not delivered in the manner of our choosing, conformed with our preferences and were not desired, expected or enjoyed in the moment. We risk confusing enjoyment with learning; they can be aligned but one isn’t necessary for the other to take place.

However, when we are viewing education from a consumer model, the specific outcomes become part of the contract. If I come to get a degree in X because I believe that the job market demands the skills and knowledge that X brings and I am paying tens of thousands of dollars and spending four or more years acquiring X then I feel entitled to expect all the benefits that X brings. Further, I expect that my journey to acquiring X will be enjoyable, because why would I spend more money than I’ve ever seen on something I don’t enjoy.

Particularly when that is money I don’t have.

A debt to pay

In Canada and the United States, student debt rates have dramatically increased. The Canadian Federation of Students note that Canadian’s attending post-secondary education now owe more than $15B to the Canadian federal government (PDF) as part of their student loan program, a number that doesn’t include debt accumulated from borrowing from banks, family, credit cards and other means. In Canada’s largest province, Ontario, the rate of graduate employment has decreased since 2001 and the overall youth unemployment rate continues to be the highest, despite the province having one of the most educated youth population in the country (and arguably, the world). And while Ontario universities continue to promote the fact that education is a better pathway to success, it is a hard pill for many students to swallow when many can’t apply what they trained for and paid for after they graduate.

Satirist John Oliver has an informative, humorous and distressing take on student debt and the state of consumer-oriented education for those who want to learn more.

None of these reasons are excuses for cyberbullying, but it does give a more complicated picture of those that might feel they are entitled to bully others and their reasoning behind it.

What we are seeing is a systems change in the way education is being produced, consumed and experienced. Even the mere fact that we can now reasonably use the language of consumerism to speak to something like education should give us pause and concern. I’ve been involved in post-secondary education for nearly 20 years and there has always been students who simply wanted the ‘piece of paper’ (degree) as a stepping stone to a job and little more than that from their time at school. They were willing to do the work — often the minimum possible — to graduate, but they knew they had to put the effort in to be successful. There was never an expectation that one was entitled to anything from going to school, although that might be changing.

Market identities and education systems

Belgian psychotherapist Paul Verhaeghe has explored the role of identity in market-based economies in his new book What About Me? In the book, Verhaeghe illustrates how we construct our identities as people drawing on the research that reflects (and often contradicts or obscures) the two major perspectives on personality and identity: the person-as-blank-slate and the person as a reflection of the environment. The former perspective assumes we come into the world as we are while the latter assumes the world makes us who we are and both have enormous amount of moral, cultural and evidentiary baggage attached to them.

What Verhaeghe does is point to the ways in which both have elements of truth to them, but that they are mediated by the manner in which we construct the very questions about who we are and what our purpose is. These questions are (for many cultural, historical, economic and political reasons that he elaborates on) frequently market-based. Thus, who we are is defined by what we do, what we own, what we produce, and how we use such things once out into the world and that the value that come with such ways of defining ourselves is considered self-evident. He makes a disturbing and convincing case when one stops to reflect on the way we think about how we think (metacognition + mindfulness) .

When viewed from the perspective of a market, knowledge and its products soon become the goal and not the journey. Indeed, I’ve even written about this in support of an argument for better research-to-action and knowledge translation. Much of the knowledge-to-action discourse is about viewing knowledge as a product even if the more progressive models also view this as part of a process and even more as part of a system. But it is the last part — the system — that we often give the shortest shrift to in our discussions. What Verhaeghe and others are doing is encouraging us to spend more time thinking about this and the potential outcomes that emerge from this line of thinking.

Unless we are willing to talk more about the systems we create to learn, explore and relate we will continue to support Verhaeghe’s thesis and uphold the conditions for the kind of education-as-a-product thinking that I suspect is contributing to students’ changing behaviour with their professors and creating a climate at universities that is toxic instead of inspiring.

Photo credit: Classroom by Esparta Palma used under Creative Commons License via Flickr. Check out Esparta’s remarkable work here.

behaviour changecomplexityemergenceevaluationknowledge translation

Bringing Design into Developmental Evaluation

on January 2, 2014
Designing Evaluation

Designing Evaluation

Developmental evaluation is an approach to understanding and shaping programs in service of those who wish to grow and evolve what is done in congruence with complexity rather than ignoring it. This requires not only feedback (evaluation), but skills in using that feedback to shape the program (design) for without both, we may end up doing neither. 

A program operating in an innovation space, one that requires adaptation, foresight and feedback to make adjustments on-the-fly is one that needs developmental design. Developmental design is part of an innovator’s mindset that combines developmental evaluation with design theory, methods and practice. Indeed, I would argue that exceptional developmental evaluations are by their definition examples of developmental design.

Connecting design with developmental evaluation

The idea of developmental design emerged from work I’ve done exploring developmental evaluation in practice in health and social innovation. For years I led a social innovation research unit at the University of Toronto that integrated developmental evaluation with social innovation for health promotion and constantly wrestled with ways to use evidence to inform action. Traditional evidence models are based on positivist social and basic science that aim to hold constant as many variables as possible while manipulating others to enable researchers or evaluators to make cause-and-effect connections. This is a reasonable model when operating in simple systems with few interacting components. However, health promotion and social systems are rarely simple. Indeed, not only are they not simple, they are most often complex (many interactions happening at multiple levels on different timescales simultaneously). Thus, models of evaluation are required that account for complexity.

Doing so requires attention to larger macro-level patterns of activity with a program to assess system-level changes and focus on small, emergent properties that are generated from contextual interactions. Developmental evaluation was first proposed by Michael Quinn Patton who brought together complexity theory with utilization-focused evaluation (PDF) and helped program planners and operators to develop their programs with complexity in mind and supporting innovation. Developmental evaluation provided a means of linking innovation to process and outcomes in a systematic way without creating rigid, inflexible boundaries that are generally incompatible with complex systems.

Developmental evaluation is challenging enough on its own because it requires appreciation of complexity and a flexibility in understanding evaluation, yet also a strong sense of multiple methods of evaluation to accommodate the diversity of inputs and processes that complex systems introduce. However, a further complication is the need to understand how to take that information and apply it meaningfully to the development of the program. This is where design comes in.

Design for better implementation

Design is a field that emerged from the 18th century when mass production was first made possible and no longer was the creative act confined to making unique objects, rather it was expanded to create mass-market ones. Ideas are among the ideas that were mass-produced as the printing press, telegraph and radio combined with the means of creating and distributing these technologies made intellectual products easier to produce as well. Design is what OCADU’s Greg Van Alsytne and Bob Logan refer to as “creation for reproduction” (PDF).

Developmental design links this intention for creation for reproduction and the design for emergence that Van Alsytne and Logan describe with the foundations of developmental evaluation. It links the feedback mechanisms of evaluation with the solution generation that comes from design together.

The field of implementation science emerged from within the health and medical science community after a realization that simple idea sharing and knowledge generation was insufficient to produce change without understanding how such ideas and knowledge were implemented. It came from an acknowledgement that there was a science (or an art) to implementing programs and that by learning how these programs were run and assessed we could do a better job of translating and mobilizing knowledge more effectively. Design is the membrane of sorts that holds all of it together and guides the use of knowledge into the construction and reconstruction of programs. It is the means of holding evaluation data and shaping the program development and implementation questions at the outset.

Without the understanding of how ideas are manifest into a program we are at risk of creating more knowledge and less wisdom, more data and less impact. Just as we made incorrect assumptions that having knowledge was the same as knowing what to do with it or how to share it (which is why fields like knowledge translation and mobilization were born) so too have we made the assumption that program professionals know how to design their programs developmentally. Creating a program from scratch from a blank slate is one thing, but doing a live transformation and re-development is something else.

Developmental design is akin to building a plane while flying it. There are construction skills that are unique to this situation that are different from, but build on, many conventional theories and methods of program planning and evaluation, but like developmental evaluation, extend beyond them to create a novel approach for a particular class of conditions. In future posts I’ll outline some of the concepts of design that are relevant to this enterprise, but in the meantime encourage you to visit the Censemaking Library section on design thinking for some initial resources.

The question remains whether we are building dry docks for ships at sea or platforms for constructing aerial, flexible craft to navigate the changing headwinds and currents?

 

Image used under license.

complexityemergencejournalismknowledge translationsocial media

Shaking the System of Knowledge Translation and Journalism

on September 10, 2013

Media covering the media talking about the media #riptide #media #harvard #journalism

Leveraging systems change comes when you are willing to examine the system itself, not just the component parts. News media is struggling to remain financially viable in a time when readership / viewership is high and revenues low by considering ways to adapt to an online world and the way it thinks about the problem will go a long way to whether it can solve it.  

Last night The Joan Shorenstein Center  at Harvard University hosted an event launching the public face of an initiative called Riptide, which sought to create an oral history of journalism as it transmophizes from independent media like paper, television and radio into what I would say is transmedia and social media. The Riptide Project has already been criticized for its lack of diversity of its subject matter to the point of being called “The History of Internet News, as Told by Rich, White Men” , although for its many faults it does bring together individuals who have shaped the landscape of the English-language news. That story is still worth listening to and learning from.

The event was organized around a panel featuring AOL Chairman Tim Armstrong, Caroline Little – head of the Newspaper Association of America, and New York Times publisher Arthur Sulzberger Jr. The one hour event featured some wide-ranging discussion on how mainstream media has responded to digital challenges and is seeking to promote quality journalism amidst all these threats (A summary of key points are summarized in a Storify  (click link)).  Among the points that stood out was one NOT discussed and that was around the news systems themselves. While AOL, local newspapers and international publishers like the NY Times were exploring different media vehicles for news — such as AOL’s Huffington Post and recently scaled back local news network Patch — the way journalism was to be done was basically the same, except for journalists this means more work.

There was much handwringing over the threats to the system of journalism and publishing without seeing it as a system that itself requires adaptation at a fundamental level.

Seeing the system

While the event was focused on news and journalism, it could have easily been a parallel lecture in the world of health and scientific publishing and knowledge translation or knowledge mobilization. The leaders were speaking about how they were adding video, using social media and pointed to the well-known (and critiqued) ‘Snowfall‘ journalistic endeavour tried at the New York Times as an example of doing things differently. Snowfall is a multi-media story that brings video, text, and audio together under a NY Times digital umbrella and was intended to show how old and new media could work together. Yet, there are many critics who point out that the apparent success of this new multi-media, long form journalism was really just window dressing and that the numbers — 3 million visits — actually obscured a harder truth that indicated that very few of those readers went through it all. Most skimmed. Few got the whole story

The parallels with academic publishing are startling. For all the talk of high-impact scientific publications, the truth is that getting an article included in a top-flight academic journal is — if it is very well received — is likely to garner less than a few dozen citations. Yet the amount of energy and resources that go into these publications is enormous.

Academic journals are seeking to respond to this challenge by using open-access and web-based publishing, but the same fundamental challenge exists: adapting to new media while keeping the old. The publishing model is not developing, it is adapting to threats and not necessarily in a way that is resilient.

A developmental challenge

Developmental evaluation and design is about transforming the system as you move it along. It means being willing to examine or re-examine commonly held assumptions and working with changing conditions as they change, not just upon past reflection as we saw last night. It also means considering what developing a program is all about, not just improving it. Slide number 17 of the presentation below illustrates how this might look in practice. Developmental evaluation is not about program improvements, it is about developing them further to adapt and respond to changing conditions. The resulting program response might be something that is more effective at achieving goals, but that is not the primary focus.

For journalism the risk is that they will add all these additional layers to their product without questioning the assumptions behind what it means to do good journalism. Are journalist going to be videographers, photographers and web coders as well? The point was raised that the Huffington Post has a climate where journalists sit next to engineers. While creative and useful for looking at innovation, it doesn’t help if journalists, editors and publishers are still also doing all of what they used to do and now need to add on additional activities. At some point it all suffers. Yet, the panelists also argued that strong brands like the NY Times will do well when quality markers fail in the sea of low-brow content. How can this be if the resources to do good reporting aren’t there? You can’t act like a budget outfit, but claim to be bespoke.

Academics and scientists are in the same situation. They are being pushed to deliver high quality science and teaching in an age of diminishing resources, with few good metrics to assess outcome,  TED-worthy presentations, Tweet, blog and get into the community to speak to end-users. It is a lot and might even be possible if the system changed to support it. Instead, fewer resources are given, less support for excellence provided and the expectations rise.

Without quality knowledge translation — whether it be great science journalists or outstanding health scientist or clinical communicators — our entire system will collapse. There is too much information to sift through, it is too complex of a system to operate in, and there are far too many actors to navigate it well. Journalists and their institutions can provide common touch points for many across the system and the woes, challenges and systems issues they face are ones we face in health sciences. Learning from what they did and didn’t do in the realm of communication is worthy as is watching where they go as we seek to question if other areas of health communication need to follow.

Audience seeking direction on the future of #journalism by hearing from leaders of the past #riptide

design thinkingknowledge translationpublic healthsystems thinking

Design (re)Thinking Health Systems

on May 10, 2013

3140253711_c2229c7f98

How might we design health systems to promote health and wellbeing and not just treat illness and disease and manage infirmary and chronic conditions? What if health systems were about health?

If we were to apply design thinking to health systems, what might be do?

In a previous post, I suggested that knowledge translation is too important to be trusted solely to health professionals, partly because they  have largely failed to take up the charge. Taking a step back — a systems thinking perspective — one realizes that to design better knowledge translation, we need to design better health systems.

Julio Frenk, Dean of the School of Public Health at Harvard, believes this too. In a 2010 paper published in PLOS Medicine, Frenk comments on the state of health systems and examines how we might re-think them in light of global health challenges.

Health systems are the main instrumentality to close the knowledge–action gap. To realize this potential, it will be necessary to mobilize the power of evidence to promote change. Yet all too often reform efforts are not evaluated adequately. Each innovation in health systems constitutes a learning opportunity.

Frenk’s article is an invitation to engage in systems and design thinking about health. Both approaches invite pause to consider what the problem is in the first place. For design thinkers, problem scoping is the first step.

For systems thinkers this is akin to setting the boundaries around the problem.

Once we set the boundaries and find the appropriate problem, we then frame it appropriately for design. Problem definition is something often over-looked or under appreciated, but is the core of effective problem solving and design.

If I had an hour to solve a problem I’d spend 55 minutes thinking about the problem and 5 minutes thinking about solutions – Albert Einstein

Health systems are typically defined in light of professional services and policies aimed at making the sick well. They are essentially illness and disease (sick care) systems.  This conceptualization, still dominant in the professional and policy discourse in many Western countries, places medicine at the centre of health services with the allied disciplines working alongside, but rarely ventures its gaze beyond the institutions of care or the conditions such institutions are designed to treat.

Frenk, writing in PLOS Medicine, suggests its time to expand our view of what makes a health system if we are to truly promote and sustain global health and see three key points as provoking such re-thinking:

First, health has been increasingly recognized as a key element of sustainable economic development [1], global security, effective governance, and human rights promotion [2]. Second, due to the growing perceived importance of health, unprecedented—albeit still insufficient—sums of funds are flowing into this sector [3]. Third, there is a burst of new initiatives coming forth to strengthen national health systems as the core of the global health system and a fundamental strategy to achieve the health-related Millennium Development Goals.

In order to realize the opportunities offered by the conjunction of these unique circumstances, it is essential to have a clear conception of national health systems that may guide further progress in global health.

Frenk offers some suggestions:

Part of the problem with the health systems debate is that too often it has adopted a reductionist perspective that ignores important aspects. Developing a more comprehensive view requires that we expand our thinking in four main directions.

First, we should think of the health system not only in terms of its component elements (like human resources, financing, hospitals, clinics, technologies, etc.) but most importantly in terms of their interrelations. Second, we should include not only the institutional or supply side of the health system, but also the population. In a dynamic view, the population is not an external beneficiary of the system; it is an essential part of it.

It’s important to note the mention of the role of the population and its dynamical impact on the system. As populations change dramatically in their composition and form of residency within countries, including a greater movement to urbanization, so too will the myriad factors that influence health systems. The people are the system and thus it will change as populations change. While Frenk lists this as one point of many, it is a radical departure for reductionists or those who see health systems as being about care, not people.

A third expansion of our understanding of systems refers to their goals. Typically, we have limited the discussion to the goal of improving health. This is, indeed, the defining goal of a health system. However, we must look not only at the level of health, but also at its distribution, which gives equity a central place in assessing a health system. In addition, we must also include other goals that are intrinsically valued beyond the improvement of health. One of those goals is to enhance the responsiveness of the health system to the legitimate expectations of the population for care that respects the dignity of persons and promotes their satisfaction. The other goal is fair financing, so that the burden of supporting the system is distributed in an equitable manner and families are protected from the financial consequences of disease.

Frenk’s third challenge is to affirm the very point of health systems at all.

While not explicitly speaking of systems thinking or design thinking, there is much that both fields have in common with Frenk’s argument. Design thinkers might ask: What have we hired our health system to do?

Frenk argues that our health systems must go well beyond just making gains in measured health outcomes towards dignity, respect and social justice.

Finally, we should expand our view with respect to the functions that a health system must perform. Most global initiatives have been concerned mainly with one of those functions, namely, the direct provision of services, whether they are medical or public health services. This is, of course, an essential function, but for it to happen at all, health systems must perform other enabling functions, such as stewardship, financing, and resource generation, including what is probably the most complex of all challenges, the health workforce.

Frenk did not identify specific solutions, but did pose some key questions for health systems design.

If we were to take this challenge up as designers and systems thinkers, what might we do? Here are some suggestions for inquiry:

  • Consider new definitions of health like the one posed in the British Medical Journal that emphasizes looking at the social and environmental influences on health beyond just the absence of physical symptoms. Further inclusion of a psychology of human flourishing might add to this definition.
  • Map out a new system visually with people at the centre, not professionals or institutions. What does that look like? Tools like a Gigamap might provide the kind of multi-media, multi-sensory visual way to conceive of the interrelationships that make up health system. System dynamic models can help this out as well.
  • Engage people across this system to validate this map and co-create possible future models that could serve to shape discussion at multiple levels and  mobilize civil society to support healthy environments.
  • Create small scale, safe-fail / fail-forward, prototypes of small-scale innovations that can be tested, developmentally designed, and rapidly re-developed as needed to start shifting the system as a whole.

Designing health requires designing health systems. Applying new thinking and envisioning a system that is dynamic, comprised of people and just institutions is a start.

Photo: Bartolomeo Eustachi: Peripheral Nervous System, c. 1722 shared by brain_blogger used under Creative Commons Licence

1 2 3 … 7