Tag: science

behaviour changecomplexitydesign thinkingevaluationpsychology

Exploding goals and their myths

on July 20, 2017

SparkWall_Snapseed.jpg

Goal-directed language guides much of our social policy, investment and quests for innovation without much thought of what that means in practice. Looking at the way ideas start and where they carry us might offer us reasons to pause when fashioning goals and whether we need them at all. 

In a previous article, I discussed the problems with goals for many of the problems being dealt with by organizations and networks alike. (Thanks to the many readers who offered comments and kudos and also alerted me that subscribers received the wrong version minus part of the second paragraph!). At aim was the use of SMART goal-setting and how it made many presumptions that are rarely held as true.

This is a follow-up to that to discuss how a focus on the energy directed toward a goal and how it can be integrated more tightly with how we organize our actions at the outset might offer a better option than addressing the goals themselves.

Change: a matter of energy (and matter)

goal |ɡōl| noun:  the object of a person’s ambition or effort; an aim or desired result • the destination of a journey

A goal is a call to direct effort (energy) toward an object (real or imagined). Without energy and action, the goal is merely a wish. Thus, if we are to understand goals in the world we need to have some concept of what happens between the formation of the goal (the idea, the problem to solve, the source of desire), the intention to pursue such a goal, and what happens on the journey toward that goal. That journey may involve a specific plan or it may mean simply following something (a hunch, a ‘sign’ — which could be purposeful, data-driven or happenstance, or some external force) along a pathway.

SMART goals and most of the goal-setting literature takes the assumption that a plan is a critical success factor in accomplishing a goal.

If you follow SMART, Specific, Measurable, Attainable, Realistic, and Time-bound (or Timely) this plan needs to have these qualities attached to them. This approach makes sense when your outcome is clear and the pathway to achieving the goal is also reasonably clear such as smoking cessation, drug or alcohol use reduction, weight loss and exercise. It’s the reason why so much of the behaviour change literature includes goals: because most of it involves studies of these kinds of problems. These are problems with a clear, measurable outcome (even if that has some variation to it). You smoke cigarettes or you don’t. You weigh X kilograms at this time point and Y kilograms at that point.

These outcomes (goals) are the areas where the energy is directed and there is ample evidence to support means to get to the goal, the energy (actions) used to reach the goal, and the moment the goal is achieved. (Of course, there are things like relapse, temporary setbacks, non-linear changes, but researchers don’t particularly like to deal with this as it complicates things, something clinicians know too well).

Science, particularly social science, has a well-noted publication bias toward studies that show something significant happened — i.e., seeing change. Scientists know this and thus consciously and unconsciously pick problems, models, methods and analytical frameworks that better allow them to show that something happened (or clearly didn’t), with confidence. Thus, we have entire fields of knowledge like behaviour change that are heavily biased by models, methods and approaches designed for the kind of problems that make for good, publishable research. That’s nice for certain problems, but it doesn’t help us address the many ones that don’t fit into this way of seeing the world.

Another problem is much less on the energy, but on the matter. We look at specific, tangible outcomes (weight, presence of cigarettes, etc..) and little on the energy directed outward. Further, these perspectives assume a largely linear journey. What if we don’t know where we’re going? Or we don’t know what, specifically, it will take to get to our destination (see my previous article for some questions on this).

Beyond carrots & sticks

The other area where there is evidence to support goals is from management and study of its/ executives or ‘leaders’ (ie. those who are labelled leaders and might be because of title or role, but whether they actually inspire real, productive followership is another matter). These leaders call out a directive and their employees respond. If employees don’t respond, they might be fired or re-assigned — two outcomes that are not particularly attractive to most workers. On the surface it seems like a remarkably effective way of getting people motivated to do something or reach a goal and for some problems it works well. However, those type of problem sets are small and specific.

Yet, as much of the research on organizational behaviour has shown (PDF), the ‘carrot and stick’ approach to motivation is highly limited and ineffective in producing long-term change and certainly organizational commitment. Fostering self-determination, or creating beauty in work settings — something not done by force, but by co-development — are ways to nurture employee happiness, commitment and engagement overall.

A 2009 study, appropriately titled ‘Goals Gone Wild’ (PDF), looked at the systemic side-effects of goal-setting in organizations and found: “specific side effects associated with goal setting, including a narrow focus that neglects non-goal areas, a rise in unethical behavior, distorted risk preferences, corrosion of organizational culture, and reduced intrinsic motivation.” The authors go on to say in the paper — right in the abstract itself!: “Rather than dispensing goal setting as a benign, over-the-counter treatment for motivation, managers and scholars need to conceptualize goal setting as a prescription-strength medication that requires careful dosing, consideration of harmful side effects, and close supervision.”

Remember the last time you were in a meeting when a senior leader (or anyone) ensured that there was sufficient time, care and attention paid to considering the harmful side-effects of goals before unleashing them? Me neither.

How about the ‘careful dosing’ or ‘close supervision’ of activities once goal-directed behaviour was put forth? That doesn’t happen much, because process-focused evaluation and the related ongoing sense-making is something that requires changes in the way we organize ourselves and our work. And as a recent HBR article points out: organizations like to use the excuse that organizational change is hard as a reason not to make the changes necessary.

Praxis: dropping dualisms

The absolute dualism of goal + action is as false as the idea of theory + practice, thought + activity. There are areas like those mentioned above where that conception might be useful, yet these are selective and restrictive and can keep us focused on a narrow band of problems and activity. Climate change, healthy workplaces, building cultures of innovation, and creating livable cities and towns are not problem sets that have a single answer, a straightforward path, specific goals or boundless arrays of evidence guiding how to address them with high confidence. They do require a lot of energy, pivoting, adapting, sense-making and collaboration. They are also design problems: they are about making the world we want and reacting the world we have at the same time.

If we’re to better serve our organizations and their greater purpose, leaders, managers, and evaluators would be wise to focus on the energy that is being used, by whom, when, how and to what effect at more close intervals to understand the dynamics of change, not just the outcomes of it. This approach is one oriented toward praxis, an orientation that sees knowledge, wisdom, learning, strategy and action as combined processes that ought not be separated. We learn from what we do and that informs what we do next and what we learn further. It’s also about focusing on the process of design — that creation of the world we live in.

If we position ourselves as praxis-oriented individuals or organizations, evaluation is part of regular attending to the systems we design to support goals or outcomes through data and sensemaking. Strategy is linked to this evaluation and the outcomes that emerge from it all is what comes from our energy. Design is how we put it all together. This means dropping our dualisms and focusing more on integrating ourselves, our aspirations and our activities together toward achieving something that might be far greater than any goal we can devise.

Image credit: Author

 

 

complexityeducation & learningevaluationinnovationknowledge translation

You Want It Darker?

on November 11, 2016

shutterstock_150465248

It is poetic irony on many levels that weeks after Leonard Cohen releases his album about the threat of death that he passes on, mere days after we saw the least poetic, most crass election campaign end in the United States with an equally dramatic outcome. This points to art, but also to the science of complexity and how we choose to approach this problem of understanding– and whether we do at all — will determine whether we choose to have things darker or not. 

A million candles burning for the love that never came
You want it darker
We kill the flame

Canadian-born and citizen-of-the-world poet, literary author, and songwriter Leonard Cohen passed away last night and the words above were part of his final musical contribution to the world. It is fitting that those words were penned at time not only when Cohen was ill and dying, but also as we’ve witnessed the flames of social progress, inclusion, and diversity fall ill.

Donald Trump is the president-elect of the United States, a fact that for many is not only unpalatable, but deeply troubling for what it represents. A Trump presidency and the social ills that have been linked to his campaign are just the latest sign that we are well into a strange, fear-ful, period of history within Western democracies. His was not a win for ideas, policy, but personality and as a vector for many other things that simply cannot be boiled down exclusively to racism, sexism, celebrity, or education — although all of those things played some part. It was about the complexity of it all and the ability for simplicity to serve as a (false) antidote.

No matter what side of the political spectrum you sit, it’s hard to envision someone less suited to the job of President of a diverse, powerful nation like the United States than Donald Trump using any standard measure of leadership, personality, experience, personal integrity or record of public conduct. Yet, he’s in and his election provides another signal that we are living in complex times and, like with Brexit, the polls got it very wrong.

We are seeing global trade shrink at a time when globalization is thought to be at its highest. We are witnessing high-profile acts of hatred, discrimination and abuse at at time when we have more means to be socially connected across contexts than ever before. We are lonely when the world and connection is at our fingertips.  It is a time of paradox and when we have so many means to cast light on the world, we seem to find new ways to kill the flame.

It is for this reason that those who deal with complexity and seek positive social change in the world need to take action lest things get darker.

Complexity just got real

The election of Donald Trump and the Brexit vote are two examples that should serve to wake-up anyone who seeks greater accounting of complexity in the making of social decisions.

This is not about voting for a Republican President or for citizens wanting greater control of Britain, it’s about understanding the premise of which those decisions were based on. The amount of cognitive dissonance required to assume that Donald Trump has the qualities befitting a leader of a country like the United States is truly astounding. And just like Brexit, the theories and models proposed post-event by the same people who predicted the opposite outcome pre-event will be just words, backed with too little understanding of complexity or why things actually happened.

Those who understand complexity know that these simplistic explanations are likely to be problematic. But that doesn’t make us better people, but it does mean we have certain responsibilities.

Complexity rhetoric vs science

For those who rely on complexity science as a means of understanding these kinds of events its now time to start matching the science to our rhetoric so we can back up the talk. In crude, but truth-speaking pop culture parlance: “This shit just got real“.

As complexity and systems thinking has gained attention in social science and policy studies we are seeing much more attention to the idea of complexity. Yet, the level of rhetoric on social complexity has overwhelmed any instances of evidence of how complexity actually is manifest, emergent, harnessed, or accounted for in practical means.

This isn’t to say that the tenets of complexity for understanding social systems aren’t true, but rather we don’t know that it’s true for sure and to what extent in what situations. I write this as a true-believer, but also as one who believes in science. Science is about challenging our beliefs and only if we cannot refute our theories through our best efforts can claim something is true. Thus, if we can’t show consistently how the principles of complexity are employed to make useful choices and inform the documentation of some of the outcomes related to our actions based on those choices, we are simply making fables not flourishing organizations, communities and societies.

Showing our work

Without something more than rhetoric to back our claims up we become no better than a politician claiming to make America great again because we’ve got great ideas and will be the greatest president ever because we have great ideas.

This is not about reverting to positivist science to understand the entire world, but about responsible practice in evaluation and research that allows us to document what we do and explore the consequences in context. Powered by complexity theory and the appropriate methods, we can do this. Yet, too often I hear reference to complexity theories in presentations, discussions and papers without any reference to how its been used in real terms (and not just extracted from some other realm of science like bee colonies, natural ecosystems and simulation models) to influence something of value beyond serving as an organizing framework.

Like little kids in math class: we need to show our work.

How did complexity manifest in practice in this case? What methods were used to systematically document the process? How does this fit / challenge the theories we know? These are questions that are what responsible scientists and evaluators ask of their subjects and its time to do this with complexity, regularly and often. No longer can we give it the relatively unchallenged ride it’s been given since first being introduced as a viable contributor to social theory about 20 years ago.

The reasons have to do with what happens when we stop trying to understand complex systems.

Evaluators and social sciences’ new moral imperative

As the US election was unfolding I became aware of some prescient, wise words that were uttered by former US Supreme Court Justice David Souter speaking at a town hall prior to the last election. His words were chilling to anyone paying attention to the world today. In the quote and interview (see link) he says on the matter of government and democracy:

What I worry about is that when problems are not addressed, people will not know who is responsible.

His words are not just about the United States or even politics alone. The further we get from understanding how our social, economic, political and environmental systems work the more we all become vulnerable to the kind of simplistic thinking that leads us to someone that embodies H.L. Mencken’s mis-paraphrased words*:

There is always an easy solution to every human problem — neat, plausible, and wrong

It is our duty as scientists and evaluators to show the world the work of the programs, policies and initiatives that are aimed at changing systems — no matter what that system is. We need to be better at telling the story of programs using data and communicating what we learn to the world. It’s our role to show the work of others and to let others see our work in the process. By doing so we can make a contribution to helping address what Justice Souter meant about people not knowing who is responsible.

And like Mencken’s message, our answer won’t be one that is all that neat, but we if we approach our work with the wisdom and knowledge of how systems work we can avoid Mencken’s trap and avoid presenting the complex as simple, but we will go further and illustrate what complexity means.

It is our moral duty to do this. For if not us, who?

People do understand complexity. Anyone with a child or garden knows that there is no ‘standard practice’ that applies to all kids or any years’ crop of vegetables all the time in all cases. It’s evident all around us. We have the tools, theories and models to help illuminate this in the world and a duty to test them and make this visible to help shed that light on how our increasingly complex world works. Without that we are at risk of demagogues and the darker forces of our nature taking hold.

We have the means for people to see light through the work of those who build programs, policies and communities to illuminate our world. In doing so we not only create the candles as Leonard Cohen speaks of, but the curiosity and love that keeps that flame burning. We can’t kill the flame.

And we could use some love right now.

Thanks Leonard for sharing your gifts with us. I hope your art inspires us to reflect on what world you left to better create a world we move to.

*Mencken’s original quote was: “Explanations exist; they have existed for all time; there is always a well-known solution to every human problem — neat, plausible, and wrong.” Alas, this doesn’t make as pithy, Powerpoint worthy comment. Despite the incorrectness of the paraphrased quote attributed to Mencken, it’s fair to say that in many organizations we see this as a true statement nonetheless.

Image Credit: Shutterstock, used under licence.

knowledge translationsocial systemssystems thinking

When More is Less: The Information Paradox

on September 22, 2016

3187207970_5e4673af9d_o

There is a point at which information ceases to increase knowledge and understand and begins to undermine it, creating a paradox. When information on nearly anything is more abundant than ever the choices we make about how to engage it become more important than ever. 

The Information Age has been described as the period where industrial production was replaced by knowledge production as the key driver of social and economic benefit for society. Underpinning the thinking behind the information age (and the digital revolution that accompanies it) is that having more information, more access to it and improved tools to use it to solve problems will improve life for everyone. Presented with a choice to have access to more information or less people will almost always choose more.

More information leads to more options, which equals more choice and more choice is about freedom and that is seen as an inherent social good derived from the capitalist system, which further leads to better choices, more freedom and greater happiness overall. At least, this is what we’ve been led to believe and Barry Schwartz explains this quite eloquently in the opening of talk embedded later in this post.

This is the theory of change that underpins information theory as its played out in modern capitalist societies. It’s also the source of many of our modern paradoxes and problems.

Systems of influence: The case of the ePatient

I’ve stopped going to health-related hackathons and design jams altogether for the simple reason that one can almost always guarantee that one third or more of the solutions generated will be some form of app or information-focused tool. These well-meaning, creative tools are part of a consumer health movement that is all about putting information in the hands of patients with the idea that putting information in the hands of patients is the key to empowerment and better health outcomes, except they rarely lead to this promised land.

Few are better at explaining — and indeed living — this reality than Dave deBronkart or ‘e-Patient Dave’ who has been a tireless advocate for better information tools, access and engagement on health for patients. His Ted Talk captures the spirit of the movement nicely.

With all due respect to the positive sentiments around what the ePatient movement is about, it is based on a series of assumptions about health systems, patients and health itself in ways that don’t always hold. For certain patients, certain conditions, and certain contexts having more information delivered in the right format is indeed empowering and may be life saving as deBronkart’s story illustrates. But what’s often missing from these stories of success are the many layers of assumptions and conditions that underpin information-driven healthcare.

A few years ago I interviewed a patient who spoke about his experience with health care decision-making and information technology and his response was that having more information didn’t make his life much better, rather it made it even more complicated because with more access to more information he had more responsibility related to that information.

“I don’t know what to do with it all and there’s an assumption that once I know (this health information) I am in a position to do something. I don’t have the foggiest idea what to do, that’s why I am going to see (the health professionals) because that is what their job is for. They are the ones who are supposed to know what is to be done. It’s their world, not mine.”

This case is less about deferral to authority, but about resources (e.g., knowledge, skill, time, networks, etc..) and expectations around what comes with those resources. When you are unwell the last thing you want is to be told you have even more work to do.

The assumptions around personal health information and decision-making are that people have:

1) access to the data in the first place, 2) time, 3) information gathering tools, 4) knowledge synthesis tools, 5) skill and knowledge of how to sift, sort, synthesize and sense-make all the information obtained (because it may be in different formats, incomplete, or require conversions), 6) access to the people and other knowledge and skills required to appropriately sense-make the data, 7) the resources to act on whatever conclusions are drawn from that process, 8) a system that is able to respond to the actions that are needed and taken (and in a timely manner), 9) the personal willpower, energy, and resolve to persist through the various challenges and pushback from the system to resist the actions taken, 10) social support (because this is virtually impossible to do without any support at all) and 11) the motivation and interest in doing all of this in the first place.

Dave deBronkart and his peers are advocating for patient engagement on a broader level and that includes creating spaces for patients to have the choice as to what kind of information they use or not. This also means having choice to NOT have information. It’s not about technology pushing, but having a choice about what to access, when and how. That’s noble and often helpful to those who are used to not having much say in what happens, but that, too has problems of its own.

The paradox of choice

Barry Schwartz’s work (pdf) doing and synthesizing research on consumer decision-making puts truth to this lie that more choice is better. Choice options add value only to a certain point after which they degrade value and even subvert it altogether. The problem is that choice options are often ‘all or nothing’ and may be addictive if left unconstrained as we’ll see below.

Schwartz addresses the matter of decision-making in healthcare in the above video and points to the shifting of responsibility away from experts to everyone. Perhaps it is not surprising that we are seeing an incredible backlash against expert-driven knowledge and science in a way that we’ve not seen in over a hundred years. This is at a time when the public has access to more scientific data — the same data that scientists and other experts have — through open data and open access scientific publications to validate the claims by experts.

As discussed in a previous post, another feature of this wealth of information is that we are now living in what some call a post-truth political climate where almost anything goes. Speaking on the matter of science and climate change former Alaska Governor and Vice Presidential candidate Sarah Palin suggested that, when compared to Dr Bill Nye (the Science Guy and a rocket scientist — yes, a real rocket scientist ), she is as much of a scientist as he is.

Why have science when you can have opinion?

Distracted driving on the information superhighway

Recent data from Canada shows that year-over-year growth in smartphone use at 24% to over two thirds of the population with 85% reporting some form of mobile phone ownership. One of the key features of modern smartphones is the ‘always on’ nature of their tools and alert systems allowing you to bring maps, address books, a digital library, video and audio telephony, and the entire Internet in your pocket.

The distractions that come from the tools meant to deliver information are becoming crippling to some to the point of distancing us from our humanity itself. The title of a beautiful, sad piece in New York Magazine by Andrew Sullivan put this into perspective: I used to be a human being. (We will come back to this in a future post.)

But even if one still feels human using information technology, its a different experience of humanity than it once was. Behaviour change writer and coach Tony Schwartz (I’m not sure if he’s related to Barry), writing in the New York Times magazine, noted how his use of information technology was affecting his ability to, ironically, glean information from something simple as a book.

One evening early this summer, I opened a book and found myself reading the same paragraph over and over, a half dozen times before concluding that it was hopeless to continue. I simply couldn’t marshal the necessary focus.

He goes on to explain what is being exchanged for the books he had aspired to read:

Instead of reading them, I was spending too many hours online, checking the traffic numbers for my company’s website, shopping for more colorful socks on Gilt and Rue La La, even though I had more than I needed, and even guiltily clicking through pictures with irresistible headlines such as “Awkward Child Stars Who Grew Up to Be Attractive.”

We can laugh at the last bit because most of us have been online and lured by something we thought was impossible or ridiculous and had to inquire about. Link bait is not new or particularly clever, but it works. It works for a variety of reasons, but largely because we need to inhabit the same space to work as well as to play. The problem comes when these worlds cross-over into one another.

For example, I recently was shopping for a safe (no, it’s not to store my non-existent millions, but rather protect hard drives and enhance data security) and wanted to return to a story I’d read in the Guardian for a different blog post. As I returned to pull the URL for this I found the page looking like this:

screen-shot-2016-09-21-at-7-25-06-pm

All of a sudden I am confronted with shopping choices amidst a quest for a URL.

Information wealth: A Faustian bargain to knowledge poverty?

“We willingly accept the loss of concentration and focus, the division of our attention and the fragmentation of our thoughts, in return for the wealth of compelling or at least diverting information we receive.”

Tony Shwartz’s comments above and below point to what we know about how information works in our brain. We can try and resist, but the evolutionary reasons we pay attention to things and the biological limitations we have to processing it all are most likely to trump any efforts to resist it without substantial shifts to our practices.

Endless access to new information also easily overloads our working memory. When we reach cognitive overload, our ability to transfer learning to long-term memory significantly deteriorates. It’s as if our brain has become a full cup of water and anything more poured into it starts to spill out.

I wish I had the answers to what these are. Schwartz, has proposed a digital vacation. As beneficial as it was for him, he was also willing to admit that it’s not a perfect strategy and that he still spends too much time online, too distracted. But, its better.

Comedian Louis C.K. has taken to ‘quitting the internet’ altogether and, in a touching moment of reflection (as he often does with wit), notes how it has improved the relationship with his daughters.

It’s these relational aspects of the new information technology and how it impacts our world that concern me the most and creates the most troubling paradox: the tools that are designed to bring us together might just be making it harder to be together and pushing us apart from each other and ourselves. This is what I will look at in the next piece in this series on paradox.

Image credit: Information by Heath Brandon used under Creative Commons License and by author

 

 

behaviour changecomplexitypsychologysocial innovationsocial systems

Decoding the change genome

on April 12, 2016

11722018066_c29148186b_k

Would we invest in something if we had little hard data to suggest what we could expect to gain from that investment? This is often the case with social programs, yet its a domain that has resisted the kind of data-driven approaches to investment that we’ve seen in other sectors and one theory is that we can approach change in the same way we code the genome, but: is that a good idea?

Jason Saul is a maverick in social impact work and dresses the part: he’s wearing a suit. That’s not typically the uniform of those working in the social sector railing against the system, but that’s one of the many things that gets people talking about what he and his colleagues at Mission Measurement are trying to do. That mission is clear: bring the same detailed analysis of the factors involved in contributing to real impact from the known evidence that we would do to nearly any other area of investment.

The way to achieving this mission is to take the thinking behind the Music Genome Project, the algorithms that power the music service Pandora, and apply it to social impact. This is a big task and done by coding the known literature on social impact from across the vast spectrum of research from different disciplines, methods, theories and modeling techniques. A short video from Mission Measurement on this approach nicely outlines the thinking behind this way of looking at evaluation, measurement, and social impact.

Saul presented his vision for measurement and evaluation to a rapt audience in Toronto at the MaRS Discovery District on April 11th as part of their Global Leaders series en route to the Skoll World Forum ; this is a synopsis of what came from that presentation and it’s implications for social impact measurement.

(Re) Producing change

Saul began his presentation by pointing to an uncomfortable truth in social impact: We spread money around with good intention and little insight into actual change. He claims (no reference provided) that 2000 studies are published per day on behaviour change, yet there remains an absence of common metrics and measures within evaluation to detect change. One of the reasons is that social scientists, program leaders, and community advocates resist standardization making the claim that context matters too much to allow aggregation.

Saul isn’t denying that there is truth to the importance of context, but argues that it’s often used as an unreasonable barrier to leading evaluations with evidence. To this end, he’s right. For example, the data from psychology alone shows a poor track record of reproducibility, and thus offers much less to social change initiatives than is needed. As a professional evaluator and social scientist, I’m not often keen to being told how to do what I do, (but sometimes I benefit from it). That can be a barrier, but also it points to a problem: if the data shows how poorly it is replicated, then is following it a good idea in the first place? 

Are we doing things righter than we think or wronger than we know?

To this end, Saul is advocating a meta-evaluative perspective: linking together the studies from across the field by breaking down its components into something akin to a genome. By looking at the combination of components (the thinking goes) like we do in genetics we can start to see certain expressions of particular behaviour and related outcomes. If we knew these things in advance, we could potentially invest our energy and funds into programs that were much more likely to succeed. We also could rapidly scale and replicate programs that are successful by understanding the features that contribute to their fundamental design for change.

The epigenetic nature of change

Genetics is a complex thing. Even on matters where there is reasonably strong data connecting certain genetic traits to biological expression, there are few examples of genes as ‘destiny’as they are too often portrayed. In other words, it almost always depends on a number of things. In recent years the concept of epigenetics has risen in prominence to provide explanations of how genes get expressed and it has as much to do with what environmental conditions are present as it is the gene combinations themselves . McGill scientist Moshe Szyf and his colleagues pioneered research into how genes are suppressed, expressed and transformed through engagement with the natural world and thus helped create the field of epigenetics. Where we once thought genes were prescriptions for certain outcomes, we now know that it’s not that simple.

By approaching change as a genome, there is a risk that the metaphor can lead to false conclusions about the complexity of change. This is not to dismiss the valid arguments being made around poor data standardization, sharing, and research replication, but it calls into question how far the genome model can go with respect to social programs without breaking down. For evaluators looking at social impact, the opportunity is that we can systematically look at the factors that consistently produce change if we have appropriate comparisons. (That is a big if.)

Saul outlined many of the challenges that beset evaluation of social impact research including the ‘file-drawer effect’ and related publication bias, differences in measurement tools, and lack of (documented) fidelity of programs. Speaking on the matter in response to Saul’s presentation, Cathy Taylor from the Ontario Non-Profit Network, raised the challenge that comes when much of what is known about a program is not documented, but embodied in program staff and shared through exchanges.  The matter of tacit knowledge  and practice-based evidence is one that bedevils efforts to compare programs and many social programs are rich in context — people, places, things, interactions — that remain un-captured in any systematic way and it is that kind of data capture that is needed if we wish to understand the epigenetic nature of change.

Unlike Moshe Szyf and his fellow scientists working in labs, we can’t isolate, observe and track everything our participants do in the world in the service of – or support to – their programs, because they aren’t rats in a cage.

Systems thinking about change

One of the other criticisms of the model that Saul and his colleagues have developed is that it is rather reductionist in its expression. While there is ample consideration of contextual factors in his presentation of the model, the social impact genome is fundamentally based on reductionist approaches to understanding change. A reductionist approach to explaining social change has been derided by many working in social innovation and environmental science as outdated and inappropriate for understanding how change happens in complex social systems.

What is needed is synthesis and adaptation and a meta-model process, not a singular one.

Saul’s approach is not in opposition to this, but it does get a little foggy how the recombination of parts into wholes gets realized. This is where the practical implications of using the genome model start to break down. However, this isn’t a reason to give up on it, but an invitation to ask more questions and to start testing the model out more fulsomely. It’s also a call for systems scientists to get involved, just like they did with the human genome project, which has given us great understanding of what influences our genes have and stressed the importance of the environment and how we create or design healthy systems for humans and the living world.

At present, the genomic approach to change is largely theoretical backed with ongoing development and experiments but little outcome data. There is great promise that bigger and better data, better coding, and a systemic approach to looking at social investment will lead to better outcomes, but there is little actual data on whether this approach works, for whom, and under what conditions. That is to come. In the meantime, we are left with questions and opportunities.

Among the most salient of the opportunities is to use this to inspire greater questions about the comparability and coordination of data. Evaluations as ‘one-off’ bespoke products are not efficient…unless they are the only thing that we have available. Wise, responsible evaluators know when to borrow or adapt from others and when to create something unique. Regardless of what design and tools we use however, this calls for evaluators to share what they learn and for programs to build the evaluative thinking and reflective capacity within their organizations.

The future of evaluation is going to include this kind of thinking and modeling. Evaluators, social change leaders, grant makers and the public alike ignore this at their peril, which includes losing opportunities to make evaluation and social impact development more accountable, more dynamic and impactful.

Photo credit (main): Genome by Quinn Dombrowski used under Creative Commons License via Flickr. Thanks for sharing Quinn!

About the author: Cameron Norman is the Principal of Cense Research + Design and assists organizations and networks in supporting learning and innovation in human services through design, program evaluation, behavioural science and system thinking. He is based in Toronto, Canada.

journalismknowledge translationpublic healthscience & technology

The Power, Peril and Promise of Health Journalism

on February 14, 2015

Online Prescription Concept

The Toronto Star, Canada’s most widely read newspaper known for its investigative reporting gifted anti-vaccination audiences armament by using poor science to point to a spurious connection between an HPV vaccine and illness. The issue points to journalism’s power to shape the discourse of health issues and it points to the power, promise and peril associated with good (and not so good) science reporting. 

With great power comes great responsibility – Uncle Ben, Spiderman

It started with a story

On Thursday February 5th, 2015 the Toronto Star, Canada’s most widely read newspaper that has a reputation for solid investigative journalism, published an story that connected the experience of young girls and negative health effects with the receiving the Gardasil HPV vaccine. The story was immediately and widely criticized by experienced science journalists and health professionals alike, who argued that it was based on terribly flawed science.

The Toronto Star’s reaction was to defend itself, arguing in many different fora that they indeed mentioned that there was little scientific evidence that supported the link between the vaccine and the negative health effects being discussed in the article. The problem is that these links are buried deep in the article and certainly are not its focus: the hypothesized harms are.

Two days later, the Star published a follow-up op-ed letter which was authored by two health professionals and co-signed / supported by dozens of Toronto’s leading physicians condemning the original article. However, by that time the damage is likely to have been done and one more bit contribution to the fictitious ‘evidence’ for vaccine harms had been added to the anti-vaccine movement’s war chest.

Perpetuating harm

This matter of poor reporting is not a trivial issue. The fraudulent science performed by Andrew Wakefield linking autism to vaccines helped spur an evidence-thin anti-vaccination movement. Today, we are seeing the resurgence of diseases once thought to be eliminated in North America (like measles) because so many people are not having their children vaccinated. Jenny McCarthy is among the celebrities who have taken up the cause of anti-vaccination and has written about and spoken at length about what she sees as the connection between autism and vaccines, using her son’s experience as an ‘example’. Oprah Winfrey, perhaps unwittingly, gave McCarthy a platform to speak about her beliefs on her show offering wider possible credibility to something that has been thoroughly discredited in the scientific literature (PDF).

For the Toronto Star, it was bad enough that the story was published — and is now online, likely for all time in various forms thanks to the Web — but what made it worse was that the Star was so vigorous in its defence of it, unwilling or unable to recognize their role in public health. Medical evidence champion, author, physician and columnist Ben Goldacre was among the many who counter-attacked, pointing to what he called The Star’s ‘smear campaign‘ against the story’s critics.

For an interesting discussion of the issue of just how the Star got it wrong, listen to Vox health reporter Julia Belluz, interviewed on the CBC’s radio show The Current. Belluz, a past MIT Knight Journalism Fellow, is one of a dwindling number of journalists who understand the practice of reporting, science, and medicine and wrote a stellar critique of the Toronto Star article, but as importantly makes the case for why there is a need for specialized, trained, supported journalists out there doing this kind of work.

…and health

I’ve argued in the past that journalism is very much a pillar of public health. When it fails, so does public health. Journalism is not and should not be an arm of public health for the very independence that good, professional journalism strives to maintain is a reason it’s often called the fourth estate, keeping governments and other forces in check to ensure they are not abusive. Yet, that distance is also what makes it a part of public health. Public health is better for journalism and journalism certainly can benefit from health stories as they continue to be popular and sought after by readers.

As a group, scientists and many clinicians are not great at communicating what they do, why their research is important to others outside their field, and what the implications of their findings are for the public and science as a whole. Some are, most are not. It’s for this reason that the entire sub-field of health sciences focused on knowledge translation, exchange and mobilization has emerged. Just as we value the ability of a graphic designer to make visuals come alive, so too have we learned to value those with the skills to communicate information well and that is what journalists are trained and paid to do. They are a big part of this process, or at least should be.

Healthy journalism, healthy science, healthy people

Science journalism is too important to be ignored. There is much skepticism of journalists by scientists and clinicians and indeed, as the Toronto Star shows, journalists sometimes get things wrong. But its one thing to get it wrong through errors of judgement or interpretation it’s quite another to get things wrong by design. The Toronto Star has some good health reporters, but they weren’t the ones on this story. Nor did they bring in the health reporters to consult on this or other health professionals prior to publication– at least as far as one can tell.

The importance to the public’s health of good reporting requires that health and science journalists have more than a rudimentary knowledge of the topics they are covering. What’s strange is how we understand this with our sports reporting, weather forecasts and foreign correspondents. You wouldn’t watch someone who has little understanding of a sport covering it in depth, would you? It’s one thing to read scores, it’s another to provide investigative and deep coverage of a game if you don’t know the players, the rules, the criteria for quality and success and so forth.

Why do we do this with health journalism and science?

Yet, journalism is under pressure and no doubt the Toronto Star, for whatever genuine contrition they experience from what happened, have to like that they are being talked about. The reason is that journalism is under threat for market reasons, the Internet and the changing ways we get our news. It is, as Jürgen Krönig wrote way back in 2004, “A crisis of the Fourth Estate”. That crisis is only getting worse.

As anyone interested in public health, we need to take actions to ensure that the fourth estate is protected, supported and not ignored. Our health might just depend on it.

Image: iStockphoto, used under licence.

education & learningresearchsystems thinking

The urban legends of learning (and other inconvenient truths)

on July 28, 2014
Learning simulacrum, simulation or something else?

Learning simulacrum, simulation or something else?

Learning styles, technology-driven teaching, and self-direction are all concepts that anyone interested in education should be familiar with, yet the foundations for their adoption into the classroom, lab or boardroom are more suspect than you might think. Today we look at the three urban legends of learning and what that might mean for education, innovation and beyond. 

What kind of learner are you? Are you a visual learner perhaps, where you need information presented in a particular visual style to make sense of it? Maybe you need to problem-solve to learn because that’s the way you’ve been told is best for your education.

Perhaps you are a self-directed learner who is one that, when given the right encouragement and tools, will find your way through the muck to the answers and that others just need to get out of the way. With tools like the web and social media, you have the world’s knowledge at your disposal and have little need to be ‘taught’ that stuff, because its online.

And if you’re a digital native (PDF), this is all second nature to you because you’re able to use multiple technologies simultaneously to solve multiple problems together with ease if given the ability to do so. After all, you’ve had these tools your entire life.

A recent article by Paul Kirschner and Jeroen van Merriënboer published in the peer-reviewed journal Educational Psychologist challenges these ‘truths’ and many more, calling them urban legends:

An urban legend, urban myth, urban tale, or contemporary legend, is a form of modern folklore consisting of stories that may or may not have been believed by their tellers to be true.

The authors are quick to point out that there are differences in the way people approach material and prefer to learn, but they also illustrate that there is relatively little evidence to support much of the thinking that surrounds these practices, confusing learning preferences for learning outcomes. I’ve commented on this before, noting that too often learning is conflated with interest and enjoyment when they are different things and if we were really serious about it we might change the way we do a great deal many things in life.

In the paper, the authors debunk — or at least question — the evidence that supports the ‘legends’ of digital natives as a type of learner, the presence of specific learning styles and the need to customize learning to suit such styles of learning, and that of the lone self-educator. In each case, the authors present much evidence to challenge these ideas so as not to take them as truths, but hypotheses that have little support for them in practice.

Science and its inconvenient truths about learning

Science has a funny way of revealing truths that we may find uncomfortable or at least challenge our current orthodoxy.

This reminds me of a terrific quote from the movie Men in Black that illustrates the fragility of ideas in the presence and absence of evidence after one of the characters (played by Will Smith) uncovers that aliens were living on earth (in the film) and is consoled by his partner (played by Tommy Lee Jones) about what is known and unknown in the world:

Fifteen hundred years ago everybody knew the Earth was the center of the universe. Five hundred years ago, everybody knew the Earth was flat, and fifteen minutes ago, you knew that humans were alone on this planet. Imagine what you’ll know tomorrow.

One of the problems with learning is that there is a lot to learn and not all of it is the same in content, format and situational utility. Knowledge is not a ‘thing’ in the way that potatoes, shoes, patio furniture, orange juice, and pencils are things where you can have more or less of it and measure the increase, decrease and change in it over time. But we often treat it that way. Further, knowledge is also highly contextualized and combines elements that are stable, emergent, and transformative in new, complex arrangements simultaneously over time. It is a complex adaptive system.

Learning (in practice) resists simple truths.

It’s why we can be taught something over and again and not get it, while other things get picked up quickly within the same person even if the two ‘things’ seem alike. The conditions in which a person might learn are cultural (e.g., exposure to teaching styles at school, classroom designs, educational systems, availability and exposure to technology, life experiences, emphasis on reflective living/practice within society, time to reflect etc..) and psycho-social/biological (e.g., attention, intelligence, social proximity, literacy, cognitive capacity for information processing, ability to engage with others) so to reduce this complex phenomena to a series of statements about technology, preference and perception is highly problematic.

Science doesn’t have all the answers — far from it — but at least it can test out what is consistent and observable over time and build on that. In doing so, it exposes the responsibility we have as educators and learners.

With great power comes great responsibility…?

Underpinning the urban legends discussed by Kirschner and van Merriënboer and not discussed is the tendency for these legends to create a hands-off learning systems where workplaces, schools, and social systems are freed from the responsibility of shaping learning experiences and opportunities. It effectively reduces institutional knowledge, wisdom and experience to mere variables in a panoply of info-bites treated as all the same.

It also assumes that design doesn’t matter, which undermines the ability to create spaces and places that optimize learning options for people from diverse circumstances.

This mindset frees organizations from having to give time to learning, provide direction (i.e., do their own homework and set the conditions for effective learning and knowledge integration at the outset). It also frees us up from having to choose, to commit to certain ideas and theories, which means some form of discernment, priority setting, and strategy. That requires work up front and leadership and hard, critical, and time-consuming conversations about what is important, what we value in our work, and what we want to see.

When we assume everyone will just find their way we abdicate that responsibility.

Divesting resources and increasing distraction

In my home country of Canada, governments have been doing this with social investment for years where the federal government divests interest to the provinces who divest it to cities and towns who divest it to the public (and private) sector, which means our taxes never go up even if the demands on services do and we find that individual citizens are responsible for more of the process of generating collective benefit without the advantage of any scaled system to support resource allocation and deployment throughout society (which is why we have governments in the first place). It also means our services and supports — mostly — get smaller, lesser in quality, more spread thinly, and lose their impact because there isn’t the scaled allocation of resources to support them.

Learning is the same way. We divest our interests in it and before you know it, we learn less and do less with it because we haven’t the cultural capital, traditions or infrastructure to handle it. Universities turn campus life to an online experience. Secondary schools stop or reduce teaching physical education that involves actual physical activity.  Scholarly research is reduced to a Google search. Books are given up as learning vehicles because they take too long to read. It goes on.

It’s not that there are no advantages to some of these ideas in some bites, but that we are transforming the entire enterprise with next to no sense of the systems they are operating in, the mission they are to accomplish, a theory of change that is backed up by evidence, or the will to generate the evidence needed to advise and the resources to engage in the sensemaking needed to evaluate that evidence.

Science, systems and learning

It is time to start some serious conversations about systems, science and learning. It would help if we started getting serious about what we mean when we speak of learning, what theories we use to underpin that language and what evidence we have (or need) to understand what those theories mean in practice and for policy. This starts by asking better questions — and lots of them — about learning and its role in our lives and work.

Design thinking and systems thinking are two thinking tools that can help us find and frame these issues. Mindfulness and its ethics associated with non-judgement, open-mindedness, compassion and curiosity are also key tools. The less we judge, the more open we are to asking good questions about what we are seeing that can lead us to getting better answers rather than getting trapped by urban legends.

Doing this within a systems thinking frame also allows us to see how what we learn and where and how we learn is interconnected to better spot areas of leverage and problems in our assumptions.

This might allow us to make many of our urban legends obsolete instead of allowing them to grow like the alligators that live in the sewers of New York City. 

 

 

journalismknowledge translationpublic healthsocial media

Sane truths in Crazy Town: What Rob Ford’s story offers politics, science and journalism

on February 17, 2014
Crazy Town

Crazy Town

A new book about Toronto’s (in)famous mayor reveals a great deal more than just a story of man known more for what he smokes and says than his governance, to what kind of world we want to live in. Robyn Doolittle’s ‘Crazy Town’ goes well beyond documenting one man’s troubling behaviour and its place in the city he governs to a broader understanding of politics, science and journalism in a day when all three are under threat. 

Toronto has been my adopted home for most of last 15 years. It’s dynamic, clean, safe and North America’s 4th largest city. Toronto is a place of tremendous ethno-cultural diversity (near 1/2 of the population is foreign-born), spectacular food, a thriving arts and culture scene, great universities, home to sports fans with a near pathological faith in their hockey team, and — even with all of that — it’s sometimes a bit dull (and that’s OK).

That last bit about being dull changed dramatically after 2010 and that has to do with one man: Rob Ford, our mayor. Maybe you’ve heard of him.

The narrative arc

Toronto Star reporter Robyn Doolittle was literally at the front line of journalists covering Toronto’s Chief Magistrate and recently published a book on that experience and the story behind the story called Crazy Town. It’s a terrific book that documents the almost surreal events and people behind Rob Ford’s rise to power and current reign as one of the world’s most well-known mayors. It’s a rare work that manages to marry true crime, history, political intrigue, suspense, biography, and a journalism textbook together. I devoured it.

Yet, as a resident and politics fan I was amazed by what I read. I already knew most of the general details of what came out in the book (although chapter 12 is a complete shocker) because I lived through this news. Yet, it was only seeing all of this painted in one long narrative piece that it took a new life and in doing so brought me to a deeper understanding of many issues I’d thought I knew. The reason is largely the narrative arc that only a book (or long-form journalism) can offer.

On the surface, one could argue that what Doolittle did was piece together hundreds of stories she and others had written and compile them with a few additional quips to produce a compendium of Rob Ford’s life in the public’s eye. That in itself is a lot of work, but it doesn’t tell those who were paying attention to the story anything new. Yet, with each story that came out the backstory shows how what was reported — and picked up by others, reacted to, or ignored — was as important as what was learned about the subject and his environment. We read about how — not unlike with police work — the public is exposed to the “facts” but not how the authors chose to disclose (or not) those details and why.

When one considers what these ‘facts’ and the stories behind them entail, it is hard not to see some parallels between the world of political reporting at city hall and the world of science, social innovation, health promotion and policy that I live (and have lived) in. Crazy Town has many lessons for those not interested in Toronto, Rob Ford, politics, journalism or science, yet it is through all of those topics that such lessons are learned. The latter three stand out.

Politics

Rob Ford has defied nearly any explanation of how he has managed to maintain some form of support above 30% (as in, 3/10 polled would vote for him if the election was today). The best I’ve read is from former Canadian hockey legend, educator and parliamentarian Ken Dryden who wrote in the Globe and Mail newspaper about how Rob Ford has found a way to be visible and get the simple things done when other politicians get mired in complexity. He channels people’s frustrations and he makes his constituents feel listened to.

Doolittle’s treatment of Ford – despite the despicable treatment he’s given her, the Toronto Star and journalists overall — is fair and, in many cases, almost flattering when it comes to politics. Ford and his team have, despite appearances on the personal side of things, been very consistent and kept things simple. While Einstein might have challenged that Ford’s simple is too much so, there are lessons for all of us in this.

For those who deal in complexity, which is most human systems, it is easy to get mired in the details and interactions. Ford was steadfast in his over-arching narrative of “the gravy train” and that resonated with people. There is no reason why any other politician couldn’t have picked something similar to drive as their narrative and done much more good than Ford has, but they didn’t.

Ford made himself visible to those who mattered most: his constituents. And they have rewarded him with support.

How often do health care officials, educators, or policy leaders spend time with their key ‘constituents’ in settings that are natural to that audience? Politicos might challenge Ford’s proclivity for door-knocking and BBQ’s in an age of big data analytics, but that resonates with people. Why don’t more leaders get away from staid events in hotel ballrooms, well-crafted PR events, or their own offices to meet with their audiences where they live, work and play?

Good designers know that the design is only good if it gets used in the environment it was intended for and the only way to know that is to go into those environments. Ford knows this.

Science

To be fair, science is my term not Doolittles, but the term ‘evidence’ is one that links my term and her experience as a reporter. By science, I am talking capital ‘S’ science — the enterprise of scientific work as well as the activity.

What follows from the narrative arc that Ford delivered was the ability to frame the evidence held against him. He is masterful at reframing the arguments and keeping people focused on the messages that fit his ongoing  construction of a narrative. For a while, he was able to keep people talking about whether or not he smoked crack or drank alcohol excessively — two very serious issues — in a speculative way and away from the evidence he associated with drug dealers, violent criminals, and lied repeatedly to the press. He still does this.

In 2012 and 2013 the city spent time debating the minutiae of the law around whether or not he was in violation of conflict of interest. Lost in much of this debate was the larger pattern of Rob Ford consistently getting into trouble over all kinds of issues, big and small and how that wasn’t appropriate for any leader, political or not. Recently, Ford was in the news for being drunk in public and speaking in some faux Jamaican patois to customers at a local restaurant.

The issue as discussed in the media was the alcohol and the patois, not the fact that this is a man who, when under the public’s eye, has the judgement to: 1) get drunk in a public place 2) with the person who is accused of extortion related to the infamous crack video, 3) and then get up in front of everyone at the front of the restaurant to make a big, public proclamation.

Two weeks later, at a funeral for his friend’s mother in Vancouver, Ford decides to go to a crowded bar on a weekend night where nearly every young person there has a mobile phone and many proceed to take pictures of him or with him .

This is exactly how scientists and policy makers often behave. The intense focus on the small details leaves out the questions of relevancy and the bigger picture of what the point of the science is. Too often we get sidetracked with specifics and lose sight of a much larger set of issues.

For example, we’ll spend forever arguing the hypothetical possibility that someone might hack into an eHealth record as an argument for not allowing for easy portability and accessibility to that information (despite the fact that it can save lives, engage people, and that banks have been doing it with our life savings and credit for 20 years). (* Note that the details in science can matter a great deal, but just like walking and chewing gum, we can fret details in science and think of the big picture at the same time)

So far, people are willing to pay attention to Ford’s bigger message. Perhaps we need to consider what the bigger message is in our other enterprises and then worry about the details.

Journalism

I love ‘behind the scenes’ looks and this book provides lot to consider when thinking about how journalism is done, particularly that of the investigative kind. Doolittle has been steadfast that Crazy Town might have her name on the cover, but the investigative work that contributed to it was part of a huge team of journalists from the Toronto Star, the Globe and Mail and other outlets. Indeed, it takes a team and the kind of institutional support that the Star has put behind Doolittle.

Alas, this may be an exception. Many journalistic outlets are imploding due to poor management, change of readership habits, shifting business models, and also the public’s unwillingness to pay for things they value online. This last point is the one that we often let skate by in our discussions about media and one that Jaron Lanier has exposed as a major flaw in the modern Internet age.

Just this past week, web pioneer Mark Andreessen speculated on the future of media and — as many who have a stake in a faster, less in depth form of media often do — completely overlooked the role of the media as the a key role in communicating and uncovering key stories for society. To him, the model is dying. Maybe the business model is problematic, but unlike Andreessen I see a big need for journalism for society and as a model for science and health.

In health and science reporting, we are at great risk of losing voices like Andre Picard, Julia Belluz, Carly Weeks and Helen Branswell who have all brought to light many key issues that public health, healthcare and policy seem to forget, hide, complicate, or deny from emergent infectious disease patterns to drug regulation policy and practice.

Would we know about Rob Ford’s fitness for mayoralty if we didn’t have the Star? Would we be talking about the perversion of science and pharmaceuticals were it not for people like Ben Goldacre in the UK? What kind of knowledge would the world have about the NSA if Edward Snowden was a lone blogger and didn’t have The Guardian or New York Times to advance his disclosure? Crazy Town makes you realize what a debt we are owed to modern investigative journalism, journalists and those that support them (and are willing to pay for their products).

A bigger story

Crazy Town ends with the acknowledgement that there is much more of this story yet to be written. This is an election year and Rob Ford is one of the few who have already filed their papers to run for office again.

Crazy Town could have been told in 10,000 tweets, videos and Instagram pics. But it would have missed the point. The book is an argument for why in-depth journalism is needed and why — journalism, science, and politics — all often require a longer narrative arc to understand the bigger picture. Bigger stories don’t fit into a social media world, even if that very social media is part of the story itself.

The book is a great read whether you’re in Toronto, Ontario; Calgary, Alberta;  Madison, Wisconsin; or Phnom Phen, Cambodia. It’s a story as much about a man and a city as it is about ourselves and the world we live in. Read that way, you’ll find that not only is there more to tell of Rob Ford, there is a much bigger story to tell all around us.

1 2 3 … 6