Evaluation is supposed to be driven by a program’s needs and activities, but that isn’t always the case. What happens when the need for numbers, metrics, ‘outcomes’ and data shape the very activities programs do and how that changes everything is something that is worth paying some attention to.
Since the Second World War we’ve seen a gradual shift towards what has been called presence of neo-liberal values across social institutions, companies, government and society. This approach to the world is characterized, among other things, by its focus on personal and economic efficiency, freedom, and policies that support actions that encourage both. At certain levels of analysis, these policies have rather obvious benefits.
Who wouldn’t like to have more choice, more freedom, more perceived control and derive more value from their products, services and outputs? Not many I suspect. Certainly not me.
Yet, when these practices move to different levels and systems they start to produce enormous complications that are at odds with — and produce distortions of — the very values that they espouse. We’ve seen the same happen with other value systems that have produced social situations that are highly beneficial in some contexts and oppressive and toxic in others – capitalism and socialism both fit this bill.
Invisible tails and wags
What makes ‘isms’ so powerful is that they can become so prevalent that their purpose, value and opportunity stop being questioned at all. It is here that the tail starts to wag the dog.
Take our economy (or THE economy as it is somewhat referred to). An economy is intended to be a facilitator and product of activities used to create certain types of value in a society. We work and produce goods (or ideas), exchange and trade them for different things, and these allow us to fulfill certain human goals. It can take various shapes, be regulated more or less, and can operate at multiple scales, but it is a human construction — we invented it. Sometimes this gets forgotten and in times when we use the economy to justify behaviour we forget that it is our behaviour that is the economy.
We see over and again with neoliberalism (which is among the most dominant societal ‘ism’ of the past 50 years in the West and more reflected globally all the time) taken at the broadest level, the economy becomes the central feature of our social systems rather than a byproduct of what we do as social beings. Thus, things like goods, experiences, relations and so on we used to consider as having some type of inherent value suddenly become transformed into objects that judgements can be made.
The role of systems
This can make sense where there are purpose-driven reasons to assign particular value scores to something, but the nature of value is tied to the systems that surround what is valued. If we are dealing with simple systems, those where there are clear cause-and-effect connections between the product or service under scrutiny and its ability to achieve its purpose, then valuation measurement makes sense. We can assert that X brand of laundry detergent is better than Y on the basis of Z. We can conduct experiments, trials and repeated measures that can compare across conditions.
It is also safe to make an assumption of value based on the product’s purpose that can be generalized. In other words, our reason for using the product is clear and relatively unambiguous (e.g., to clean clothes using the above example). There may be additional reasons for choosing X brand over Y, but most of those reasons can be also controlled for and understood discretely (e.g., scent, price, size, bottle shape etc..).
This kind of thinking breaks down in complex systems. And to make it even more complex, it breaks down imperfectly so we have simple systems interwoven within complex ones. We have humans using simple products and services that operate in new, innovative and complex conditions. Unfortunately, what comes with simple systems is simple thinking. Because they are — by their nature — simple, these system dynamics are easy to understand. Returning to our example of the economy, classical micro-economic models of supply and demand as illustrated below.
Relationships and the systems that surround them
Using this model, we can do a reasonable job of predicting influence, ascertaining value and hypothesizing relationships between both.
In complex systems, the value links are often in flux, dynamic, and relative requiring a form of adaptive evaluation like developmental evaluation. But that doesn’t happen as much as it should, mostly because of a failure to question the systems and their influence. Without questioning the values and value that systems create — the isms that were mentioned earlier — and their supposed connection to outcomes, we risk measuring things that have no clear connection to value and worse, we create systems that get designed around these ineffective measures.
What this manifests itself in is mindless bureaucracy, useless meetings, pompous and intelligible titles, and innovation-squashing regulations that get divorced from the purpose that they are meant to solve. And in doing so, this undermines the potential benefit that the original purpose of a bureaucracy (to document and create an organizational memory to guide decisions), meetings (to discuss and share ideas and solve problems), titles (to denote role and responsibility — although these aren’t nearly as useful as people think in the modern organization), and regulations (to provide a systems lens to constrain uncoordinated individual actions from creating systems problems like the Tragedy of the Commons).
More importantly, this line of thinking also focuses us on measuring the things that don’t count. And as often quoted and misquoted, the phrase that is apt is:
Not everything that counts can be counted, and not everything that can be counted counts.
Counting what counts
It is critical to be mindful of the purpose — or to reconnect, rediscover, reinvent and reflect upon the purposes we create lest we allow our work to be driven by isms. Evaluators and their program clients and partners need to stand back and ask themselves: What is the purpose of this system I am dealing with?
What do we measure and is that important enough to matter?
Perhaps the most useful way of thinking about this is to ask yourself: what is this system being hired to do?
Regular mindful check-ins as part of reflective practice at the individual, organizational and, where possible, systems level are a way to remind ourselves to check our values and practices and align and realign them with our goals. Just as a car’s wheels go out of alignment every so often and need re-balancing, so too do our systems.
In engaging in reflective practice and contemplating what we measure and what we mean by it we can better determine what part of what we do is the dog, what is the tail and what is being wagged and by whom.
Photo credit: Wagging tail by Quinn Dombrowski used under Creative Commons License via Flickr. Thanks Quinn for making your great work available to the world.
Economic model image credit from Resources for Teachers used under Creative Commons License. Check out their great stuff for helping teachers teach better.