Developmental evaluation is an approach (much like design thinking) to program assessment and valuation in domains of high complexity, change, and innovation. These three terms are used often, but poorly understood in real terms for evaluators to make much use of. This first in a series looks at the term complexity and what it means in the context of developmental evaluation.
Science writer and professor Neil Johnson is quoted as saying: “even among scientists, there is no unique definition of complexity – and the scientific notion has traditionally been conveyed using particular examples…” and that his definition of a science of complexity (PDF) is: “the study of the phenomena which emerge from a collection of interacting objects.” The title of his book Two’s Company, Three’s Complexity hints at what complexity can mean to anyone who’s tried to make plans with more than one other person.
The Oxford English Dictionary defines complexity as:
complexity |kəmˈpleksitē|
noun (pl. complexities)
the state or quality of being intricate or complicated: an issue of great complexity.
• (usu. complexities) a factor involved in a complicated process or situation: the complexities of family life.
For social programs, complexity involves multiple overlapping sources of input and outputs that interact with systems in dynamic ways at multiple time scales and organizational levels in ways that are highly context-dependent. Thats a mouthful.
Developmental evaluation is intended to be an approach that takes complexity into account, however that also means that evaluators and the program designers that they work with need to understand some basics about complexity. To that end, here are some key concepts to start that journey.
Key complexity concepts
Complexity science is a big and complicated domain within systems thinking that brings together elements of system dynamics, organizational behaviour, network science, information theory, and computational modeling (among others). Although complexity has many facets, there are some key concepts that are of particular relevance to program designers and evaluators, which will be introduced with discussion on what they mean for evaluation.
Non-linearity: The most central start point for complexity is that it is about non-linearity. That means prediction and control is often not possible, perhaps harmful, or at least not useful as ideas for understanding programs operating in complex environments. Further complicating things is that within the overall non-linear environment there exist linear components. It doesn’t mean that evaluators can’t use any traditional means of understanding programs, instead it means that they need to consider what parts of the program are amenable to linear means of intervention and understanding within the complex milieu. This means surrendering the notion of ongoing improvement and embracing development as an idea. Michael Quinn Patton has written about this distinction very well in his terrific book on developmental evaluation. Development is about adaptation to produce advantageous effects for the existing conditions, improvement is about tweaking the same model to produce the same effects across conditions that are assumed to be stable.
Feedback: Complex systems are dynamic and that dynamism is created in part from feedback. Feedback is essentially information that comes from the systems’ history and present actions that shape the immediate and longer-term future actions. An action leads to an effect which is sensed, made sense of, which leads to possible adjustments that shape future actions. For evaluators, we need to know what feedback mechanisms are in place, how they might operate, and what (if any) sensemaking rubrics, methods and processes are used with this feedback to understand what role it has in shaping decisions and actions about a program. This is important because it helps track the non-linear connections between causes and effects allowing the evaluator to understand what might emerge from particular activities.
Emergence: What comes from feedback in a complex system are new patterns of behaviour and activity. Due to the ongoing, changing intensity, quantity and quality of information generated by the system variables, the feedback may look different each time an evaluator looks at it. What comes from this differential feedback can be new patterns of behaviour that are dependent on the variability in the information and this is called emergence. Evaluation designs need to be in place that enable the evaluator to see emergent patterns form, which means setting up data systems that have the appropriate sensitivity. This means knowing the programs, the environments they are operating in, and doing advanced ‘ground-work’ preparing for the evaluation by consulting program stakeholders, the literature and doing preliminary observational research. It requires evaluators to know — or at least have some idea — of what the differences are that make a difference. That means knowing first what patterns exist, detecting what changes in those patterns, and understanding if those changes are meaningful.
Adaptation: With these new patterns and sensemaking processes in place, programs will consciously or unconsciously adapt to the changes created through the system. If a program itself is operating in an environment where complexity is part of the social, demographic, or economic environment even a stable, consistently run program will require adaptation to simply stay in the same place because the environment is moving. This means sufficiently detailed record-keeping is needed — whether through program documents, reflective practice notes, meeting minutes, observations etc.. — to monitor what current practice is, link it with the decisions made using the feedback, emergent conditions and sensemaking from the previous stages and then tracking what happens next.
Attractors: Not all of the things that emerge are useful and not all feedback is supportive of advancing a program’s goals. Attractors are patterns of activity that generate emergent behaviours and ‘attract’ resources — attention, time, funding — in a program. Developmental evaluators and their program clients seek to find attractors that are beneficial to the organization and amplify those to ensure sustained or possibly greater benefit. Negative (unhelpful) attractors do the opposite and thus knowing when those form it enables program staff to dampen their effect by adapting activities to adjust and shift these activities.
Self-organization and Co-evolution: Tied with all of this is the concepts of self-organization and co-evolution. The previous concepts all come together to create systems that self-organize around these attractors. Complex systems do not allow us to control and predict behaviour, but we can direct actions, shape the system to some degree, and anticipate possible outcomes. Co-evolution is a bit of a misnomer in that it refers to the principles that organisms (and organizations) operating in complex environments are mutually affected by each other. This mutual influence might be different for each interaction, differently effecting each organization/organism as well, but it points to the notion that we do not exist in a vacuum. For evaluators, this means paying attention to the system(s) that the organization is operating in. Whereas with normative, positivist science we aim to reduce ‘noise’ and control for variation, in complex systems we can’t do this. Network research, system mapping tools like causal loop diagrams and system dynamics models, gigamapping, or simple environmental scans can all contribute to the evaluation to enable the developmental evaluator to know what forces might be influencing the program.
Ways of thinking about complexity
One of the most notable challenges for developmental evaluators and those seeking to employ developmental evaluation is the systems thinking about complexity. It means accepting non-linearity as a key principle in viewing a program and its context. It also means that context must be accounted for in the evaluation design. Simplistic assertions about methodological approaches (“I’m a qualitative evaluator / I’m a quantitative evaluator“) will not work. Complex programs require attention to the macro level contexts and moment-by-moment activities simultaneously and at the very least demand mixed method approaches to their understanding.
Although much of the science of complexity is based on highly mathematical, quantitative science, it’s practice as a means of understanding programs is quantitative and qualitative and synthetic. It requires attention to context and the nuances that qualitative methods can reveal and the macro-level understanding that quantitative data can produce from many interactions.
It also means getting away from language about program improvements towards one of development and that might be the hardest part of the entire process. Development requires adaptation to the program, thought and rethinking about the program’s resources and processes that integrate feedback into an ongoing set of adjustments that perpetuate through the life cycle of the program. This requires a different kind of attention, methods, and commitment from both a program and its evaluators.
In the coming posts I’ll look at how this attention gets realized in designing and redesigning the program as we move into developmental design.
Comments are closed.