Designing Developmental Evaluations

A Developmental Evaluation (DE) is a means to gather information and data systematically about a program, make sense of it, and apply it to the evolution of a program in service of innovation and ongoing learning.

For that reason, they aren’t plug-and-play methods or approaches, rather they are fundamental shifts in how learning is structured within an organization. That is, if you’re going to do it well.

Too often, I see DE applied like any other approach, which I believe is a mistake. It’s not that we can’t take developmental learnings from a program or undertake a retrospective or prospective lens on how something evolves, it’s that the learning process of undertaking DE needs certain receptors to work.

Why is DE an Organizational Design Matter?

One of the first major DE projects I worked on involved a self-described learning organization doing international development work. This was a well-resourced organization that had a talented, if widely-dispersed workforce and a commitment to ensuring that they did whatever was necessary to advance their goals. They were principled, too. The programming and ethos of the organization was wrapped around a set of values that guided much of what they did.

In spite of all of this, they struggled to take advantage of the DE opportunity. This was not from lack of intent or effort, but because they weren’t designed to absorb the learning and make the adjustments to their operations based on it. They weren’t set up to do what they said they wanted to do.

One example stuck out for me. When I asked staff about how they currently integrated new information and experiences (learning) into their work, one example was their evidence review sessions. The way these sessions worked was that a member of the team would present new information to their peers for discussion and critique, and then the insights from that session would be used to foster changes. Sounds good, right? Except, here were the design characteristics of that process:

1. The meeting was to be held in-person at the organization’s office, although staff could phone in (this was before highly integrated video systems were readily available to facilitate Zoom or Teams calls).
2. A staff member needed to nominate a topic. This had to be done at least 7 days in advance to ensure it fit on the agenda. This was all voluntary and not all individuals participated in this. There were no guidelines for what was to be shared, in what detail, and what work was necessary ahead of time to prepare.
3. Feedback was encouraged, but optional. Lessons were discussed, not recorded.
4. The learning session itself was part of a larger meeting that discussed operational matters. Sometimes, pressing operational issues would require that the learning session be postponed. This might occur multiple weeks in a row during busy times.
5. The scheduling of these meetings were on Friday afternoon.

You might already imagine how this functioned in practice. Often, learning sessions were cancelled, so they became infrequent. This didn’t offer any incentive to someone who had prepared something to share, knowing that whatever was top-of-mind might slip or that there would be some need to refresh their thinking in a few weeks’ time.

Because these meetings were held on a Friday afternoon, the distractions of the coming weekend and the inevitable drift of attention came with it. Yet, Fridays were also problematic because some people had to travel, and that, because it was the last day of the week, it was also a time when some people would work from home. That lack of physical presence and the inability for people to see one another, created a barrier to communication and the kind of engagement that learning requires.

It was more information delivery, not learning and sensemaking.

Because things weren’t recorded in a systematic way, there was little means to reflect deeply on content, trusting that people would just remember things. But this was on a Friday afternoon, when people are possibly fatigued from the workweek while looking forward to the weekend, and also that there would be at least two workdays before staff would even have an opportunity to enage with this material.

If you’re like many, you know that Mondays often start with catching up on weekend-sent correspondence and preparing for the week, not revisiting content from Friday afternoon. This is partly why learning is such a challenge in organizations.

Designing Better Learning Systems

As I noted in an earlier article, the systems we create are what shape learning and introduce barriers and facilitators, and these function by design.

If you’re seeking to promote learning and create a culture where individuals across an organization can adapt, develop, and grow learning requires much more than simply sending people to conferences, hosting seminars, providing books and other materials or watching some instructional videos. Without a means to integrate and promote that new knowledge as part of a praxis, organizations and individuals alike will continue to get frustrated, lag in their efforts to anticipate and respond to changing conditions and will ultimately fail to achieve anything close to their potential.

Designing for learning is as much about a curriculum as the context for how that curriculum is delivered and how learners are set up to engage with it all in their organizations and everyday lives.

At the heart of this is asking questions about what does a learning organization do? How might evaluative information like that which comes from a DE, contribute to that learning? And what needs to be put in place for that to happen?

Without a container to put knowledge in, learning can’t take place. And it is that learning — and the adaptations that are to come through that — which is what defines whether a DE can actually do its job.

Developmental Evaluation by Design: Features

I’ve written on DE preparedness elsewhere, which include shaping a mindset and skillset. As I noted:

The most important aspect of a Developmental Evaluation is creating the right mindset. This mindset is focused on complexity, adaptation, and development (vs. improvement, linear growth).

These mindset features include:

1. Openness to experience. A DE will likely take a journey that can’t be fully predicted, much like a road trip. An openness to a non-linear pathway toward a preferred destination and the willingness to adapt the path and the destination based on evaluation data and evidence is key.
2. Tolerance for ambiguity. Few of us enjoy uncertainty, but we all need to deal with it when working on a DE.
3. Self-awareness. We often get in our own way. Understanding how we think and the biases and perspectives we hold is important to knowing when they serve us and when they are a barrier to innovation. Mindfulness is a part of this quality.
4. Patience. This is a skill and mindset quality. Knowing that things will unfold at a time and pace that might change is useful — and requires patience in just knowing that.
5. Evaluative thinking. This is a form of thinking that gets us connecting our activities, outcomes, and effects and asks three key questions about how we know something (and what to do about it).

A skillset is also needed.

While many of the qualities inherent in a DE are natural to people, they are not commonly practiced in organizations. DE is an approach, and while a skilled evaluator might have many methodological skills to do much of what is in a DE, without the following skills — with the evaluator, the evaluation team, the organization, or all of them — you won’t likely succeed.

These qualities include:

• Facilitation. A DE is a collaborative endeavour. You need facilitation skills to engage the diversity of perspectives in the system and coordinate the discussion, feedback, and opportunities for everyone to engage with these perspectives.
• Sensemaking. Sensemaking is a participatory process that involves taking data that may be difficult to understand, ambiguous in its conclusions, or incomplete and helps people make sense of what it means for the organization within the present context and strategic needs. Sensemaking for DE helps guide decision-making and strategy.
• Complexity and Systems Thinking. A distinguishing feature of DE is the use of applied thinking from systems science and complexity to guide the entire process. This means creating designs and processes that are sensitive to detecting emergent properties, network effects, organizational behaviour (like self-organization), and non-linear or algorithmic change dynamics.
• Dynamism. Complexity also requires that an evaluation be designed with consideration of the dynamics of the program, actors, and system. This means adapting or developing from a moving position, not a static position. It also involves designing evaluations that take into account the growth pattern and evolution of a program that suits the system. It’s about a design for changing conditions.
• Visualization. System complexity is made more so by the inability to effectively ‘see’ the systems in our heads – there’s too much information. The ability to create system maps, visualize the dynamics and relationships within it, and facilitate discussion of those systems is critical to understanding where a program fits within it all. Visualization can be sophisticated or relatively simple in nature.
• Design. What makes DE distinct is that it is about making modifications to the program as it unfolds using evaluation data to guide the strategy. These modifications require design and the skills — like with evaluation — require designing a program while it is moving. These developmental design skills are often overlooked and bring together service design and innovation with elements of systemic design.

Developmental Evaluation by Design: Practice

If you have the mindset and skillset, the practice of DE is building regular conversations between your evaluation team (whether internal or contracted), and engage in ongoing sensemaking about the findings and the design implications of the findings.

Time is required to consider the evidence and to make the requistie design decisions. If you don’t provide your teams with the time — and at a time that is less distracting or prone to disruption like the case I mentioned earlier — you will not be able to learn.

If there is no capacity to integrate those findings into a design decision or strategic activity, DE will also not produce much of value. These things need to be knitted together within the fabric of the organization’s activities to ensure the lessons learned are applicable, the design decisions from those lessons are relevant, and that the entire experience is aligned with the goals, aims, values, and capacities of the organization.

This requires some thought ahead of time. A good consultant can help you with this.

So when you’re setting up a DE, consider what you have and how you can set your organization up to best use it. The rewards are many.

For further reading:

What Does Developmental Evaluation Look Like?

A mindset for developmental evaluation

Developmental Evaluation and Design

Developmental Evaluation Preparedness