Evaluation is such an influential constraint in organizational and community life. When resources and attention are tied to evaluation results, a kind of tautology gets set up. One begins managing projects towards the evaluation outcomes, in order to give the best chance of an initiative surviving and continuing to attract resources. One of the things I appreciate about developmental evaluation is its deliberate engagement with emergence. Making sense of emergence however can be a really time consuming affair, and so I’m thinking about how we can use good use of time to use dialogue and collective meaning making to help make sense of data and direction.
Developmental evaluation is for the complex domain. That means that we are not working with evaluating actions against desired end states, but instead noticing and paying attention to vectors and directions – intentions and hypotheses that help shape emerging strategy. Developmental evaluation is the process of gathering information about our work to give us some intelligence about what we are doing.
Think of the information needs of two different kinds of athletes. A golf player relies on solid objective data (how many yards to the hole, where the wind is coming from, the nature of the lie of the ball and so on) and interprets that data through her own self-knowledge (I hit a five iron 160 yards. Adjusting for wind and lie and the target topography, I should hit a 4 iron with backspin…) Of course the better a golfer one is, the easier it is to execute a plan and understand exactly where one succeeded or failed.
By contrast soccer players work in a dynamic environment. The information available to them only becomes apparent as they begin to play the match. They may know something about the other team, but they learn rapidly in the first ten minutes or so how the game is going to go. A team will discover where the opposition’s weakness is, or what its attacking strategy is, or where the open spots are on the pitch. Making good use of this information requires excellent communication in real time to share what is being learned. It requires players to play with potentials and patterns rather than certainties. Every move provides yet more information. The better a team works together, the faster they can adjust their strategy to take advantage of potentials.
When we are evaluating work there is a mix of these two types of approaches at play. Summative evaluation will look at the gap between expected outcomes and what actually happened and suggest how to adjust for next time. Budget planning and auditing is a good example of this technical kind of results based evaluation. Count the money and compare against projections. Look for causes. Some of these causes will be technical and some will be down to culture.
Developmental evaluation requires a different strategic approach, and simply put, it might fall into these four things (I’m trying for simplicity here, to try to be able to describe this in an easy way):
- Data points that give us the ability to capture information about a current state of an evolving system. This can render a series of pictures that will allow us to see patterns and trends. You need multiple snapshots over time to make sense of what is happening. One photo of a soccer game in progress tells you nothing. You need to monitor indicators not manage end points. Soccer is much more than just putting the ball in the net, even though that is the desired end result.
- Feedback loops from data to human sensemaking so that data can be used in real time to develop strategy and adjustments to the directionality of work.
- A facilitated sensemaking process to bring together multiple perspectives to interpret what is happening. In a complex system the data won’t give you answers. It will provide information to form hypotheses about the patterns that are emerging, and that information can give you guidance for action.
- A way of acting that doesn’t over commit resources to emerging potential strategies, but which gives enough momentum to see if we can shift things in a desired way. Snowden calls this “safe-to-fail.” This is tricky and calls for good context dependant leadership, but it is the essence of good decision making.
There are all kinds of ways of implementing these strategies. You can use surveys to discover what people are accessing on your website and you can use interviews or sensemaking tools to find out HOW they are using that information. You can use a strategic group to interpret these results and see how they are either coherent with our intentions, or at odds with them. You can then create new initiatives that support what is emerging or figure out ways to abandon what is not working. There are thousands of dialogue methods and processes to use to ask questions about and develop action around the data that is emerging.
Importantly, developmental evaluation needs to be a part of the way you work strategically. It needs a rhythm and a cadence to it, so that you know you are coming back on a regular basis to the emerging picture of what is happening. You need outsiders occasionally to come in and disrupt your point of view and offer alternative views of the patterns, and you need to choose a longer rhythm to continue to develop and refine your evaluation strategy as a whole.
I want this to be simple as a process to use. Strategy without information is just a wild guess. But if we tie our decisions too closely to the data emerging from dynamic systems we can get equally stuck making decisions that try to game the system towards desired results, with sometimes disastrous results for clients, customers and ultimately, organizational integrity. It’s a balance and a practice. How can we make this easy?
I’m in a period of recovering from travel and work, over what has been a very busy spring. This weekend I just took right off and did some reading, cleaning and planning for a major kitchen renovation we will be doing this spring.
Reading-wise, it has been a luxury to sit on my front porch and spend hours in a book. My choice this week has been Kim Stanley’ Robinson’s “Aurora” which is a story about a human voyage to colonize a planet 11 light years away. It is an amazing book about problems solving and ontology and should be on every reading list for those who are trying to understand the kinds of philosophy, thinking skills and patterns that make it possible to live with complexity. It’s also a lovely meditation on the difference between technical and adaptive problem solving and leadership. Yes, this is a relaxing piece of fiction for me! I’m lucky to enjoy my work!
On other notes, several interesting links and articles have come my way through different sources this week. Here are a few of note:
So, you don’t think you directly benefit from nonprofits? / Nonprofit With Balls . On why you actually do.
Some Corals Survive Environmental Assault: Scientists Want to Know Why – Plexus Institute. An interesting summary of some of the ways that corals are beginning to demonstrate resilience in the face of massive environmental changes to their habitat. If you’v read Aurora, you’ll appreciate why this article in particular interests me.
Creative Leadership Workshop | Johnnie Moore . A pitch for a cool looking course from my friends Johnny Moore and Viv McWaters in Cambridge this summer.
A Modern-Day Viking Voyage | Hakai Magazine . A few years ago I was staying in Montreal with a Manx friend and learned about this form of governance. My maternal great grandmother’s family is Manx so I’ve always had a passing interest in the little country in the Irish Sea. But the viking connection and the form of council used to govern the country is fascinating.
Complexity Labs . A very interesting new site on complexity, featuring a lot of learning resources.
Saving the planet from governments and markets | Henry Mintzberg. This is the quote that you never expect to see from a business school professor, unless it’s Henry Minstzberg: ”
“It is not plans from some elite “top” that will begin the world over again, but actions on the ground. We are the feet that will have to walk all the talk, connected to heads that will have to think for ourselves. We shall have to confront the perpetrators of climate change—and that includes ourselves—not with violent resistance or passive resistance, but with clever resistance. Some years ago, the angry customers of a Texas telephone company paid 1 extra cent on their telephone bills. This tied the company in knots. It got the message.
Beyond resistance will have to come the replacement of destructive practices by more constructive ones, as has been happening with wind and solar energy. There will be more of this when we “human resources” pursue our resourcefulness as human beings. Imagine, for example, an economy based on growth in qualities instead of quantities, of better instead of more—in education, health care, and nutrition.”
The Secret History of Bioluminescence | Hakai Magazine : Hakai Magazine is one of my favourites, because it’s funded locally but covers global ocean issues. And because I live on an island in the global ocean, that matters. This article is a beautiful meditation on the natural and social history of bioluminecense, one of the many incredibly beautiful things that happens in the ocean here.
It’s good to have Dave Snowden back from his treks in the Himalayas. He’s been a big influence on my thinking and practice over the past few years and his near daily blog posts are always rich, irreverent and practical. He is in the process of creating an important body of theory and practice that is useful even if the language and the concepts are sometimes a lot of work to grasp. The payoff from wrestling with his ideas is rich.
Today he’s discussing “dispositionality” which simply means that making change in a system is much easier when you have a sense of what the system is pre-disposed to do (and what it is NOT pre-disposed to do…)
Back in the summer Caitlin and I led a learning lab for the board and staff members of various community foundations from around British Columbia. The five principles that Dave articulated today were very much embedded in our work and they are becoming very much the basis for any change and planning work I do. Here’s how we made it work, pen and paper style.
1. Map the current state of the system, including its dominant flows, eddy points and whirlpools.
We began with a World Cafe design based on small stories of change. It is always good to ask people about actual decisions or stories that they remember to ground their experience in discovery. If you run a cafe on “What are the big sources of change in our sector?” you get a data set that is divorced from reality, meaning that it is subject to being gamed by the participants. I can just insert the things I want to see in there. But if I am asked to tell a story about a particular decision I had to made, the data set is richer and we have a good chance to see emerging patterns.
And so our Cafe ran like that: “Tell a story of a time when you knew things needed to change?”
Each person told a story and the other three at the table listened and wrote down what they heard was the impetus for change, with one data point on a post-it note. We did several rounds of story telling. At the end of the round, we asked people to give the post-its to the story teller, and we gave the story tellers time to rank each post it note on a scale of 1-3. A one meant that the impetus for change was just known to me (a weak signal), two meant that a few other people know about this impetus, and a three meant that this change trigger was known by everybody.
We then had the group cluster all the post-its to find major categories, and we sorted post it notes within the categories to produce a map that was rendered by our graphic recorder, Corrina Keeling. You can see that above.
2. Identify the energy gradient associated with existing dominant patterns and what adjacent possible states to any undesirable pattern present themselves.
The resulting map shows the major areas for change making, specific “acupuncture points” and the “energy gradients associated with the dominant patterns.” Practically what this means is that items marked in yellow were very weak signals and could be candidates for a change initiative that would appear out of left field for the dominant system. Not a bad thing to do, but it requires a lot of resources and political capital to initiate. The red items were things that EVERYBODY was talking about, which meant that the space for innovation was quite closed down. There are a lot of experts, large consulting firms, influential funding pots and politically committed people tackling change at this level because it is perceived to be an influential place to play. As a result it is generally a zone that is not failure tolerant and so these items are not good candidates for a probe or prototyping approach.
But the orange items were in a kind of Goldilocks zone: there are a few people who know that you can make change here, so you have allies, but the field is not cluttered with competing experts trying to assert their ideological solutions.
The whole map allows you to make choices.
3. Engage in safe-to-fail experiments in parallel either to change the energy gradient or to nudge (or shift) a dominant pattern to a more desirable state ideally through action rather than platitude.
This is of course the best approach to making change within complex systems. We took time to develop prototypes that were intended to tell us something about the system. A bonus would be that we might might create ideas that would turn into interesting new initiatives, but the primary function of running prototypes is to probe the system to tell us something about what is possible. Making tentative conclusions from action inspires people to try more, on a path that is a little more blazed. Just creating platitudes such as “Let’s build networks for knowledge transfer” doesn’t do enough to help change makers poke around and try things that are likely to work.
Each participant in the group created one or two prototypes which they rolled out, seeking to make a bit of change and learn about what helped or hindered change making in a relatively conservative sector of civil society.
4. Monitor the impact in real time and take multiple small actions to reinforce the good and disrupt the bad.
We kept the group together over a few months, having them check in over webinars to share the progress on their prototypes. We deliberately created a space where things were allowed to fail or radically change and we harvested learning all the way along. Where things were working, prototypes evolved in that direction, and we had a little funding to help accelerate them. By simply starting, participants discovered oblique strategies and in some cases entirely new ways to address their basic desire for changing some element of their environment. Without engaging in a deliberate yet loosely held action-based project, it is very difficult to see the opportunities that lie in the blind spots.
This learning was summarized in a report, but the bigger harvest was the capacity that each participant built to take steps to sense, design and implement change initiatives with a better informed complexity approach.
5. At all costs avoid any announcement of a change initiative or idealistic outcome based targets
I think this goes without saying. Change making in the complex space is essentially learning on overdrive. When we are truly stuck and yet we have a sense that “this might just work” we need good support to explore that instinct. Being deliberate about it helps. But announcing that “this is what we are doing and here are the targets we have to meet” will collapse people’s inherent creativity down to narrowing the focus of their work on achieve the pre-determined outcomes. That is a perfect strategy for destroying the capacity to engage with complexity, and it can result in a myopic approach to change that guarantees “black swan events” and other nasty surprises.
When I was up in Whitehorse last week I got to spend time with folks from the Public Service Commission discussing a project that would see us looking at discriminations in the workplace from a complexity angle. Using Cynefin and SenseMaker(tm), we hope to understand the ways in which the landscape of discrimination shifts and changes over time so that the PSC can make wiser decisions about the kinds of initiatives it sculpts. One of the problems with diversity initiatives in the public service (in any large public organization really) is the feeling that they need to be broad based and rolled out to everyone. This usually results in a single initiative that spreads across the whole organization, but except for a little awareness raising, does little to address specific instances of discrimination. Everything from awareness raising “cultural competency training” to zero tolerance accountability measures have limited effect because a) discriminatory behaviour is highly context and situation dependant and b) the public service has a permeable boundary to the outside world, meaning ideas, behaviours and people move between the two contexts all the time. The larger your organization, the more like the real world you have to be.
At any rate, I took a bit of time to do a mini-Cynefin teaching to explain how strategy works in the complex domain. and my friend Pawa Haiyupis and I added two Ps to my concentric circles of intervention in a complex system. So to review:
- Patterns: Study the patterns in a complex setting using narrative capture and sense-making. This can be done with the SenseMaker(tm) software, and it can also be done with dialogic interventions. The key thing is to let the people themselves tag their stories or at the very least have a group of people reviewing data and finding patterns together. For example, you might notice a correlation between stressful times in an organization and an increase in feelings of discriminatory behaviour
- Probe: Once you have identified some patterns, you can make some hypotheses about what might work and it’s time to develop some safe to fail probes. These aren’t meant to be successful: they are meant to tell you whether or not the patterns you are sensing have developmental potential. Failure is entirely welcome. What if we offered stress reduction activities during high stress times to help release pent up feelings? We want to be okay with te possibility that that might not work.
- Prototype: If a probe shows some promise, you might develop a prototype to develop a concept. Prototypes are designed to have tolerance for failure, in that failure helps you to iterate and improve the concept. The goal is to develop something that is working.
- Pilot: A pilot project is usually a limited time proof of concept. Roll it out over a year and see what you learn. In Pilot projects you can begin to use some summative evaluation methods to see what has changed over time. Because of their intensive resource commitment, pilot projects are hardly ever allowed to fail, making them very poor ways of learning and innovating, but very good ways to see how stable we need to make an approach.
- Project/Program/Policy: Whatever the highest level and most stable form of an initiative is, you will get to there if your pilot shows promise, and the results are clear. Work at this level will last over time, but needs regular monitoring so that an organization knows when it’s time to tinker and when it’s time to change it.
Cynefin practitioners will recognize that what I’m writing about here is the flow between the complicated and the complex domains, (captured by Dave Snowden’s Blue dynamic in this post.) My intention is to give this some language and context in service organizations, where design thinking has replaced the (in some ways more useful) intuitive planning and innovation used in non-profits and the public service.
Since October, when I first starting sketching out these ideas, I’ve learned a few things which might be helpful as you move through these circles.
- Dialogue is helpful at every scale. When you are working in a complex system, dialogue ensures that you are getting dissent, contrary views and outlying ideas into the process. Complex problems cannot be addressed well with a top-down roll out of a change initiative or highly controlled implementations of a single person’s brilliant idea. If at any point people are working on any stage of this alone, you are in danger territory and you need another pair of eyes on it at the very least.
- Evaluation is your friend and your enemy. At every stage you need to be making meaning and evaluating what is going on, but it is critically important to use the right evaluation tools. Developmental evaluation tools – with their emphasis on collective sense making, rapid feedback loops and visible organizational and personal learning – are critical in any complexity project, and they are essential in the first three stages of this process. As you move to more and more stable projects, you can use more traditional summative evaluation methods, but you must always be careful not to manage to towards targets. Such an error results in data like “We had a 62% participation rate in our diversity training” which tells you nothing about how you changed things, but can shift the project focus to trying to acheive a 75% participation rate next cycle. This is an especially pervasive metric in engagement processes. And so you must…
- Monitor, monitor, monitor. Intervening in a complex system always means acting without the certainty that what you are doing is helpful. You need data and you need it on a short term and regular basis. This can be accomplished by formal and informal ongoing conversations and story captures about what is happening in the system (are we hearing more stories like the ones we want?) or through a SenseMaker(tm) monitoring project that allows employees to end their data with a little data capture.
- These practices are nested, not linear. An always to remember that this is not a five step process to intervening in a complex system. In a large organization, you can expect all of these things to be going on all the time. Building the capacity for that is a kind of holy grail and would constitute a 21st century version of the Learning Organization in my books.
In the complex space, Paul Hobcraft shares some very good guiding principles, but the whole post shimmers with good advice about transformation, and is applicable to movement building, network organizing and enterprise.
Today corporate transformations must be designed and executed quickly and routinely—not as once-a-decade events. Management teams are looking for best practices that increase speed and reduce the risk of pursuing business model innovation and change. That’s where minimum viable transformation comes into play. Before diving in, management teams should consider these five principles:
1. Learn how to learn. The central goal of minimum viable transformation is to learn from a true field experiment.
2. Pick up speed. There’s a reason this approach starts with the word “minimum”: The learning has to happen fast. As soon as a company executes the idea it’s pursuing, it shows its hand to competitors— who will quickly respond with their own strategies.
3. Embrace constraints. Much has been written about the counter-intuitive effect of constraints—they don’t foil creativity, but fuel it. It’s worth noting that the very constraints we’ve been talking about here—few bells and whistles and scarce time—take real creativity to address. At the very least, they compel a focus on the goal—the need to learn and reduce risk around the key objective.
4. Have a hypothesis. To succeed, transformation initiatives must clearly articulate both the need for change and its direction. Such a statement of direction helps identify key assumptions driving the change effort (assumptions that will need to be tested and refined along the way). Leaders will also need to develop metrics that measure short-term progress.
5. Start at the edge. Find an “edge” of the current business—a promising arena that can showcase the potential of a fundamentally different, highly scalable business model that could even become a new core. Starting at the edge gives the transformation team far more freedom to test and experiment, and more ability to learn and react quickly.
In short, these five key principles can help bypass traditional barriers to transformation, ultimately supporting more effective response to mounting performance pressures.
via Deeper read or quick summary- finding the time | Paul4innovating’s Innovation Views.