Regular readers will know that I’ve been thinking a lot about evaluation for many years now. I am not an evaluator, but almost every project I am involved in contains some element of evaluation. Sometimes this evaluation is well done, well thought through and effective and other times (the worst of times, more often than you think) the well thought through evaluation plan crumbles in the face of the HIPPO – the Highest Paid Person’s Opinion. So how do we really know what is going on?
When I stumbled across Michael Quinn Patton’s work in Developmental Evaluation, a whole bunch of new doors opened up to me. I was able to see the crude boundaries of traditional evaluation methods very clearly and was able to see that most of the work I do in the world – facilitating strategic conversations – was actually a core practice of developmental evaluation. Crudely put, traditional “merit and worth” evaluation methods work well when you have a knowable and ordered system where the actual execution can be evaluated against a set of ideal causes that lead to an ideal state. Did we build the bridge? Does it work according to the specifications of the project? Was it a good use of money? All of that can be evaluated summatively.
In the unordered systems where complexity and emergence is at play, summative evaluation cannot work at all. The problem with complex systems is that you cannot know what set of actions will lead to the result you need to get to, so evaluating efforts against an ideal state is impossible. Well, it’s POSSIBLE, but what happens is that the evaluator brings her judgements to the situation. Complex problems (or more precisely, emergent problems generated from complex systems) cannot be solved, per se. While it is possible to build a bridge, it is not possible to create a violence free society. Violent societies are emergent.
So that’s the back story. Last December I went to London to do a deep dive into how the Cynefin framework and Cognitive Edge’s work in general can inform a more sophisticated practice of developmental evaluation. After a few months of thinking about it and being in conversation with several Cognitive Edge practitioners including Ray MacNeil in Nova Scotia, I think that my problem is that that term “evaluation” can’t actually make the jump to understanding action in complex systems. Ray and I agreed that Quinn Patton’s work on Developmental Evaluation is a great departure point to inviting people to leave behind what they usually think of as evaluation and to enter into the capacities that are needed in complexity. These capacities include addressing problems obliquely rather than head on, making small safe to fail experiments, undertaking action to better understand the system rather than to effect a change, practicing true adaptive leadership which means practicing anticipatory awareness and not predictive planning, working with patterns and sense-making as you go rather than rules and accountabilities, and so on.
Last night a little twitter exchange between myself, Viv McWaters and Dave Snowden based on Dave’s recent post compelled me to explore this a bit further. What grabbed me was especially this line: “The minute we evaluate, assess, judge, interpret or whatever we start to reduce what we scan. The more we can hold open a description the more we scan, the more possibility of seeing novel solutions or interesting features.”
What is needed in this practice is monitoring. You need to monitor the system in all kinds of different ways and monitor yourself, because in a complex system you are part of it. Monitoring is a fine art, and requires us to pay attention to story, patterns, finely grained events and simple numbers that are used to measure things rather than to be targets. Monitoring temperatures helps us to understand climate change, but we don’t use temperatures as targets. Nor should we equate large scale climate change with fine grained indicators like temperature.
Action in complex systems is a never ending art of responding to the changing context. This requires us to be adopting more sophisticated monitoring tools and using individual and distributed cognition to make enough sense of things to move, all the while watching what happens when you do move. It is possible to understand retrospectively what you have done, and that is fine as long as you don’t confuse what you learn by doing that with the urge to turn it into a strategic plan going forward.
What role can “evaluation” have when your learning about the past cannot be applied to the future?
For technical problems in ordered systems, evaluation is of course important and correct. Expert judgement is required to build safe bridges, to fix broken water mains, to do the books, audit banks and get food to those who need it. But in complex systems – economies, families, communities and democracies, I’m beginning to think that we need to stop using the word evaluation and really start adopting new language like monitoring and sense-making.
Two weeks ago in our Leadership 2020 program I experimented with using a signification framework to harvest a World Cafe. We are beginning another cohort this week and so I had a chance to further refine the process and gather much more information.
We began the evening the same way, with a World Cafe aimed at exploring the shared context for the work that these folks are in. Our cohort is made up of about 2/3rds staff from community social services agencies and 1/3 staff from the Ministry of Children and Family Development. This time I used prepared post it notes for the sense making exercise, which you can see here:
Our process went like this:
- At Cafe tables of five for 20 minutes, discuss the question “What is a story of the future you are anticipating for this sector?”
- Second round, new tables, same question, 20 minutes
- About ten minutes of hearing some random insights from the group, and checking to see how those resonate.
- 2 minutes of silent reflection on the question of ‘What do you need to learn here that will help us all move forward?”
- Each participants took a pink and blue post it note. On the blue post it they wrote what they needed to learn that would be immediately applicable and on the red ones, learning that is needed to prepare for the future.
- Participants filled out the post-its and then were instructed on how to signify the data on a triangle framework that helped them signify whether what they needed to learn would help them “in their personal life,” “do their jobs” and/or “make change.”
- Participants also indicated on the post-its whether the worked for the Ministry or worked for a community organization.
At the conclusion of the exercise we had a tremendous amount of information to draw from. Our immediate use was to take a small group and use affinity grouping to identify the themes that the whole has around their learning and curiosity. We have used these themes to structure a collective story harvest exercise this morning.
But there is some much more richness that can come from this model. Here are some of the ways people are playing with the date:
- Removing all the pink post-its to see what the immediate learning needs are and vice versa.
- Looking at and comparing the learning needs between the two sectors to see where the overlaps and differences are
- examining the clusters at the extremes to see what ot tells us about personal needs, and professional needs.
- Uncovering a theory of change by looking at the post its clustered around the “Making change” point and also seeing if these theories of change are different between the community and the government.
And of course because the data has been signified on each post it, we can recreate the framework easily. The next level for me will be using this data to create a Cynefin framework using the four-points contextualization exercise. Probably won’t happen in this cohort.
Big learning is the rich amount of data that proceeds from collecting finely-grained objects, allowing for disintermediated sense-making, and seeing all these multiple ways in which signified data can be used to address complex challenges obliquely, which allows you to get out of the pattern entrainment that blinds you to the weak signals and emergent patterns that are needed to develop emergent practice. This pen and paper version is powerful on its own. You can imagine how working with SenseMaker across multiple signification frameworks can produce patterns and results that are many magnitudes richer.
Just off a call with a potential client today and we were scoping out some of the work that we might do together, with a small organization facing unprecedented change. They are in a place of finally realizing that they are not in control of what is happening to them. They are completely typical in this respect.
I am constantly struck by the fact that we have so few skills, frameworks and so little language for dealing with complexity. Clients all the time approach me looking for certainty, answers and clear outcomes. It’s as if they are searching for the one person who will promise them the relief they are looking for. And no one can. Because mostly what they are FEELING is their emotional reponse to the reality of a complex world. And no amount of rational and linear planning will address that feeling. in fact quite the opposite. Sitting down and deciding on a vision, goals, objectives and plan just defers the pain, because it fools you into thinking you are in control but it sets up a false ideal against which your progress will always be measured to be short.
Confronting complexity is hard. It is not merely that we need better tools to think about it. We need better tools to emotionally deal with it. it is overwhelming, infuriating, confusing, and frightening. And almost every organization I work with that fails to address it well fails because they don’t attend to the fear. They build fears into their processes, or they build processes to avoid confronting what they are afraid of: usually that we don’t know what’s going and we don’t know what to do.
My potential client asked me if I could say what outcomes would come from working with me. In brief they are this:
- We will build the capacity to understand and work with the problems you are facing in context by confronting and changing the view we take around complexity
- We will work strategically with the content of the project, and build participatory processes together that will change the way we do the work of addressing complex problems
- We will build resilient containers for the work that will allow us to confront our fears and limiting beliefs about the work and the change we are in, and that will provide a solid strategic framework for our project.
- We will arrive at a set of strategic decisions about the present moment and be prepared to make strategic decisions about the future.
That’s it. Sometimes those outcomes are incredibly concrete, sometimes it is more about building capacity, but it is always about acting strategically, and that sometimes means learning a new language and a new set of skills. I find that it’s the learning part with which people are most impatient. They seems to want to be able to accelerate the outcomes they want without having to change their approach. But, if you found yourself teleported to rural Bangladesh and you now had to make a living as a rice farmer, do you think your current language and skill set would be applicable, if only you applied yourself harder?
There are projects that fit the ordered domain of work, in which project management and strategic planning leads to predictable outcomes. And there is work for which “learning” is both the outcome and the new organizational structure and leadership practice. It is very important not to confuse the two contexts. And it is surprising just how much we are willing to turn a blind eye to complexity (as both a friend and a foe) in favour of a stable and knowable future, no matter how impossible that idea is.
One has to be very careful attributing causes to things, or even attributing causality to things. in complex systems, causality is a trap. We can be, as Dave Snowden says “retrospectively coherent” but you can not know which causes will produce which effects going forward. That is the essence of emergent phenomena in the complex world.
But even in complicated problems, where causality should be straightforward, our thinking and view can confuse the situation. Consider this example.
Imagine someone, a man, who has never seen a cat. I know, highly implausible, but this is a hypothetical from Alan Watts’ book, On the Taboo Against Knowing Who You Are, which was written in the sixties; pre-YouTube. Watts uses this fictional fella to illustrate the unfelt influence of perspective and the dangers inherent in our strong inclination to seek cause-and-effect relationships.
“He is looking through a narrow slit in a fence, and, on the other side a cat walks by. [The man] sees first the head, then the less distinctly shaped furry trunk, and then the tail. Extraordinary! The cat turns round and walks back, and again he sees the head and a little later the tail. The sequence begins to look like something regular and reliable. Yet again the cat turns round and he witnesses the same regular sequence: first the head and later the tail. Thereupon he reasons that the event head is the invariable and necessary cause of the event tail which is the head’s effect. This absurd and confusing gobbledygook comes from his failure to see the head and tail go together; they are all one cat.”
We often create and embed the wrong patterns because we are looking through a slit. As Watts says, by paying very close attention to something, we are ignoring everything else. We try and infer simple cause-and-effect relationships much, much more often than is likely in a complex world. For example, making everyone in an organisation focus on hitting a few key performance indicators isn’t gong to mean that the organisation is going to get better at anything other than hitting those key performance indicators. All too often this will lead to damaging unintended consequences; absurd and confusing gobbledygook.
via abc ltd.
Our traditional Boxing Day movie this year was The Imitation Game, the new film about Alan Turing and his team’s efforts to break the Enigma code used by the German Navy in the Second World War. While the film itself was good it was full of fictional scenes that were intended to point at some of the interesting things that happened at Bletchley Park during that war. Having done a bit of reading on the subject, it’s clear that the film simplified many things, took liberties with others and glossed over what is a really interesting story, but the movie itself still holds up, even if Cumberbatch basically turns Turing into his Sherlock Holmes.
At any rate there were a few things in the fils that provided interesting reflections on some of the ideas I have working with a learning about both through my study of Cynefin dynamics (the way problems and solutions move through the Cynefin domains) and with the two loop theory of change which I am using a lot. So here are a few examples.
Solving problems obliquely. Complex problems can’t be solved by taking a head-on, brute force approach to the solution. The film is basically about this writ large, but one vignette stands out as interesting. When Turing needs new staff he devises a way to find them by running a crossword contest in a newspaper. Anyone who solves the problem in under ten minutes gets contacted by MI6 and invited to come and write a test. Although this is not how Joan Clarke joined the project, it was a good way of sorting out the talent from the confirmation biases that riddled the intelligence establishment (in this case gender bias).
Disintermediated sensemaking. The idea of letting everyone have the data and find patterns there is an important aspect of working with complexity. While the problems that the team were solving were indeed complicated, they needed to exploit complex human behaviours in order to have a chance to solve them. A complex problem is solvable with enough expertise, and indeed making a code HAS to be solvable if it is to work. If you don’t want others to solve it you simply make the encryption keys so elaborate that there isn’t enough time in the history of the universe to solve the problem. So while in theory, code breaking is a merely technical problem, in order to solve it, you have to narrow down the permutations to make it possible for the technical solutions to be applied. At Bletchley Park, this came down to reading human factors, which is something only the human operators could do. But they could only do that by having access to the raw data and by creating safe-to-fail probes of the system (by using these factors to solve the codes). When they worked, they were exploited.
There are some incredible stories about the way which the women who were intercepting messages came to know their counterparts in Germany. Each German communications officer had his own style, his own signature. And human error in creating predictable procedures meant that people could use these patterns as weak signal detection in order to break some messages, in the case of the Polish codebreakers that did much of the early work on cracking Enigma, even discern the wiring of the machines themselves. This is a classic pattern of what Dave Snowden calls Cynefin dynamics, specifically how we move from safe-to-fail probes in the complex domain to exploiting findings using complicated and in some cases obvious solutions.
This is a really interesting story, and I’ve ordered a couple of books to read in further. I’m very interested to see how the human factors were sensed, discerned, exploited. Combining that capacity with the incredible engineering talents of Turing and his crew provides some excellent stories and examples of Cynefin dynamics at work.