Recently in BC, we have a had a child die in the care of the state. This does happen from time to time, and when it does a process is triggered whereby the Representative for Children and Youth lanuches an investigation and makes recommendations which usually result in more rules and procedures to govern the child welfare system with the express purpose of never having it happen again.
I work closely with child protection social workers in BC and there is not a single one I know of whose heart does not break when something like this happens. Everyone wears the failure. Social work is difficult not because of the kinds of predictable situations that can be mitigated but because of the ones no one saw coming. The Ministry of Children and Family Development operates under a massive set of procedures and standards about social work practice. But no amount of rules will prevent every case of child death. Just like no amount of rules will eliminate every case of discrimination, every war, every instance of every bad thing that happens to humans.
A couple of good blog posts in my feed this morning that provoked some thinking. These quotes reminded me how much evaluation and planning is directed towards goals, targets and patterns that cause us to look for data that supports what we want to see rather than learning what the data is telling us about what’s really going on. These helped me to reflect on a conversation I had with a client yesterday, where we designed a process for dealing with this.
Regular readers will know that I’ve been thinking a lot about evaluation for many years now. I am not an evaluator, but almost every project I am involved in contains some element of evaluation. Sometimes this evaluation is well done, well thought through and effective and other times (the worst of times, more often than you think) the well thought through evaluation plan crumbles in the face of the HIPPO – the Highest Paid Person’s Opinion. So how do we really know what is going on?
When I stumbled across Michael Quinn Patton’s work in Developmental Evaluation, a whole bunch of new doors opened up to me. I was able to see the crude boundaries of traditional evaluation methods very clearly and was able to see that most of the work I do in the world – facilitating strategic conversations – was actually a core practice of developmental evaluation. Crudely put, traditional “merit and worth” evaluation methods work well when you have a knowable and ordered system where the actual execution can be evaluated against a set of ideal causes that lead to an ideal state. Did we build the bridge? Does it work according to the specifications of the project? Was it a good use of money? All of that can be evaluated summatively.
In the unordered systems where complexity and emergence is at play, summative evaluation cannot work at all. The problem with complex systems is that you cannot know what set of actions will lead to the result you need to get to, so evaluating efforts against an ideal state is impossible. Well, it’s POSSIBLE, but what happens is that the evaluator brings her judgements to the situation. Complex problems (or more precisely, emergent problems generated from complex systems) cannot be solved, per se. While it is possible to build a bridge, it is not possible to create a violence free society. Violent societies are emergent.
So that’s the back story. Last December I went to London to do a deep dive into how the Cynefin framework and Cognitive Edge’s work in general can inform a more sophisticated practice of developmental evaluation. After a few months of thinking about it and being in conversation with several Cognitive Edge practitioners including Ray MacNeil in Nova Scotia, I think that my problem is that that term “evaluation” can’t actually make the jump to understanding action in complex systems. Ray and I agreed that Quinn Patton’s work on Developmental Evaluation is a great departure point to inviting people to leave behind what they usually think of as evaluation and to enter into the capacities that are needed in complexity. These capacities include addressing problems obliquely rather than head on, making small safe to fail experiments, undertaking action to better understand the system rather than to effect a change, practicing true adaptive leadership which means practicing anticipatory awareness and not predictive planning, working with patterns and sense-making as you go rather than rules and accountabilities, and so on.
Last night a little twitter exchange between myself, Viv McWaters and Dave Snowden based on Dave’s recent post compelled me to explore this a bit further. What grabbed me was especially this line: “The minute we evaluate, assess, judge, interpret or whatever we start to reduce what we scan. The more we can hold open a description the more we scan, the more possibility of seeing novel solutions or interesting features.”
What is needed in this practice is monitoring. You need to monitor the system in all kinds of different ways and monitor yourself, because in a complex system you are part of it. Monitoring is a fine art, and requires us to pay attention to story, patterns, finely grained events and simple numbers that are used to measure things rather than to be targets. Monitoring temperatures helps us to understand climate change, but we don’t use temperatures as targets. Nor should we equate large scale climate change with fine grained indicators like temperature.
Action in complex systems is a never ending art of responding to the changing context. This requires us to be adopting more sophisticated monitoring tools and using individual and distributed cognition to make enough sense of things to move, all the while watching what happens when you do move. It is possible to understand retrospectively what you have done, and that is fine as long as you don’t confuse what you learn by doing that with the urge to turn it into a strategic plan going forward.
What role can “evaluation” have when your learning about the past cannot be applied to the future?
For technical problems in ordered systems, evaluation is of course important and correct. Expert judgement is required to build safe bridges, to fix broken water mains, to do the books, audit banks and get food to those who need it. But in complex systems – economies, families, communities and democracies, I’m beginning to think that we need to stop using the word evaluation and really start adopting new language like monitoring and sense-making.
Two weeks ago in our Leadership 2020 program I experimented with using a signification framework to harvest a World Cafe. We are beginning another cohort this week and so I had a chance to further refine the process and gather much more information.
We began the evening the same way, with a World Cafe aimed at exploring the shared context for the work that these folks are in. Our cohort is made up of about 2/3rds staff from community social services agencies and 1/3 staff from the Ministry of Children and Family Development. This time I used prepared post it notes for the sense making exercise, which you can see here:
Our process went like this:
- At Cafe tables of five for 20 minutes, discuss the question “What is a story of the future you are anticipating for this sector?”
- Second round, new tables, same question, 20 minutes
- About ten minutes of hearing some random insights from the group, and checking to see how those resonate.
- 2 minutes of silent reflection on the question of ‘What do you need to learn here that will help us all move forward?”
- Each participants took a pink and blue post it note. On the blue post it they wrote what they needed to learn that would be immediately applicable and on the red ones, learning that is needed to prepare for the future.
- Participants filled out the post-its and then were instructed on how to signify the data on a triangle framework that helped them signify whether what they needed to learn would help them “in their personal life,” “do their jobs” and/or “make change.”
- Participants also indicated on the post-its whether the worked for the Ministry or worked for a community organization.
At the conclusion of the exercise we had a tremendous amount of information to draw from. Our immediate use was to take a small group and use affinity grouping to identify the themes that the whole has around their learning and curiosity. We have used these themes to structure a collective story harvest exercise this morning.
But there is some much more richness that can come from this model. Here are some of the ways people are playing with the date:
- Removing all the pink post-its to see what the immediate learning needs are and vice versa.
- Looking at and comparing the learning needs between the two sectors to see where the overlaps and differences are
- examining the clusters at the extremes to see what ot tells us about personal needs, and professional needs.
- Uncovering a theory of change by looking at the post its clustered around the “Making change” point and also seeing if these theories of change are different between the community and the government.
And of course because the data has been signified on each post it, we can recreate the framework easily. The next level for me will be using this data to create a Cynefin framework using the four-points contextualization exercise. Probably won’t happen in this cohort.
Big learning is the rich amount of data that proceeds from collecting finely-grained objects, allowing for disintermediated sense-making, and seeing all these multiple ways in which signified data can be used to address complex challenges obliquely, which allows you to get out of the pattern entrainment that blinds you to the weak signals and emergent patterns that are needed to develop emergent practice. This pen and paper version is powerful on its own. You can imagine how working with SenseMaker across multiple signification frameworks can produce patterns and results that are many magnitudes richer.
Just off a call with a potential client today and we were scoping out some of the work that we might do together, with a small organization facing unprecedented change. They are in a place of finally realizing that they are not in control of what is happening to them. They are completely typical in this respect.
I am constantly struck by the fact that we have so few skills, frameworks and so little language for dealing with complexity. Clients all the time approach me looking for certainty, answers and clear outcomes. It’s as if they are searching for the one person who will promise them the relief they are looking for. And no one can. Because mostly what they are FEELING is their emotional reponse to the reality of a complex world. And no amount of rational and linear planning will address that feeling. in fact quite the opposite. Sitting down and deciding on a vision, goals, objectives and plan just defers the pain, because it fools you into thinking you are in control but it sets up a false ideal against which your progress will always be measured to be short.
Confronting complexity is hard. It is not merely that we need better tools to think about it. We need better tools to emotionally deal with it. it is overwhelming, infuriating, confusing, and frightening. And almost every organization I work with that fails to address it well fails because they don’t attend to the fear. They build fears into their processes, or they build processes to avoid confronting what they are afraid of: usually that we don’t know what’s going and we don’t know what to do.
My potential client asked me if I could say what outcomes would come from working with me. In brief they are this:
- We will build the capacity to understand and work with the problems you are facing in context by confronting and changing the view we take around complexity
- We will work strategically with the content of the project, and build participatory processes together that will change the way we do the work of addressing complex problems
- We will build resilient containers for the work that will allow us to confront our fears and limiting beliefs about the work and the change we are in, and that will provide a solid strategic framework for our project.
- We will arrive at a set of strategic decisions about the present moment and be prepared to make strategic decisions about the future.
That’s it. Sometimes those outcomes are incredibly concrete, sometimes it is more about building capacity, but it is always about acting strategically, and that sometimes means learning a new language and a new set of skills. I find that it’s the learning part with which people are most impatient. They seems to want to be able to accelerate the outcomes they want without having to change their approach. But, if you found yourself teleported to rural Bangladesh and you now had to make a living as a rice farmer, do you think your current language and skill set would be applicable, if only you applied yourself harder?
There are projects that fit the ordered domain of work, in which project management and strategic planning leads to predictable outcomes. And there is work for which “learning” is both the outcome and the new organizational structure and leadership practice. It is very important not to confuse the two contexts. And it is surprising just how much we are willing to turn a blind eye to complexity (as both a friend and a foe) in favour of a stable and knowable future, no matter how impossible that idea is.