Hmmm…maybe I should have let the group do this…
When I popped off to London last week to take a deep dive into Cognitive Edge’s work with complexity, one of the questions I held was about working with evaluation in the complex domain.
The context for this question stems from a couple of realities. First, evaluation of social programs, social innovation and other interventions in the human services is a huge industry and it holds great sway. And it is dominated by a world view of linear rationalism that says that we can learn something by determining whether or not you achieved the goals that you set out to achieve. Second, evaluation is an incredibly privileged part of many projects and initiatives and itself becomes a strange attractor for project planning and funding approval. In order for funders to show others that their funding is making a difference, they need a “merit and worth” evaluation of their funds. The only way to do that is to gauge progress against expected results. And no non-profit in its right mind will say “we failed to achieve the goals we set out to address” even though everyone knows that “creating safe communities” for example is an aspiration out of the control of any social institution and is subject to global economic trends as much as it is subject to discrete interventions undertaken by specific projects. The fact that folks working in human services are working in a complex domain means that we can all engage in a conspiracy of false causality in order to keep the money flowing (an observation Van Jones inspired in me a while ago.) Lots of folks are making change, because they know intuitively how to do this, but they way we learn about that change is so tied to an inappropriate knowledge system, that I’m not convinced we have much of an idea what works and what doesn’t. And I’m not talking about articulating “best practices.”
The evaluation methods that are used are great in the complicated domain, where causes and effects are easy to determine and where understanding critical pathways to solutions can have a positive influence on process. in other words, where you have replicable results, linear, summative evaluation works great. Where you have a system that is complex, where there are many dynamics working at many different scales to produce the problems you are facing, an entirely different way of knowing is needed. As Dave Snowden says, there is an intimate connection between ontology, epistemology and phenomenology. In plain terms, the kind of system we are in is connected to the ways of knowing about it and the ways of interpreting that knowledge.
I’m going to make this overly simplistic: If you are working with a machine, or a mechanistic process, that unfolds along a linear trajectory, than mechanistic knowledge (problems solving) and interpretive stratgies are fantastic. For complex systems, we need knowledge that is produced FROM the system and interpreted within the system. Evaluation that is done by people “outside” of the system and that reports finding filtered through “expert” or “disinterested” lenses is not useful for a system to understand itself.
Going into the Cynefin course I was interested to learn about how developmental evaluation fit into the complex domain. What I learned was the term “disintermediated sensemaking” which is actually the radical shift I was looking for. Here is an example of what it looks like in leadership practice.
Most evaluation uses processes employing a specialized evaluator undertaking the work. The problem with this is that it places a person between the data and experience and the use of the knowledge. And it also increases the time between an experience and the meaning making of that experience, which can be a fatal lag with strategy in emergent systems. The answer to this problem is to let people in the system have direct experience of the data, and make sense of it themselves.
There are many many ways to do this, depending on what you are doing. For example:
- When clustering ideas, have the group do it. When only a few people come forward, let them start and then break them up and let others continue. Avoid premature convergence.
- When people are creating data, let them tag what it means, for example, in the decision making process we used last weekend, participants tagged their thoughts with numbers, and tagged their numbers with thoughts, which meant that they ordered their own data.
- Produce knowledge at a scale you can do something about. A system needs to be able to produce knowledge at a scale that is usable, and only the system can determine this scale. I see many strategic plans for organizations that state things like “In order to create safe communities for children we must create a system of safe and nurturing foster homes.” The job of creating safe foster homes falls into the scope of the plan, but tying that to any bigger dynamics gets us into the problem of trying to focus our work on making an impact we have no ability to influence.
- Be really clear about the data you want people to produce and have a strategy for how they will make sense of it. World Cafe processes for example, often produce scads of data on table cloths at the centre of the table, but there is often so little context for this information that it is hard to make use of. My practice these days is to invite people to use the table cloths as scratch pads, and to collect important data on post it notes or forms that the group can work with. AND to do that in a way that allows people to be tagging and coding the data themselves, so that we don’t have to have someone else figure out what they meant.
- Have leaders and teams pour over the raw data and the signification frameworks that people have used and translate it into strategy.
These just begin to scratch the surface of this inquiry in practice. Over the next little while I’m going to be giving this approach a lot of thought and try it out in practice as often as I can, and where the context warrants it.
If you would like to try an exercise to see why this matters try this. the next time you are facilitating a brainstorm session, have the group record dozens of insights on post its and place them randomly on a wall. Take a break and look over the post its. Without touching the post its, start categorizing them and record your categorization scheme. Then invite the group to have a go at it. Make sure everyone gets a chance to participate. Compare your two categorization schemes and discuss the differences. Discuss what might happen if the group were to follow the strategy implicit in your scheme vs. the strategy implicit in their scheme.
My mate Geoff Brown has a great post on complexity in practice as he aimed to get well after an injury. This is a fantastic story.
After a couple of session of observations, conversations about pain sites, work habits and past physical/medical history, my Osteopath discovered a complex pattern of underlying, causal factors that contributed to my injury. Together, these factors like tight hamstrings and assymetrical posture interact with each other to create an emerging pattern of dysfunction. Translated to English = the many little problems with my body mechanics create weaknesses that make me more vulnerable to injury. So, like in all complex systems watch the Wolf in Yellowstone National Park video below the underlying cause of my sudden injury is not that clearcut. The remediation of these multiple factors is even more complex.
Hope you’re feeling better Geoff!
via How to work with a complex system – lessons from an Osteopath | Tangent Consulting.
This afternoon I’m coming home after a morning running a short process for a church in Victoria, BC. The brief was pretty straightforward: help us decide between four possible scenarios about our future. Lucky for me, it gave me an instant application for some of the stuff I was learning in London last week.
The scenarios themselves were designed through a series of meetings with people over a number of months and were intended to capture the church’s profile for its future, as a way of advertising themselves for new staff. What was smart about this exercise was the fact that the scenarios were left in very draft form so there was no way they could be confused for a “vision” of the future. It is quite common in the church world for people to engage in “visioning exercises” to deal with the complex problems that they face, but such visions are doomed evermore to failure as the bigger organization is beginning to enter into a period of massive transformation and churches are suffering from all kinds of influences over which they have no control.
Visioning therefore is not as useful as selecting a lens through which the organization can make some decisions.
Each scenario contained some possible activities and challenges that the church would be facing, and the committee overseeing the work was charged with refining these down to a report that would, to use my own terms, be a collection of heuristics for the way the organization would act as it addressed future challenges.
Our process was very informed by some thinking I have been doing with Dave Snowden’s “Simple rules for dealing with complexity.” Notably principles about avoiding premature convergence, distributing cognition and disrupting pattern entrainment. Furthermore, the follow up work will be informed by the heuristic of “disintermediation” meaning that the team working on the project will all be working with the raw data. There is no consultants report here. The meaning making is still very much located with the participants.
So here was our process.
- At small tables of four, participants were given 5 minutes to read over the scenarios silently.
- We then entered a period of three 15 minute small group conversations on the topic of “what do you think about these scenarios?” Cafe style, each conversation happened with three different groups of people. I was surprised how much introduction was going on as people met new folks. The question was deliberately chosen not to be too deep or powerful because with a simple question, the participants will provide their own depth and power. When you have a powerful need, you don’t need to contrive anything more powerful than what people are already up for.
- Following the cafe conversations, a round of silent reflection in which people were given the following direction. “Express your preference for each of the scenarios on a scale of 1-7. Seven means “Let’s do it” and one means “No Way.” For each scenario write your preference on your post it and write a short sentence about the one concrete thing that would make your vote one point higher.” So there is lots in this little exercise. First it’s a way of registering all of the objections to the scenarios without personalizing them. Secondly it gets at concrete things that the team can do to improve scenarios and third it harvest preferences and not simple yes/no decisions which are not appropriate for this kind of work.
- At each table someone gathered all the posts its of the same colour and by colour folks came to the front and placed them on the scale. Doing it this way meant that no one was sure whose preference was going where and it also meant that people couldn’t revise their post its once they saw how the preferences were being expressed.
The whole thing took about 75 minutes.
The result of this sense making was the chart you see above. Two hundred pieces of finely grained information ordered by the people themselves. The project team now has at least three things they can do with this material.
- They can recreate the scale, as each post it is colour and preference coded. That way they have a rough idea of the scenario with the greatest support, and they can show anyone who wants to see metrics where we stand on the proposals.
- They can cluster post its for each scenario according to “work that will make it better” which means they don’t have to pay attention to the scale. The scale is completely subjective, but each of these post-its contains one piece of concrete information to make the scenario better, so in some ways the numbers don’t really matter. They can cluster these ideas by each scenario AND they can re-cluster them by each topic to give an idea of overall issues that are happening within the organization.
- If we wanted to go a step further, we could use these post it notes to do a number of Cognitive Edge exercises including a Cynefin contextualization (which would tell us which things were Obvious, Complicated and Complex (and maybe Chaotic) and we could also do some archetype extraction which might be very useful indeed for constructing the final report, which would stand as an invitation to thier new personal and an invitation to the congregation.
Back from London now from a four day deep dive into complexity theory and Cynefin practice with Dave Snowden and Tony Quinlan from Cognitive Edge. It was a packed full four days with many many many bits and pieces of philosophy, natural science, organizational theory and a few exercises thrown in. It was presented in a straight up lecture format, eight hours a day with one or two exercises and some short periods of conversation. The best reflection periods were the three to four hours afterwards with classmates engaged in what my new Welsh friend Sion Charles and I called “Celtic reflection” which obviously involves pints, craic and towards the end of the evening, a night cap of whisky (and I do owe you a round of malt Sion!). Oh, and really terrific chats about what we had learned and how it can be applied.
At least I was ready for it, having had several friends tell me that the pedagogy is all about “download from the front, try a few things at your tables.” When you know this is what you are going to get, you just go in prepared. It’s not how I teach, but then I wasn’t there to see an imitation of myself. I was there to hear the latest thinking from Dave (especially around the Cynefin sub domains, Cynefin dynamics and the other bits and pieces of theory he’s chasing down) and to hear practitioner stories and experience some of the methods, which Tony showered down on us on days one and four. I was alos curious to explore how Cynefin might better inform my thinking about developmental evaluation. I think now I have a good idea of Cognitive Edge’s approach with clients, and some of the heuristics and principles for applying Cynefin and designing exercises that help us work in the complex domain. And I have a few new lines of inquiry and practice around developmental evaluation that might make their way into some new teaching material, and perhaps a new offering.
I am not new to this framework at all, and it has been a staple of my work with clients over the past few years. I find that it helps to present a strong and clear sense of why you need to do things differently when you are faced with complexity. It helps us understand the point of dialogic approaches to problem-addressing, and in deeper applications, it helps us to adopt better strategic practices for working with emergent and evolutionary situations. I have even worked with SenseMaker(TM) on a project in the United States and learned quickly how people with traditional social science research mindsets hit the wall with gathering data for collective sense-making rather than expert analysis.
As a reference point I though I would gather together a few of my pieces on Cynefin, including videos of me teaching the framework, illustrated with stories from my own practice. So here is a recap of what I know about the framework so far, and it will be interesting to see how that changes as the future unfolds.
Video of me teaching
Webinar recording
Blog posts
I’ve had a couple of requests to share the exercise that helps people understand Cynefin physically. I’m happy to do so here.
I enjoy designing these kinds of exercises, as it gives people a number of ways of understanding the framework and I find that it actually helps the penny drop for folks who otherwise have a hard time groking the nature of the different domains. I am continuing to develop this exercise but here’s how I do it now:
1. Begin by having the group stand and clear a large space so that everyone can move around comfortably. To do this well, you need a large open space with lots of room for people to move. As you give directions, just give folks a simple instruction and don’t allow questions. They have to figure the rest out themselves. At the end of each mini exercise take a little conversation to reflect on questions such as “how did you do this?, what is happening here?, how did you gather data? How did you evaluate your efforts?” Use questions that are relevant to the applications of Cynefin you are dealing with.
2. Exploring the obvious. Have people divide into four groups (they don’t have to be even numbers). Instruction: “Organize yourselves by height.” Things to note: this can happen quickly, top-down leadership works well, it can be evaluated objectively. You can constrain the exercise further by instructing people to complete the task in 15 seconds. It is unlikely you will be surprised by the results of this exercise.
3. Exploring the complicated. Have people divide into four new groups. Instruction: “Organize yourselves by birth month and year.” Notes: there are many ways to do this, each can be objective;y verified. It requires getting hidden data that is easily discovered and top-down leadership still works well. You will find some surprising solutions for this problem.
4. Exploring the complex. Have everyone stand in a circle and introduce “The Systems Game” (I learned this version from Joanna Macy’s work). In this well known game, individuals must identify two other people and move to a place equidistant from each of them. You cannot let your “targets” know you are connected to them. It helps to demand that people try to achieve a high degree of accuracy in this triangulation. Done well, and with lots of space in the room, the group should be set into a pattern of constant motion. Notes: small rules initiate constant complex motion. You will see times when a group is clumped up and other times when it is spread out. Notice how some folks are naturally influential in the group – tall men wearing bright clothes seem often to have a higher number of connections to themselves. Notice how it feels to be constantly moving and adjusting. If people stop moving ask why (usually they are tired of the game, a fact of life that translates into dealing with real world complexity). Leadership is participatory and top-down leadership cannot help. When the group gets tired of the exercise, invite some probes to see what happens when certain people move. You will start to see the patterns of connection better that way. This is a good introduction to developmental evaluation. Once the system is at rest, it’s difficult to evaluate the connections. Probes (inviting certain people to move to a very different place, for example) gives us lots of information. Have the group devise their own probes to illuminate more of the situation.
5. Exploring chaos. Have people start “milling.” Milling is a practice from theatre training where participants are instructed to walk into space, rather than walk in a circle. Keep the speed medium pace, and ask them to listen to your instructions. Instructions proceed as follows:
- “When I say stop, stop. When I say go, go.” Do this for a while, giving commands to the group.
- “When I say clap, clap,. When I say jump, jump.” Do this for a while mixing up commands to stop, go, jump and clap.
- “When I say go, stop and when I say stop, go” Instroducing this kind of disruption starts making following directions difficult.
- “When I say bow, bow, when I say whoopee, shout whoopee! When I say clap, jump, when I say jump, clap.” Continue and increase the pace of your commands.
- “When I say shhh say shhh, when I say thigh, slap your thigh. Whoopee, bow; bow, whoppee…” We add one more pair of commands and continue disrupting people’s experiences.
- Continue to flip commands. It will get very chaotic.
Notes: “leadership” is increasingly difficult. Any strategy you develop for keeping the commands straight will be disrupted by randomized instructions. It takes a lot of attention to keep going, and eventually a breakdown is going to happen. Some will simply follow instructions as best the can, rendering the exercise simple. Others will try to devise coping strategies; others will give up and do their own thing. You could notice the tip from a simple exercise to a chaotic one and how difficult it is to cope as a group when you enter into chaos this way.
6. Exploring disorder. Have people divide the group into four groups. Invite people to organize themselves by a word that is both a verb and a noun. Pick one from this list. Words like this are sufficiently ambiguous that the groups have to figure out what is meant by the word before they can do the exercise. Any word will do. Notes: the group will become keenly aware of the difference between chaos and disorder. Have people reflect on their initial reaction to hearing the word. It is likely that each person instantly developed a strategy to address the challenge. you could slow the exercise down and have everyone take a minute to write down their strategy and then share them with the group. People will be surprised at the variety. This is a good lesson in what happens when a groups makes a decision without getting clear on what the problem is.
After the exercises I then give my own standard teaching of the framework, which can take from 30 minutes to an hour depending on how much discussion we have.
Hope this helps. Leave me a comment if you try the exercise so we can all learn from your experience.