Hmmm…maybe I should have let the group do this…
When I popped off to London last week to take a deep dive into Cognitive Edge’s work with complexity, one of the questions I held was about working with evaluation in the complex domain.
The context for this question stems from a couple of realities. First, evaluation of social programs, social innovation and other interventions in the human services is a huge industry and it holds great sway. And it is dominated by a world view of linear rationalism that says that we can learn something by determining whether or not you achieved the goals that you set out to achieve. Second, evaluation is an incredibly privileged part of many projects and initiatives and itself becomes a strange attractor for project planning and funding approval. In order for funders to show others that their funding is making a difference, they need a “merit and worth” evaluation of their funds. The only way to do that is to gauge progress against expected results. And no non-profit in its right mind will say “we failed to achieve the goals we set out to address” even though everyone knows that “creating safe communities” for example is an aspiration out of the control of any social institution and is subject to global economic trends as much as it is subject to discrete interventions undertaken by specific projects. The fact that folks working in human services are working in a complex domain means that we can all engage in a conspiracy of false causality in order to keep the money flowing (an observation Van Jones inspired in me a while ago.) Lots of folks are making change, because they know intuitively how to do this, but they way we learn about that change is so tied to an inappropriate knowledge system, that I’m not convinced we have much of an idea what works and what doesn’t. And I’m not talking about articulating “best practices.”
The evaluation methods that are used are great in the complicated domain, where causes and effects are easy to determine and where understanding critical pathways to solutions can have a positive influence on process. in other words, where you have replicable results, linear, summative evaluation works great. Where you have a system that is complex, where there are many dynamics working at many different scales to produce the problems you are facing, an entirely different way of knowing is needed. As Dave Snowden says, there is an intimate connection between ontology, epistemology and phenomenology. In plain terms, the kind of system we are in is connected to the ways of knowing about it and the ways of interpreting that knowledge.
I’m going to make this overly simplistic: If you are working with a machine, or a mechanistic process, that unfolds along a linear trajectory, than mechanistic knowledge (problems solving) and interpretive stratgies are fantastic. For complex systems, we need knowledge that is produced FROM the system and interpreted within the system. Evaluation that is done by people “outside” of the system and that reports finding filtered through “expert” or “disinterested” lenses is not useful for a system to understand itself.
Going into the Cynefin course I was interested to learn about how developmental evaluation fit into the complex domain. What I learned was the term “disintermediated sensemaking” which is actually the radical shift I was looking for. Here is an example of what it looks like in leadership practice.
Most evaluation uses processes employing a specialized evaluator undertaking the work. The problem with this is that it places a person between the data and experience and the use of the knowledge. And it also increases the time between an experience and the meaning making of that experience, which can be a fatal lag with strategy in emergent systems. The answer to this problem is to let people in the system have direct experience of the data, and make sense of it themselves.
There are many many ways to do this, depending on what you are doing. For example:
- When clustering ideas, have the group do it. When only a few people come forward, let them start and then break them up and let others continue. Avoid premature convergence.
- When people are creating data, let them tag what it means, for example, in the decision making process we used last weekend, participants tagged their thoughts with numbers, and tagged their numbers with thoughts, which meant that they ordered their own data.
- Produce knowledge at a scale you can do something about. A system needs to be able to produce knowledge at a scale that is usable, and only the system can determine this scale. I see many strategic plans for organizations that state things like “In order to create safe communities for children we must create a system of safe and nurturing foster homes.” The job of creating safe foster homes falls into the scope of the plan, but tying that to any bigger dynamics gets us into the problem of trying to focus our work on making an impact we have no ability to influence.
- Be really clear about the data you want people to produce and have a strategy for how they will make sense of it. World Cafe processes for example, often produce scads of data on table cloths at the centre of the table, but there is often so little context for this information that it is hard to make use of. My practice these days is to invite people to use the table cloths as scratch pads, and to collect important data on post it notes or forms that the group can work with. AND to do that in a way that allows people to be tagging and coding the data themselves, so that we don’t have to have someone else figure out what they meant.
- Have leaders and teams pour over the raw data and the signification frameworks that people have used and translate it into strategy.
These just begin to scratch the surface of this inquiry in practice. Over the next little while I’m going to be giving this approach a lot of thought and try it out in practice as often as I can, and where the context warrants it.
If you would like to try an exercise to see why this matters try this. the next time you are facilitating a brainstorm session, have the group record dozens of insights on post its and place them randomly on a wall. Take a break and look over the post its. Without touching the post its, start categorizing them and record your categorization scheme. Then invite the group to have a go at it. Make sure everyone gets a chance to participate. Compare your two categorization schemes and discuss the differences. Discuss what might happen if the group were to follow the strategy implicit in your scheme vs. the strategy implicit in their scheme.
I had the great pleasure of coaching a team of folks last night who were running their first World Cafe. I’ve been working with this crew for a while – a core team looking at the future of the Victoria Presbytery of the United Church of Canada – and this was the first time they’ve stepped up to run their own conversational process as part of our work. Last night it was a Cafe to sense the future of what the Presbytery could be and do. And they did great.
One of the advantages of coaching is that one gets to reflect on the little bits and pieces of practice that make things work. Last night a number of them came up, so I thought I’d share them here.
Give instructions one at a time. Don’t give a long list of instructions. At the beginning of the Cafe let people know how the time will flow, but when it comes time to invite people to do certain things (move between tables, change questions, reflect, summarize…whatever) just give one instruction at a time. It is important that people know WHY we are doing a thing, but not important that they have the whole flow. And especially if your instruction involves them moving, then don’t give any more instructions until they have stopped.
Invite people to mark the paper early. The paper in the middle of the table is for all to use. “Typical” facilitated sessions imprint people with the pattern that someone will take notes while everyone else talks. It’s important that before the conversation begins, you invite people to pick up a marker, write something and draw something on the paper in front of them. That way, before the conversation begins, folks know that the paper is for everyone to use, there is no top or bottom, and images and words are equally welcome.
Have one more marker and one fewer post it note than people. If you have tables of four, give them five markers. This means that people can trade colours without prying a marker from someone’s hand. And if you are summarizing key findings, have three post-its for a table of four, to encourage people to pick three things together rather than just having everyone put their best thought down. World Cafe is about tapping and making visible collective intelligence. You lose that if you just have individual thoughts.
Build in silence. At the conclusion of a round, have a minute or two of silence. It calms the room down, allows people to reflect and integrate what they are hearing and makes it easier to give directions. This is especially important if you are wanting people to raise their level of awareness from what is important personally to what patterns are emerging. It requires a shift in awareness to see that.
Collect post its before having a summary conversation. Last night we used post its at the conclusion of the third round to capture the patterns that people were hearing consistently in all three rounds. Collecting the post-its before we had a summary conversation meant that people couldn’t “report out” and instead we hosted a “conversation with the whole” whereby we roved around asking people what stood out for them. What emerged was indeed a conversation and not a boring reporting out of things that everyone knew anyway.
Avoid the temptation to use a different question for each round. This is important. Having a different question for all three rounds creates three shallow conversations and inhibits pattern finding. It can also leave people feeling like they are being led down a garden path and it doesn’t leave a lot of space for emergent conversation. For all Cafe beginners, I always suggest they do their first Cafe with a single question for all three rounds. This gives you a clear picture of how the process can work to surface COLLECTIVE intelligence.
Keep the question simple and broad and make sure you can answer it on your own. Trust the group. They want to have a conversation, not guess at answers that you are trying to get them to. Last night our question was simple; given a context in which the structures of the Church are becoming increasingly unsustainable and in which congregations still need to be connected on a local level “What should Presbytery be and what should it do?” That was it. Three rich rounds on that, with lots of great insight and some amazingly courageous admissions (“Time to finally admit that this structure is dead.” etc.)
Invitation matters. Even though the 50 people we had out last night are used to being together every few months, the core team mworked on their invitation for a month. They held the purpose of the event close (discovering what the new shape and function of the Presbytery could be) and they shared the question with participants, even before we had decided on what the final question was. The team made sure people RSVP’d on the invitation which helped us to know the logistics of food and space, and also gave a chance for the conversation to begin as folks started sharing what they were thinking right away. This primed the conversation and meant that people were really ready for the work. Ninety minutes was not enough.
Know what you will do with the harvest and tell people. People learned in the invitation what our plans were for the harvest. This even was about helping the core team design some experiments over the next year for new ways that the Presbytery could meet and be useful to the two dozen United Church congregations on southern Vancouver Island. We summarized the patterns that people found (above photo) and began right away writing a report. But the bigger piece of work will be engaging in design over the next couple of months to create new and interesting gatherings in line with what the Presbytery members actually want.
This week I was hosting at a moderately sized conference in Victoria BC with 100 regional public sector union members. The purpose of the gathering was to increase the number of active members and to inspire members to engage and improve local communities. These union members all work in the public service and so they have a close ear to the ground on the issues facing communities from homelessness to addictions to environmental degradation to service levels in health and education. Many of them took public service jobs in the first place because they are caring and committed people, intent on making a better world, especially for the most vulnerable.
This is the fourth year we have done this conference, and the structure has remained pretty much the same over the past four years. The first evening there is a keynote from the union president (who then stays and participates through the whole two days) and a special speaker, in this case a well-known progressive lawyer who is currently running for office in a local federal by-election. That is usually followed by a plenary panel, which this year featured some provincial politicians from the labour movement and the current legislature and a journalist.
Day two begins with morning workshops on community organizing. in the afternoon we begin with a World Cafe. This year we took the Cafe through the following flow:
- Two rounds on the question of “What does all of this inspiration mean for my own community activism?”
- One round on the question “what do I still need to learn to deepen my activism?” The harvest from that round was a post it note from each participant outlining some of their learning needs, which union staff will use to help support the members with resources and materials.
- Following that round I invited participants to reflect on an area of focus for their activism, such as homelessness, environment, youth engagement and so on. Participants wrote their focus on the blank side of their name tags and then milled around the room and found others who shared those areas of focus. We ended up with about 12 groups composed of people from across the region who didn’t know each other and who were interested in working in the same issue area.
- Using this network we next invited the participants to consider the question “What are some of the key strategic actions we can take in this sector?” The harvest from this was simply to inspire and connect each other in preparation for the next day’s work.
That was the end of our days work. A quick poll of the room showed that perhaps 20 people had some ideas for action that were considering.
This morning was devoted to a ProAction Cafe. We had 21 tables in the room and I opened up the marketplace. It took about 20 minutes for 21 hosts to come forward and for everyone to get settled. From there we followed a standard ProAction Cafe format. During the reflection period, when participants are given a break and hosts are able to take a breath and make sense of all the advice we heard, three people all working on engagement strategies got together to compare notes. This helped them a lot before the fourth round as they were able to point to work the others were doing. The action networks were already taking shape!
We finished in just under 2.5 hours. In previous years we ran Open Space meetings on the last morning, but this year the shift in format gave a more concrete set of actions and surfaced more leadership in the room. With a quarter of the room engaged as hosts, we topped the average 20% of the room from previous years using Open Space. ProAction Cafe, used at the end of a conference to generate and develop concrete actions is so far the best process in my practice for getting good ideas out of the room with passion, precision and participation.
SItting here with Geoff Brown and Steven Wright at the World Indigenous Housing Conference here in Vancouver. We are on the back end of what has been a terrific gig.
We were hired by the Aboriginal Housing Management Association of BC to facilitate dialogue at this 800 person international gathering. The sponsor made dialogue a clear priority and after talking about intentions, we arrived on the design of three World Cafes: one in the plenary with everyone present and two in more focused breakout sessions. The first cafe would look at stories of success, the second would think about how to build capacity to support success and the third was focused on institutional development. each one built on the last.
The theme of the conference was “Sharing our Stories, Sharing our Successes.” With that theme to play with, we knew the cafes needed to be about connecting people and ensuring that stories were central to the work. Our first challenge was to think about how to harvest stories and connections quickly from 800 people. We looked at several tech solutions and realized that we needed something simple, unobtrusive and accessible. The ubiquitous tool at hand was the text equipped smart phone. Almost everyone has one, and almost everyone can text. Our basic problem was first how to gather text messages and second how to make meaning from them quickly. Geoff, Steven and I were familiar with Wordle.net which makes a word cloud out of blocks of text, and which I have used in the past to get a visual and intuitive sense of what concepts and words are weighted highly.
So our question became, how can we combine smart phones, text messages and wordle?
Through our networks we found Luke Closs, a local developer/hackerwho put together a simple solution that he called “Text to Cloud.” At the back end he connected Twillio to world using an interface that we could control with commands sent by text message. groups of texts that come in can be tagged and sorted and then sent straight to Wordle for processing. We also enabled the software to produce a CSV output that we can use for other purposes. Luke was great, developing the tool right up to the moment that his daughter was born on Tuesday. Of course, the tool is open source and you can find it on Github, download and install it and use it for yourself.
Armed with Text to Cloud, we began our first cafe by inviting people to text in the name of their tribe of origin. We created an instant wordle that showed who was in the room. That immediately connected people together (and showed we were blessed with Crees!)./ Following that we had people enter into the cafe to start telling stories of successes with listeners paying attention to the factors that made those successes possible. People gathered information on tablecloths and texted in wisdom and insights and by the end of the cafe we had 438 text messages to make meaning from. We had a half hour to do something with all this.
So we sent it all to Wordle and discovered a theme: Building Homes, Building Communities and Building Nations. There were six key areas we needed to think about for capacity building: governance, building, partnerships, community, education and ownership. Steven whipped up a digital mind map which we projected on our screens. We invited people at each table to choose one of the topics and dive into stories of capacity building. In our third cafe, we thought about how institutions can support sustained capacity building.
By the end of the day we were soaking in flip chart paper, but we had some great high level meaning through the Text to Cloud output, the wordles and the developmental nature of the conversation. We retreated to Steven’s room and started trying to figure out how to share what we had learned. We realized early on that there was absolute gold on the flip charts, so we decided to create a presentation that combined what Geoff calls “vox pops” – short pithy and insightful comments – along with longer stories. While Steven created a map to chart the highlights, Geoff and I prepared a slideshow that touches on the headlines. Our plan this afternoon is to call the storytellers up to the stage to share their stories with the audience. They are the true key notes.
This gig has been fun. Our client has been fantastic, we’ve created new tools, connected people doing important work, pushed our own edges and done stuff we’ve never done before, and that we could never have done alone. It was a superb co-creative experience and a great way to spend time with good friends.
I was up north on the weekend, working with a small community that has been driven apart by a large and contentious decision. It doesn’t matter what it was, or what either side wanted – the result is the same result that happens in many small communities: people who are friends and neighbours shouting and fighting with each other.
The team I was working with are trying to reinvent the way this community is engaged. We used a lovely redux of Peter Block’s work to help frame our conversation about design and implementation. A few things stood out for this group with respect to Peter’s work.
Changing the room changes the conversation. We talked a lot about the fact that changing engagement starts in this room and in this moment because this room IS the community. When we dove in about what was missing from the way the community engages it was clear that the ownership piece was the biggest one. As in many community meetings the way people traditionally engage is with passion that is directed outward. There is an expectation that someone else needs to change. We joked about the sentiment that says “I’ll heal only after every else has healed!” It was a joke but the laughter was nervous, because that statement cuts close to the bone. So we DID change the room and decided to hold a World Cafe. gathered around smaller tables, paper in the middle, markers available for everyone to write with…
So how do you begin a meeting with people who are invited to take up the ownership of the outcome? I am not a fan of giving people groundrules, because as a facilitator it puts me in the position of enforcer, and gives people an out for how the behave towards one another. So instead we considered the question of what it looks like when people are engaged. What stood out is how people “lean in” to the centre of the conversation. So the question became, how do we get people to lean in right away and take ownership of the centre?
The solution was simple but was later revealed to have tons of power. At the outset of the cafe as I was introducing the process I gave the following instructions:
“That paper in the middle is for all of you to use, as are the markers. We want you each to record thoughts and insights that other need to hear about. So before we begin I invite you to pick up a marker and write your name in front of you. <people write their names>. Now I want to invite you to answer this question: what is one thing you can do to make sure that this meeting is different? Write your answer beneath your name.”
People took a moment to write their names and their commitments. And they shared them with each other at the table. That is how we began.
The first round of conversation proceeded as usual, but I noticed something very powerful in the second round. When everyone got up and moved around they took a seat in someone else’s place, and often the first thing they did was to read the name and the commitment that was in front of them. Can you imagine coming across the name of someone who you have a disagreement with only to see that they have written “I won’t fight anymore” beneath their name? The core team is now going through all of the tablecloths and making a list of the commitments that people made. Taken on their own, they form a powerful declaration of willingness.
People reported that this was the best meeting the community had in a long time. And it had a lot to do with this tiny intervention of public ownership for the outcomes.