Sometimes people see that I’m a dialogue practitioner and the assume that I am not a fan of quantitative measurement. I think this has to do with the fact that the dialogue practitioner community has been a kind of antithesis to the “measure and manage” world of empirical scientific management.
In any endeavour both qualitative and quantitative measurements are important. The issue isn’t whether or not numbers are to be more trusted than meaning making; the issue is whether we are measuring thing properly.
The issue is whether or not we use measurements as targets or gauges.
Again, this is helpful in understanding the distinction between summative and developmental evaluation and sensemaking. In a linear system, you are aiming for certain end states and targets. In a complex and non-linear system you are aiming to keep to vectors. So using technology to increase production by 5% and decrease expense by 15% can be achieved and you can look back and see how well you achieved that target. You can also do tests and host conversations with workers and customers to discuss the quality of your product, aiming for a general score of “happy” which in turn might be reflected in numbers like sales, returns, recommendations and so on.
In a complex system, lilke an organization’s culture however, you are not managing for a target, but rather you are managing a kind of balance and a direction. You get to choose that direction from your own moral and ethical sense of what is right to do. For example, maintaining an organizational culture of openness, respect, creativity and support requires monitoring your culture in real time, a lot, and noticing how things are shifting and changing. Dialogic methods play an important role here, especially in perceiving patterns and making decisions about what to do, as well as engaging people in the endless negotiation about what those values look like on a daily basis. As a management tool, developing skillful dialogue tools allow you to manage the day to day issues with departures from your preferred set of values, beliefs or practices. Being complex, things like organizational cultures won’t always act they way you want them too, and so good leaders do two things well: they help resolve the inevitable violations of standards and practices in a manner that reflects the preferred way, and they gather together people over time to discuss what everyone is learning about the way the culture is working.
It’s not good enough to convene an annual meeting about the organization’s values and culture. That simply gives you a snapshot in time and tells you nothing about how an organization is evolving and changing, nor does it provide information about promising practices. To monitor over time, you can use a tool like CultureScan or a series of other regular ways of documenting the small observations of daily life that together help provide a picture of what the organization is doing.
Evaluation is such an influential constraint in organizational and community life. When resources and attention are tied to evaluation results, a kind of tautology gets set up. One begins managing projects towards the evaluation outcomes, in order to give the best chance of an initiative surviving and continuing to attract resources. One of the things I appreciate about developmental evaluation is its deliberate engagement with emergence. Making sense of emergence however can be a really time consuming affair, and so I’m thinking about how we can use good use of time to use dialogue and collective meaning making to help make sense of data and direction.
Developmental evaluation is for the complex domain. That means that we are not working with evaluating actions against desired end states, but instead noticing and paying attention to vectors and directions – intentions and hypotheses that help shape emerging strategy. Developmental evaluation is the process of gathering information about our work to give us some intelligence about what we are doing.
Think of the information needs of two different kinds of athletes. A golf player relies on solid objective data (how many yards to the hole, where the wind is coming from, the nature of the lie of the ball and so on) and interprets that data through her own self-knowledge (I hit a five iron 160 yards. Adjusting for wind and lie and the target topography, I should hit a 4 iron with backspin…) Of course the better a golfer one is, the easier it is to execute a plan and understand exactly where one succeeded or failed.
By contrast soccer players work in a dynamic environment. The information available to them only becomes apparent as they begin to play the match. They may know something about the other team, but they learn rapidly in the first ten minutes or so how the game is going to go. A team will discover where the opposition’s weakness is, or what its attacking strategy is, or where the open spots are on the pitch. Making good use of this information requires excellent communication in real time to share what is being learned. It requires players to play with potentials and patterns rather than certainties. Every move provides yet more information. The better a team works together, the faster they can adjust their strategy to take advantage of potentials.
When we are evaluating work there is a mix of these two types of approaches at play. Summative evaluation will look at the gap between expected outcomes and what actually happened and suggest how to adjust for next time. Budget planning and auditing is a good example of this technical kind of results based evaluation. Count the money and compare against projections. Look for causes. Some of these causes will be technical and some will be down to culture.
Developmental evaluation requires a different strategic approach, and simply put, it might fall into these four things (I’m trying for simplicity here, to try to be able to describe this in an easy way):
- Data points that give us the ability to capture information about a current state of an evolving system. This can render a series of pictures that will allow us to see patterns and trends. You need multiple snapshots over time to make sense of what is happening. One photo of a soccer game in progress tells you nothing. You need to monitor indicators not manage end points. Soccer is much more than just putting the ball in the net, even though that is the desired end result.
- Feedback loops from data to human sensemaking so that data can be used in real time to develop strategy and adjustments to the directionality of work.
- A facilitated sensemaking process to bring together multiple perspectives to interpret what is happening. In a complex system the data won’t give you answers. It will provide information to form hypotheses about the patterns that are emerging, and that information can give you guidance for action.
- A way of acting that doesn’t over commit resources to emerging potential strategies, but which gives enough momentum to see if we can shift things in a desired way. Snowden calls this “safe-to-fail.” This is tricky and calls for good context dependant leadership, but it is the essence of good decision making.
There are all kinds of ways of implementing these strategies. You can use surveys to discover what people are accessing on your website and you can use interviews or sensemaking tools to find out HOW they are using that information. You can use a strategic group to interpret these results and see how they are either coherent with our intentions, or at odds with them. You can then create new initiatives that support what is emerging or figure out ways to abandon what is not working. There are thousands of dialogue methods and processes to use to ask questions about and develop action around the data that is emerging.
Importantly, developmental evaluation needs to be a part of the way you work strategically. It needs a rhythm and a cadence to it, so that you know you are coming back on a regular basis to the emerging picture of what is happening. You need outsiders occasionally to come in and disrupt your point of view and offer alternative views of the patterns, and you need to choose a longer rhythm to continue to develop and refine your evaluation strategy as a whole.
I want this to be simple as a process to use. Strategy without information is just a wild guess. But if we tie our decisions too closely to the data emerging from dynamic systems we can get equally stuck making decisions that try to game the system towards desired results, with sometimes disastrous results for clients, customers and ultimately, organizational integrity. It’s a balance and a practice. How can we make this easy?
A few years ago, Juanita Brown shared a very powerful image with me. She talked about how those of us that practice dialogue and facilitation in a deep way have access to various gateways that take us into a “central garden.” All of our pathways invite us into this garden where we come to discover and realize something about the role of dialogue, meaning making and collaboration. It is a set of realizations that lies beneath the practice of methods.
On a call today with my friend Mark McKergow, we were discussing this image There are a bunch of us – although not a large bunch of us – from different practitioner communities who are always interested in transcending our methods and entering into this conversation. Alongside Juanita, Mark has also been wondering “where is everybody else, and how come we’re not connecting?”
Today we were discussing the failure of dialogue to have enough presence to provide workable and practical alternatives to everything from public policy decisions (such as the EU referendum in Britain, or the polarization of US society) to the everyday challenges of managing and running large organizations, evaluating, strategizing and controlling outcomes, people and money.
We know that our field of dialogic practice is massive, well researched and well documented. We know that leadership literature is filled with the importance of relational and sense making work. We know that that mid-career professionals end up coming to our various workshops to take on skills and ideas that are fundamentally transformative to their work and lives and that they go back to places where “it’s difficult to implement” because other mid-career professionals are wedded to globalized management practices that are good enough for what they are trying to do, within the highly constrained performance frameworks within which they are forced to operate. We even know (thanks to people like Jon Husband) that global organizations like Hay Associates have spent the better part of a century ensuring that these management science constraints are widely deployed and understood. They frame everything, not without utility, but to the exclusion of almost every other way of organizing and being together in human endeavour.
So what is the problem? Are we just lousy storytellers? Are we being deliberately marginalized? Is there something fundamentally flawed about the ability of dialogic practice to actually be of value? And how do we disrupt the standard set of management tools and the narcissism of our own communities of practice in a way that creates some serious openings for change?
What do you think?
Just finished out first day of work with Navajo Area Health Promotion practitioners and 30 community wellness workers, Elders and healers from across the Navajo Nation. We are blending an Art of Hosting workshop with content and process from some recent research in neuroscience, epigenetics, and adverse childhood experiences and with wellness and leadership models from our Navajo colleagues with whom we have a ten year relationship.
One of the pieces of work we are doing is supporting these folks in launching or accelerating some community based projects using the social networks they have in place here. This will involve us spending time in Open Space tomorrow and on Friday running a ProAction Cafe in which 6-8 projects will be able to to be developed. As a way of grounding these projects in patterns that are useful for this context we spent this afternoon generating a Navajo pattern language for resiliency. We did this with a two round World Cafe in which we asked for stories of supporting resiliency and stories of leadership challenges. This is a kind both/and appreciative inquiry. At the conclusion of the Cafe, I asked people to reflect on one teaching or piece of advice that would be useful based on the discussions. I invited them to write a word or a phrase on one side of an index card and write some explication on the back.
The attached collection of 27 or so patterns include both expected patterns such as “Presence” and “Listening.” They also include some Navajo principles like “Ádáhodí?zin” meaning ‘Letting our children go, to learn and discover who they are.”
On Friday we will use these principles to help design projects. I’ll hand a few out to each project proponent and ask them to take a few minutes to brainstorm how to incorporate these pieces of ground tested advice in their project design.
When I was up in Whitehorse last week I got to spend time with folks from the Public Service Commission discussing a project that would see us looking at discriminations in the workplace from a complexity angle. Using Cynefin and SenseMaker(tm), we hope to understand the ways in which the landscape of discrimination shifts and changes over time so that the PSC can make wiser decisions about the kinds of initiatives it sculpts. One of the problems with diversity initiatives in the public service (in any large public organization really) is the feeling that they need to be broad based and rolled out to everyone. This usually results in a single initiative that spreads across the whole organization, but except for a little awareness raising, does little to address specific instances of discrimination. Everything from awareness raising “cultural competency training” to zero tolerance accountability measures have limited effect because a) discriminatory behaviour is highly context and situation dependant and b) the public service has a permeable boundary to the outside world, meaning ideas, behaviours and people move between the two contexts all the time. The larger your organization, the more like the real world you have to be.
At any rate, I took a bit of time to do a mini-Cynefin teaching to explain how strategy works in the complex domain. and my friend Pawa Haiyupis and I added two Ps to my concentric circles of intervention in a complex system. So to review:
- Patterns: Study the patterns in a complex setting using narrative capture and sense-making. This can be done with the SenseMaker(tm) software, and it can also be done with dialogic interventions. The key thing is to let the people themselves tag their stories or at the very least have a group of people reviewing data and finding patterns together. For example, you might notice a correlation between stressful times in an organization and an increase in feelings of discriminatory behaviour
- Probe: Once you have identified some patterns, you can make some hypotheses about what might work and it’s time to develop some safe to fail probes. These aren’t meant to be successful: they are meant to tell you whether or not the patterns you are sensing have developmental potential. Failure is entirely welcome. What if we offered stress reduction activities during high stress times to help release pent up feelings? We want to be okay with te possibility that that might not work.
- Prototype: If a probe shows some promise, you might develop a prototype to develop a concept. Prototypes are designed to have tolerance for failure, in that failure helps you to iterate and improve the concept. The goal is to develop something that is working.
- Pilot: A pilot project is usually a limited time proof of concept. Roll it out over a year and see what you learn. In Pilot projects you can begin to use some summative evaluation methods to see what has changed over time. Because of their intensive resource commitment, pilot projects are hardly ever allowed to fail, making them very poor ways of learning and innovating, but very good ways to see how stable we need to make an approach.
- Project/Program/Policy: Whatever the highest level and most stable form of an initiative is, you will get to there if your pilot shows promise, and the results are clear. Work at this level will last over time, but needs regular monitoring so that an organization knows when it’s time to tinker and when it’s time to change it.
Cynefin practitioners will recognize that what I’m writing about here is the flow between the complicated and the complex domains, (captured by Dave Snowden’s Blue dynamic in this post.) My intention is to give this some language and context in service organizations, where design thinking has replaced the (in some ways more useful) intuitive planning and innovation used in non-profits and the public service.
Since October, when I first starting sketching out these ideas, I’ve learned a few things which might be helpful as you move through these circles.
- Dialogue is helpful at every scale. When you are working in a complex system, dialogue ensures that you are getting dissent, contrary views and outlying ideas into the process. Complex problems cannot be addressed well with a top-down roll out of a change initiative or highly controlled implementations of a single person’s brilliant idea. If at any point people are working on any stage of this alone, you are in danger territory and you need another pair of eyes on it at the very least.
- Evaluation is your friend and your enemy. At every stage you need to be making meaning and evaluating what is going on, but it is critically important to use the right evaluation tools. Developmental evaluation tools – with their emphasis on collective sense making, rapid feedback loops and visible organizational and personal learning – are critical in any complexity project, and they are essential in the first three stages of this process. As you move to more and more stable projects, you can use more traditional summative evaluation methods, but you must always be careful not to manage to towards targets. Such an error results in data like “We had a 62% participation rate in our diversity training” which tells you nothing about how you changed things, but can shift the project focus to trying to acheive a 75% participation rate next cycle. This is an especially pervasive metric in engagement processes. And so you must…
- Monitor, monitor, monitor. Intervening in a complex system always means acting without the certainty that what you are doing is helpful. You need data and you need it on a short term and regular basis. This can be accomplished by formal and informal ongoing conversations and story captures about what is happening in the system (are we hearing more stories like the ones we want?) or through a SenseMaker(tm) monitoring project that allows employees to end their data with a little data capture.
- These practices are nested, not linear. An always to remember that this is not a five step process to intervening in a complex system. In a large organization, you can expect all of these things to be going on all the time. Building the capacity for that is a kind of holy grail and would constitute a 21st century version of the Learning Organization in my books.