Menu

Evaluation and monitoring

February 16, 2015 By Art of Harvesting, Complexity, Emergence, Evaluation, Facilitation, Featured 8 Comments

sense-making

Regular readers will know that I’ve been thinking a lot about evaluation for many years now.  I am not an evaluator, but almost every project I am involved in contains some element of evaluation.  Sometimes this evaluation is well done, well thought through and effective and other times (the worst of times, more often than you think) the well thought through evaluation plan crumbles in the face of the HIPPO – the Highest Paid Person’s Opinion.  So how do we really know what is going on?

When I stumbled across Michael Quinn Patton’s work in Developmental Evaluation, a whole bunch of new doors opened up to me.  I was able to see the crude boundaries of traditional evaluation methods very clearly and was able to see that most of the work I do in the world – facilitating strategic conversations – was actually a core practice of developmental evaluation.  Crudely put, traditional “merit and worth” evaluation methods work well when you have a knowable and ordered system where the actual execution can be evaluated against a set of ideal causes that lead to an ideal state.  Did we build the bridge?  Does it work according to the specifications of the project? Was it a good use of money?  All of that can be evaluated summatively.

In the unordered systems where complexity and emergence is at play, summative evaluation cannot work at all.  The problem with complex systems is that you cannot know what set of actions will lead to the result you need to get to, so evaluating efforts against an ideal state is impossible.  Well, it’s POSSIBLE, but what happens is that the evaluator brings her judgements to the situation.  Complex problems (or more precisely, emergent problems generated from complex systems) cannot be solved, per se.  While it is possible to build a bridge, it is not possible to create a violence free society.  Violent societies are emergent.

So that’s the back story. Last December I went to London to do a deep dive into how the Cynefin framework and Cognitive Edge’s work in general can inform a more sophisticated practice of developmental evaluation.  After a few months of thinking about it and being in conversation with several Cognitive Edge practitioners including Ray MacNeil in Nova Scotia, I think that my problem is that that term “evaluation” can’t actually make the jump to understanding action in complex systems.  Ray and I agreed that Quinn Patton’s work on Developmental Evaluation is a great departure point to inviting people to leave behind what they usually think of as evaluation and to enter into the capacities that are needed in complexity.  These capacities include addressing problems obliquely rather than head on, making small safe to fail experiments, undertaking action to better understand the system rather than to effect a change, practicing true adaptive leadership which means practicing anticipatory awareness and not predictive planning, working with patterns and sense-making as you go rather than rules and accountabilities, and so on.

Last night a little twitter exchange between myself, Viv McWaters and Dave Snowden based on Dave’s recent post compelled me to explore this a bit further. What grabbed me was especially this line: “The minute we evaluate, assess, judge, interpret or whatever we start to reduce what we scan.  The more we can hold open a description the more we scan, the more possibility of seeing novel solutions or interesting features.”

What is needed in this practice is monitoring.  You need to monitor the system in all kinds of different ways and monitor yourself, because in a complex system you are part of it.  Monitoring is a fine art, and requires us to pay attention to story, patterns, finely grained events and simple numbers that are used to measure things rather than to be targets.  Monitoring temperatures helps us to understand climate change, but we don’t use temperatures as targets.  Nor should we equate large scale climate change with fine grained indicators like temperature.

Action in complex systems is a never ending art of responding to the changing context.  This requires us to be adopting more sophisticated monitoring tools and using individual and distributed cognition to make enough sense of things to move, all the while watching what happens when you do move.  It is possible to understand retrospectively what you have done, and that is fine as long as you don’t confuse what you learn by doing that with the urge to turn it into a strategic plan going forward.

What role can “evaluation” have when your learning about the past cannot be applied to the future?

For technical problems in ordered systems, evaluation is of course important and correct.  Expert judgement is required to build safe bridges, to fix broken water mains, to do the books, audit banks and get food to those who need it.  But in complex systems – economies, families, communities and democracies, I’m beginning to think that we need to stop using the word evaluation and really start adopting new language like monitoring and sense-making.

When citizens do the work

February 10, 2015 By Art of Hosting, Community, Conversation, Facilitation

The people at work

The people at work

Tonight in Vancouver I’m acting as a provocateur at an event sponsored by my friends and colleagues at Waterlution.  Water City 2040 is a ten-city scenario planning process which engages people about the future of water across 10 Canadian cities.  Tonight’s event is part of a pilot cohort to see what the process can offer to the conversation nationally.

What’s powerful about this work is that it’s citizens convening, hosting and engaging with one another.  This is not a local government engagement process or a formal consultation.  This is a non-profit organization convening deliberative conversations.  The advantage of that is that the process is free from the usual constraints that governments put on engagement.  So tonight we are thinking about possibilities that push out 25 years into the future and absolutely everything is one the table.  In fact I’m asking people to consider that in these kinds of complex systems the biggest problem you have in addressing change comes from your assumptions about what will remain the same.  It’s one thing to confront demographic, economic and environmental change, but are we also questioning things we take for granted like governance models, planning mindsets, innovation processes, value systems and infrastructure?

Organizations like Waterlution offer an unconstrained look at the future and if local governments are smart, they will pay attention to what’s happening here.  (And they are – Metro Vancouver has sent a film crew to document the evening!).

Waterlution teaches these skills to citizen practitioners, government employees and private sector staff through our Waterlution Art of Hosting Water Dialogues workshops.  We have workshops happening in April 20-22 on Bowen Island and April 27-29 near Toronto.  If this is work you want to do more of, think about joining us.  And if you contact me to inquire, you might get a little incentive…

Embedding assumptions in the question

February 9, 2015 By Art of Hosting, Collaboration, Complexity, Conversation, Design, Facilitation, Invitation, Leadership

I was working with a couple of clients recently who were trying to design powerful questions for invitations to their strategic conversations.  Both organizations are dealing with complex situations and specifically with complex changes that were overtaking their ability to respond.  Here are some of the questions that cam up:

  • How can we be more effective in accomplishing our purpose?
  • How can we create more engagement to address our outcomes?
  • What can we do to innovate regardless of our structure?
  • Help us create new ideas for executive alignment around our plan to address the change we are now seeing?

Can you see what is wrong with these questions, especially as they relate to addressing complexity?

The answer is that each of these questions contains a proposed solution to the problem, buried as assumptions in the question itself.  In these questions the answers to addressing complexity are assumed to be: sticking to purpose, creating more engagement, innovating except structurally, aligning executives around our plan.  In other contexts these may well be powerful questions: they are questions which invite execution once strategic decisions have been taken.  But in addressing complex questions, they narrow the focus too much and embed assumptions that some may actually think are the cause of their problems in the first place

The problem is that my clients were stuck arguing over the questions themselves because they couldn’t agree on solutions.  As a result they found themselves going around and around in circles.

The right question for all four of these situations is something like “What is going on?” or “How can we address the changes that are happening to us?”

You need to back up to ask that question first, before arriving at any preferred solutions.  It is very important in discerning and making sense of your context that you are able to let go of your natural inclination to want to DO something, in favour of first understanding what we have in front of us.  Seeing the situation correctly goes a long way to be able to make good strategic choices about what to do next. From there, planning, aligning, purpose and structure might be useful responses, but you don’t know that until you’ve made sense of where you are.

Pen and paper sense-making 2.0

February 5, 2015 By Art of Hosting, Complexity, Emergence, Evaluation, Facilitation, Featured, Learning, World Cafe

Two weeks ago in our Leadership 2020 program I experimented with using a signification framework to harvest a World Cafe.  We are beginning another cohort this week and so I had a chance to further refine the process and gather much more information.

We began the evening the same way, with a World Cafe aimed at exploring the shared context for the work that these folks are in.  Our cohort is made up of about 2/3rds staff from community social services agencies and 1/3 staff from the Ministry of Children and Family Development.  This time I used prepared post it notes for the sense making exercise, which you can see here:

2015-02-02 16.29.20

Our process went like this:

  1. At Cafe tables of five for 20 minutes, discuss the question “What is a story of the future you are anticipating for this sector?”
  2. Second round, new tables, same question, 20 minutes
  3. About ten minutes of hearing some random insights from the group, and checking to see how those resonate.
  4. 2 minutes of silent reflection on the question of ‘What do you need to learn here that will help us all move forward?”
  5. Each participants took a pink and blue post it note.  On the blue post it they wrote what they needed to learn that would be immediately applicable and on the red ones, learning that is needed to prepare for the future.
  6. Participants filled out the post-its and then were instructed on how to signify the data on a triangle framework that helped them signify whether what they needed to learn would help them “in their personal life,” “do their jobs” and/or “make change.”
  7. Participants also indicated on the post-its whether the worked for the Ministry or worked for a community organization.

At the conclusion of the exercise we had a tremendous amount of information to draw from.  Our immediate use was to take a small group and use affinity grouping to identify the themes that the whole has around their learning and curiosity.  We have used these themes to structure a collective story harvest exercise this morning.

But there is some much more richness that can come from this model.  Here are some of the ways people are playing with the date:

  • Removing all the pink post-its to see what the immediate learning needs are and vice versa.
  • Looking at and comparing the learning needs between the two sectors to see where the overlaps and differences are
  • examining the clusters at the extremes to see what ot tells us about personal needs, and professional needs.
  • Uncovering a theory of change by looking at the post its clustered around the “Making change” point and also seeing if these theories of change are different between the community and the government.

And of course because the data has been signified on each post it, we can recreate the framework easily.  The next level for me will be using this data to create a Cynefin framework using the four-points contextualization exercise.  Probably won’t happen in this cohort.

Big learning is the rich amount of data that proceeds from collecting finely-grained objects, allowing for disintermediated sense-making, and seeing all these multiple ways in which signified data can be used to address complex challenges obliquely, which allows you to get out of the pattern entrainment that blinds you to the weak signals and emergent patterns that are needed to develop emergent practice.  This pen and paper version is powerful on its own.  You can imagine how working with SenseMaker across multiple signification frameworks can produce patterns and results that are many magnitudes richer.

Anticipatory awareness and predictive anticipation

January 30, 2015 By Collaboration, Complexity, Evaluation, Improv, Leadership, Learning, Organization

Two Tim Merry references in a row.  Yesterday Tim posted a video blog on planning vs. preparation.  It is a useful and crude distinction about how to get ready for action in the complicated vs. complex domains of the Cynefin framework.  I left a comment there about a sports metaphor that occurred to me when Tony Quinlan was teaching us about the differences between predictive anticipation (used in the complicated domain) and anticipatory awareness (used in the complex domain).

In fact this has been the theme of several conversations today.  Complicated problems require Tim’s planning idea: technical skills and expertise, recipes and procedures and models of forecasting and backcasting using reliable data and information.  Complex problems require what Dave Snowden has named an artisian approach which is characterized by anticipatory awareness, theory and practice (praxis) and methods of what they call “side casting” which is simply treating the problem obliquely and not head on.

When I was listening to Tony teach this last month, I thought that this distinction can be crudely illustrated with the difference between playing golf and playing football (proper football, mind.  The kind where you actually use your feet.) In golf there is a defined objective and reasonably knowable context, where you can measure the distance to the hole, know your own ability with golf clubs, take weather conditions into account and plan a strategic line of attack that will get you there in the fewest strokes possible.

In football it’s completly different. The goal is the goal, or more precisely to score more goals than your opponent, but getting there requires you to have all kinds of awareness. More often than not, your best strategy might be to play the ball backwards. It may be wise to move the ball to the goal in AS MANY passes as possible, in a terribly inefficient way because doing so denies your opponent time on the ball. And the context for action is constantly changing and impossible to fully understand. And the context also adjusts as you begin to get entrained in patterns. If you stick to a long ball game, the defending team can adjust, predict your next move and foil the strategy.  You have to evolve or be owned.

This is, I believe, what drives many Americans crazy about world football. There is rarely a direct path to goal and teams can go for whole games simply holding on to the ball and then make one or two key finishing moves. Some call that boring, and it is, if you are in a culture that is about achieving the goal as quickly as possible and moving on.  And God knows we are in a culture that loves exactly that.

You plan golf holes by pre-selecting the clubs you will use in each shot and making small adjustments as you go. In football you prepare by doing drills that improve your anticipatory awareness, help you operate in space and become more and more physically fit, so that you have more physical options. You become resilient.  Yes you can scout an opponent and plan a strategy and a tactic, but football is won on the pitch and not in the strategy room. Golf is very often won in the strategy room, as long as your execution is masterful.

It’s a crude distinction and one has to be mindful all the time of downright folly of “this vs, that”, but sometimes these kinds of distinctions are useful to illustrate a point.