We are embarking on a innovative approach to a social problem and we need a framework to guide the evaluation process. As it is a complex challenge, we’re beginning with a developmental evaluation framework. To begin creating that,I was at work for most of the morning putting together a meta-framework, consisting of questions our core team needs to answer. In Art of Hosting terms, we might call this a harvesting plan.
For me, when working in the space of developmental evaluation, Michael Quinn Patton is the guy whose work guides mine. This morning I used his eight principles to fashion some questions and conversation invitations for our core team. The eight principles are:
- Developmental purpose
- Evaluation rigor
- Utilization focus
- Innovation niche
- Complexity perspective
- Systems thinking
- Co-creation
- Timely feedback
The first four of these are critical and the second four are kind of corollaries to the first and the first two are essential.
I think in the Art of Hosting and Art of Harvesting communities we get the first principle quite well, that participatory initiatives are, by their nature, developmental. They evolve and change and engage emergence. What I don’t see a lot of however is good rigour around the harvesting and evaluation.
All conversations produce data. Hosts and harvesters make decisions and choices about the kind of data to take away from hosted conversations. Worse, we sometimes DON’T make those decisions and then we end up with a mess, and nothing useful or reliable as a result of our work.
I was remembering a poorly facilitated session I once saw where the facilitator asked for brainstormed approaches to a problem. He wrote them in a list on a flip chart. When there were no more ideas, he started at the top and asked people to develop a plan for each one.
The problems with this approach are obvious. Not al ideas are equal, not all are practical. “Solve homlessness” is not on the same scale as “provide clothing bundles.” No one would seriously believe that this is an effective way to make a plan or address an issue.
You have to ask why things matter. When you are collecting data, why are you collecting that data and how are you collecting it? What is it being used for? Is it a reliable data source? What is your theoretical basis for choosing to work with this data versus other kinds of data?
I find that we do not do that enough in the art of hosting community. Harvesting is given very little thought other than “what am I going to do with all these flipcharts?” at which point it is too late. Evaluation (and harvesting) rigour is a design consideration. If you are not rigourous in your data collection and your harvesting methods, others can quite rightly challenge your conclusions. If you cannot show that the data you have collected is coherent with a strategic approach to the problem you are addressing, you shouldn’t be surprised if your initiative sputters.
In my meta-framework the simple questions I am using are:
- What are our data collection methods?
- What is the theoretical basis and coherence for them?
That is enough to begin the conversation. Answering these has a major impact on what we are hosting.
I high recommend Quinn Patton et. al.’s book Developmental Evaluation Exemplars for a grounded set of principles and some cases. Get rigourous.
I’ve been for a beautiful walk this morning in the warm mist of a spring day in the highlands near Victoria. It was quiet but for the cacophony of bird song, and everything was wet with mist and dew. This is the greenest time of year on the west coast, and the mossy outcroppings and forest floor were verdant.
There is a beauty in what is, in any given moment.
I’ve been thinking about this as I have been struggling with watching people be evaluated in their work recently. My daughter is a jazz musician, training her art in a university program where she is judged on her performance and where that number assigned to that moment in time affects much in her life. My son laid out the papers he has been graded on, showing me a variety of marks that surprised him and made him proud of what he had accomplished. All of it a shallow judgement applied to a limited action in a tiny slice of time. Do these numbers take into consideration my daughter’s love of jazz or my son’s pride in the story he wrote or his ability to solve quadratic equations? Do they take into account how my kids approached this test, what it meant to them, what they were trying to do? How do these numbers track their changes, their growth, the affect that they are having on the world around them?
The evaluator’s job comes with enormous privilege. The privilege is in determining the frame within which the noticing takes place. Poorly done evaluation happens when an evaluator reduces a complex outcome like “impact” into a few arbitrary indicators developed in isolation with a poorly articulated rationale and coherence with what is happening. When an evaluator walks into a process it is amazing how much gravity also enters the work.
At some point in our culture – and maybe it was always thus – evaluation became something of an investigation used to justify accountability pursued with a particular agenda in mind. Frameworks became both too narrow and too fuzzy. I have been in processes where evaluators wanted a single number on a scale from 1-5 to rate the effectiveness of an experience. And I have been in processes where evaluators are seeking to measure “impact” without every defining it, or only defining it on how a process has advanced their client’s singular needs and not the need of the whole ecosystem. I have never seen an evaluation that says to a client “these people are discovering some stuff that has nothing to do with what you funded them for, and therefore your assumptions about change are wrong.”
Done well however, evaluation contributes a tremendous amount of knowledge, awareness and confidence to a process. It allows us to make sense of our work, it opens our eyes to different questions we should be asking and it can put the tools of meaning making in the hands of people doing work. In complex environments, it can give us a new set of senses that help us see and hear and feel what is happening, and that open up promising new directions to nudge an effort.
When evaluation is part of the work it makes a huge difference. When evaluation is a separate project, laid on top of the work or done at a distance, it can bring the work to a standstill as everyone organizes around what the evaluator is looking for instead of where the project is at in its evolution or what the needs are.
Evaluations conducted with principles such as these ones are amazingly useful and empowering. They are deeply powerful influencers in the life of a project, and they need to be done with intense awareness of this power. We need to demand from our clients and funders and stakeholders, a more sophisticated standard of engagement around evaluation, and we need to hold evaluators to these principles too.
There is tremendous beauty in the moments of people working together, learning, creating, trying to improve the lives of others. Some days are rich with green and lush life and others are despairing failures. I would love to read an evaluation report that is as rich as Thoreau’s observations of life at Walden capturing the changes and the beauty, witnessing the growth all around, understanding its meaning and being open to the surprises that come with being immersed in an experience.
I’ll be writing more about this topic in the next little while. What are your longings for or experiences of great evaluation?
Last weekend I took a ramble across Bowen Island, where I live, with a friend and colleague, Annemarie Travers. Annemarie and I have been teaching the Leadership 2020 program for a number of years now and we both love walking: she on the long pilgrimages of the Camino and Shikoku and me in the mountains of southern British Columbia. We are also both interested in managing in complexity.
My friends over at the Social Labs Revolution website have been fielding questions about the prototyping phase of labwork and today published a nice compilation of prototyping resources. It’s worth a visit. It got me thinking this morning about some of the tools I use for planning these days.
As Bronagh Gallagher and I have been musing about our offering on complexity, facilitation and social justice, we have been discussing the shift in activism from ideology to evolutionary. Ideological movements try to coalesce activities and people along a line towards a fixed end state. Evolutionary movements start with intentions, principles and move outward in multiple directions along vectors. They adjust and learn as they go, and they both respond to and change their context.
This nice post from Network Centered Advocacy capgtues what I’m talking about by first looking at how a lacrosse player’s artistry evolves in changing contexts and then concludes with these important paragraphs:
Being labeled a “movement” is a reflection of evolutionary status. One person or organization does not qualify as a movement, yet there is no set size of a movement. Movements are messy, complex and organic. The movement label is shorthand, an inclusive term of many independent leaders and supporters, their support structures, all that they can tap into, as well as their capacity to disagree as often as they align on work.
Movements are a reflection of self-directed, adaptive, resilient, self-sacrificing, supported and persistent initiatives to work on complex problems. There are no movement structures, but instead a movement is a mass migration of people, organizations, businesses and communities unified in common story, driving to shift culture, policy, behavior and norms. Successful movements build and transform the landscape as they progress providing a base for further progress. A quick scan of the first few pages of google news for” movements” produces a snapshot of the current movements that come to mind, including the movement against fracking, the climate change movement, the tea party movement, Occupy, #blacklivesmatter, the anti-austerity movement, the dump-Trump movement, the maker-movement, the LGBTQ movement–the list goes on.
A key evolution point in a movement’s trajectory is the transition away from any single point of failure, to be loosely structured and resilient enough to absorb setbacks. The agility and adaptive characteristics of movements are fueled not only by personal stakes, individualism, driven leadership, passion and local control, but also by unpredictable solidarity and a distributed organizing approach that resists centralization. The difference between an organization, coalition, centralized campaign and a genuine movement is the way each fuels smart local initiatives and the ways leaders align power.
Building a movement is actually more aptly perceived as unleashing a movement, creating new spaces that help the movement surge in wider, expansive and still supportive directions. As a movement gains organizing momentum, strategies shift to broadly unfold and push a wide set of actions that draw opposition thin rather than clustering and making defense easy. This distributed layout requires a shift in thinking and strategy.
The key thing to notice here is that culture is changed by evolving movements, not linear programs. Movements are not led TOWARDS a goal, but rather emanate from a set of connected and coherent stories, actions and intentions, and self-correct, fail and adapt as they go. This is true whether the venue of action is organizational or societal. Cultures are complex and require complexity to change them. Diving more into the examples given in the quote will give you more insight into how these movements have become a part of, and transformative agents within, the cultures they are aiming to change.