We are embarking on a innovative approach to a social problem and we need a framework to guide the evaluation process. As it is a complex challenge, we’re beginning with a developmental evaluation framework. To begin creating that,I was at work for most of the morning putting together a meta-framework, consisting of questions our core team needs to answer. In Art of Hosting terms, we might call this a harvesting plan.
For me, when working in the space of developmental evaluation, Michael Quinn Patton is the guy whose work guides mine. This morning I used his eight principles to fashion some questions and conversation invitations for our core team. The eight principles are:
- Developmental purpose
- Evaluation rigor
- Utilization focus
- Innovation niche
- Complexity perspective
- Systems thinking
- Co-creation
- Timely feedback
The first four of these are critical and the second four are kind of corollaries to the first and the first two are essential.
I think in the Art of Hosting and Art of Harvesting communities we get the first principle quite well, that participatory initiatives are, by their nature, developmental. They evolve and change and engage emergence. What I don’t see a lot of however is good rigour around the harvesting and evaluation.
All conversations produce data. Hosts and harvesters make decisions and choices about the kind of data to take away from hosted conversations. Worse, we sometimes DON’T make those decisions and then we end up with a mess, and nothing useful or reliable as a result of our work.
I was remembering a poorly facilitated session I once saw where the facilitator asked for brainstormed approaches to a problem. He wrote them in a list on a flip chart. When there were no more ideas, he started at the top and asked people to develop a plan for each one.
The problems with this approach are obvious. Not al ideas are equal, not all are practical. “Solve homlessness” is not on the same scale as “provide clothing bundles.” No one would seriously believe that this is an effective way to make a plan or address an issue.
You have to ask why things matter. When you are collecting data, why are you collecting that data and how are you collecting it? What is it being used for? Is it a reliable data source? What is your theoretical basis for choosing to work with this data versus other kinds of data?
I find that we do not do that enough in the art of hosting community. Harvesting is given very little thought other than “what am I going to do with all these flipcharts?” at which point it is too late. Evaluation (and harvesting) rigour is a design consideration. If you are not rigourous in your data collection and your harvesting methods, others can quite rightly challenge your conclusions. If you cannot show that the data you have collected is coherent with a strategic approach to the problem you are addressing, you shouldn’t be surprised if your initiative sputters.
In my meta-framework the simple questions I am using are:
- What are our data collection methods?
- What is the theoretical basis and coherence for them?
That is enough to begin the conversation. Answering these has a major impact on what we are hosting.
I high recommend Quinn Patton et. al.’s book Developmental Evaluation Exemplars for a grounded set of principles and some cases. Get rigourous.
“Vision” is one of those words that is overused in our work and the reason it is so elusive is that is is so context dependant.
You can have a vision of a full bath tub of steaming hot water. You can have a vision of making your home run on rain water alone. You can have a vision of safe drinking water for all humans.
The first is simple, short term and you have all the tools and abilities to make it happen.
The second is more complicated and you require a few experts to make it happen, but with the right people and resources, you can achieve it.
The third is not up to you. It is a complex and adaptive system. You may be motivated by a desire to see safe drinking water for all humans but you are unlikely to achieve it because it is a complex problem. Intention can make a difference here and instead of working TOWARDS a tangible vision you can work FROM an intention and guide your actions against that.
The problem comes when people want tangible outcomes from linear processes. “We need a vision of our future” can sometimes lead to work that ignores all the opportunities and threats that come up in a living and evolving system. Without good methods of understanding what is happening, what a system is inclined to do, or iterating work based on learning (in other words developmental evaluation), in my experience those with power and a mandate to accomplish something will eventually narrow the work down to mere deliverables. The vision maybe in there somewhere but the context renders it useless.
So these days when a client asks me for a vision I want to know why and whether they have the means and desire to actually achieve it, or whether they are simply calling for a conversation on “what we’re all trying to do” so that work and opportunities can be evaluated against that.
At some level, in complex systems, vision and purpose become moral centres and ethical guidelines and not targets. That seems important to me.
I think that doing strategic work with organizations and communities is really about learning. If a group is trying to confront newness and changes in its environment and needs to come up with new strategies to address those changes, then it needs to learn.
I love the term “desire lines.” Most of my initial work with organizations tries to get at the desire lines in the organization; the patterns embedded in the culture that help or hinder change and resilience. Naming and making visible these entrained desire lines (including the ones that that group takes into the darkness of conflict and unresourcefulness) is a helpful exercise in beginning to first reflect and then disrupt and develop capacity. When a group can see their patterns, and see which are helpful and which are not, they can make the choice to develop new ones or strengthen the stuff that works.
When problems are complex, then the people in the group need to focus on learning strategies in order to discover and try new things, rather than adopt a best practice from elsewhere. It is, as Steve Wheeler says in this video, the difference between designed environments and personal choice:
“Students will always find their own unique pathways for learning. They will always choose their own personal tools and technologies. Our job is not to try and create pathways for them, but to help them create the pathways for themselves and the scaffold and support them as they go through those pathways.”
Hosting groups is always about learning – in fact one core question of the Art of Hosting community is “what if learning was the form of leadership required now?” To support learning, help groups find the desire lines for learning and good strategic work to address change that is owned by the group will follow. That is how learning builds capacity and capacity builds sustainability.
Reflecting these days on some two day courses I have coming up, including one on complexity and social change, one on invitation practice and one on Open Space.
Each of these courses is workshop to introduce people to a practice or a set of practices, as opposed to techniques and skills. In each of these workshops people will come away with an ability to go into the practice, literally as artists. These are not technical trainings designed to download procedures and methods. They are courses that will leave you ready to practice, ready to make mistakes and learn as you go, and ready to improve.
It’s always hard to explain to people when they come on these courses that they will not leave as competent practitioners of the stuff they are learning. All artists make mistakes when they are first using a tool. What’s most important is that you have a way of developing your mastery with a tool, which is to say that you have a framework that helps you understand what you are doing and how well your are doing it. In traditional settings, mentorship is an important piece of this, to help one develop mastery from every attempt as you learn. The point is that these kinds of tools are useful in complexity, meaning that they are context and practitioner dependant. How you use these tools and where you use them matters.
Teaching, therefore, requires a disruption to the pedagogy of filling another person’s brain and body with competence. In my courses, my favourite answer to questions about application is “it depends.” But what doesn’t change over time is the body of theory that needs to inform one’s practice.
Theory is the constant, and therefore a heuristic (a basic set of measurable principles) are they way to develop practice that is appropriate in context. By theory I mean a serious understanding from natural sciences that underpin the ways systems work.
Courses that are pure theory are generally not helpful without grounding them in practice, and courses that are just collections of tools and practices are somewhat useful but can lead practitioners astray if they don;t understand why things work (or they aren’t able to see why things aren’t likely to work). So my basic approach to teaching these kinds of things is to use the following heuristic:
- Theory
- Framework
- Practice examples
- Application
Teaching theory – in my case usually complexity theory – is critical for setting the groundwork for the practices that follow. If you don’t understand the nature of the context you are working in, you are likely to make serious errors in applying practices: linear problem solving doesn’t work in non-linear settings. That seem intuitive but you need to know why and be able to explain it.
Frameworks are helpful because they provide touchstones to connect theory to practice. When we were teaching the harvesting course last year, we came up with the mnemonic PLUME to describe five heuristics that help practitioners design methods that are coherent with good theory. (We have a new one for the invitation course by the way: VALUE. You can learn more about it on the course or in the blog posts that come as a result of the teaching). Sometimes that framework is Cynefin, sometimes it’s the chaordic path.
The important thing about a framework is that it helps you to create something and then it can fall away and what you have created can stand on its own. If your practice relies on maintaining the integrity of the framework then your framework isn’t effective. This is an issue I see sometimes with things like sociocracy where in poor application it’s important that people retain accountability to the framework (but not even necessarily the theory). Frameworks should be important enough containers to inspire grounded and coherent action, but not so critical that the action depends on the framework.
Dave Snowden uses the metaphor of the scaffold, which is useful. Build a scaffold to build your house. But if the scaffolding is a part of your house and your house depends on the scaffolding for it’s structural integrity, you haven’t succeeded
Once we are grounded in theory and have a way of carrying it with us, we can share practices that help practitioners to ground this in real life. I always combine this with an opportunity to apply the learning on real projects. This gives people an opportunity to work together to make sense of what they are learning. It means that folks working on projects get a variety of perspectives from people who have just learned something, including naive and oblique perspectives, which is good when you are trying to do new things. For those that are giving their help with projects, they learn a lot by stepping into the coach or critic role, as they are forced to think about what they have been learning in an application context.
So that’s my basic pedagogy these days. I’ve been on a few facilitation workshops over the years and been shocked at two things: the lack of theory (so how do I know how your methods work) and the over reliance on tips and tricks, which is basically a kind of addictive mechanism for people learning facilitation. many people are super-interested in adding a few things to their tool box, and while I love helping people add tools, I would never give an apprentice carver a knife without helping them understand why this thing works and what happens if you use it incorrectly. And I would never say “here’s a knife, now go make your masterpiece.” Their first effort is going to be terrible, and that’s what practice is. We need more folks teaching the art of facilitation as artists teaching artists and less shady selling of recipes and tools for guaranteed success.
My friends over at the Social Labs Revolution website have been fielding questions about the prototyping phase of labwork and today published a nice compilation of prototyping resources. It’s worth a visit. It got me thinking this morning about some of the tools I use for planning these days.