I’ve been working in the world of program development with a lot of complexity and innovation and co-creation lately and have seen these three terms used sometimes interchangeably to describe a strategic move. As a result, I’ve been adopting a more disciplined approach to these three kinds of activities.
First some definitions.
Taken explicitly from Cynefin, a probe is an activity that teaches you about the context that you are working with. The actual outcome of the probe doesn’t matter much because the point is to create an intervention of some kind and see how your context responds. You learn about the context and that helps you make better bets as you move forward – “more stories like this, less stories like this” to quote Dave Snowden. Probes are small, safe to fail and easily observed. They help to test different and conflicting hypotheses about the context. If 8 out of 10 of your probes are not failing, you aren’t learning much about the limits of your context. Probes are actually methods of developmental evaluation.
A prototype is an activity that is designed to give you an idea of how a concept might work in reality. Prototypes are designs that are implemented for a short time, adjusted through a few iterations and improved upon. The purpose of a prototype is to put something into play and look at its performance. You need to have some success with a prototype in order to know what parts of it are worth building upon. Prototypes straddle the world of “safe to fail” and fail safe. They are both developmental evaluations tools and they also require some level of summative evaluation in order to be fully understood. Prototypes are also probes, and you can learn a lot about the system from how they work.
A pilot is a project designed to prove the worthiness of an approach or a solution. You need it to have an actual positive effect in its outcomes, and it’s less safe to fail. Pilots are often designed to achieve success, which is a good approach if you have studied the context with a set of probes and maybe prototyped an approach or two. Without good intelligence about the context you are working with, pilots are often shown to work by manipulating the results. A pilot project will run for a discrete amount of time and will then be summatively evaluated in order to determine its efficacy. If it shows promise, it may be repeated, although there is always a danger of creating a “best practice” that does not translate across different contexts. If a pilot project is done well and works, it should be integrated with the basic operating procedure of an organization, and tinkered with over time, until it starts showing signs of weakened effectiveness. From then on, it can become a program. And pilots are alos probes, and as you work with them they too will tell you a lot about what is possible in the system.
The distinctions between these three things are quite important. Often change is championed in the non-profit word with the funding of pilot projects, the design of which is based on hunches and guesses about what works, or worse, a set of social science research data that is merely one of many possible hypotheses, privileged only by the intensity of effort that went into the study. We see this all the time with needs assessments, gap analyses and SWOT-type environmental scans.
Rather than thinking of these as gradients on a line though, I have been thinking of them as a nested set of circles:
Each one contains elements of the one within it. Developing one will be better if have based your development on the levels below it. When you are confronted with complexity and several different ideas of how to move forward, run a set of probes to explore those ideas. When you have an informed hunch, start prototyping to see what you can learn about interventions. What you learn from those can be put to use as pilots to eventually become standard programs.
By far, the most important mindshift in this whole area is adopting the right thinking about probes. Because pilot projects and even prototyping is common in the social development world, we tend to rely on these methods as ways of innovating. And we tend to design them from an outcomes basis, looking to game the results towards positive outcomes. I have seen very few pilot projects “fail” even if they have not been renewed or funded. Working with probes turns this approach inside out. We seek to explore failure so we can learn about the tolerances and the landscape of the system we are working in. We “probe” around these fail points to see what we can learn about the context of our work. When we learn something positive we design things to take advantage of this moment. We deliberately do things to test hypotheses and, if you’re really good and you are in a safe-to-fail position, you can even try to create failures to see how they work. That way you can identify weak signals of failure and notice them when you see them so that when you come to design prototypes and pilots, you “know when to hold ‘em and know when to fold ‘em.”
One of my favourite concepts from the complexity world is the fallacy of thinking that comes from the truth of retrospective coherence. The mistake is that, because we can look back in time to understand causes of our current condition, we can therefore see forward in time and affect the causes of a future condition. Complex systems are emergent, so we can never be sure what the future holds, regardless of how well we can trace how we got here.
Despite the fact that it is illegal to sell an investment instrument without the warning that “past performance does not guarantee future results” falling for the trap that retrospective coherence gives you a reliable path forward is basically a feature of doing any strategic work at all. It leads to planning that puts out a future preferred state and then backcasts a set of steps that, if we follow them, will take us there or nearly there.
So there are all kinds of issues with this, and the Cynefin framework’s greatest gift is that it helps us create strategy to avoid to pitfall of retrospective coherence.
Today though, a surprise in my morning reading. A lovely article on Robert Frost’s “The Road Not Taken.” We all think we know what that poem is about: about the adventure that will ensue if we just take the less beaten path. But you might be surprised to learn that the poem is actually about retrospective coherence and not adventures strategic planning (emphasis mine):
Most readers consider “The Road Not Taken” to be a paean to triumphant self-assertion (“I took the one less traveled by”), but the literal meaning of the poem’s own lines seems completely at odds with this interpretation. The poem’s speaker tells us he “shall be telling,” at some point in the future, of how he took the road less traveled by, yet he has already admitted that the two paths “equally lay / In leaves” and “the passing there / Had worn them really about the same.” So the road he will later call less traveled is actually the road equally traveled. The two roads are interchangeable.According to this reading, then, the speaker will be claiming “ages and ages hence” that his decision made “all the difference” only because this is the kind of claim we make when we want to comfort or blame ourselves by assuming that our current position is the product of our own choices (as opposed to what was chosen for us or allotted to us by chance). The poem isn’t a salute to can-do individualism; it’s a commentary on the self-deception we practice when constructing the story of our own lives. “The Road Not Taken” may be, as the critic Frank Lentricchia memorably put it, “the best example in all of American poetry of a wolf in sheep’s clothing.” But we could go further: It may be the best example in all of American culture of a wolf in sheep’s clothing.
Brilliant.
“We shall not cease from exploration, and the end of all our exploring will be to arrive where we started and know the place for the first time.”
— TS Eliot
Our Beyond the Basics team is about to host our last gathering of the current cycle of offers, back in North America. Over the past five Beyond the Basics offerings I have learned more than I feel like I’ve shared. I can feel that my practice has changed as a result of doing this work, and I’ve become interested in the way our team’s ideas and lessons from working at scale have begun to outline a form and practice of leadership that is needed in much of our work now.
I was back at St. Aidan’s United Church in Victoria yesterday, hosting another conversation in their continued evolution into their next shape. Last December we worked together to explore four possible scenarios that were being proposed for the congregation. In the past few months they have been working on implementing one of these scenarios – the one which featured a plan to develop a Spiritual Learning Centre. Yesterday was a short strategic conversation called to explore the shape of what that Centre could be and how it will change life at the church.
Recently in BC, we have a had a child die in the care of the state. This does happen from time to time, and when it does a process is triggered whereby the Representative for Children and Youth lanuches an investigation and makes recommendations which usually result in more rules and procedures to govern the child welfare system with the express purpose of never having it happen again.
I work closely with child protection social workers in BC and there is not a single one I know of whose heart does not break when something like this happens. Everyone wears the failure. Social work is difficult not because of the kinds of predictable situations that can be mitigated but because of the ones no one saw coming. The Ministry of Children and Family Development operates under a massive set of procedures and standards about social work practice. But no amount of rules will prevent every case of child death. Just like no amount of rules will eliminate every case of discrimination, every war, every instance of every bad thing that happens to humans.