Experiments, bets and probes

One of our mottos at PsychSafety is “everything is an experiment”. The outcome of work shouldn’t just be getting the thing done, it should be learning how to do it better next time. 

Experiments don’t mean thoughtlessly or recklessly trying anything out to see what happens. In an experiment, we try to define what, in our context, is ‘safe to fail’ and what we might learn from trying it. In other words, what can we try that has minimal risk if it doesn’t turn out as hoped, and how can we maximise the learning from it regardless of the outcome? This is also the basis for Dr Amy Edmonson’s concept of “Intelligent Failures”: “undesired results in new territory”.

Characteristics of well designed experiments, and intelligent failures, include:

• They’re carried out in new, potentially complex and unknown contexts. There’s some value in repeating experiments that have already been done, usually to confirm the results or bolster the evidence, but the real value lies in uncharted territory.
• They’re quick. Or at least, they don’t take any longer than necessary. A large experiment, like a large project, might not tell you anything useful until it concludes, which could take a while. As such, the cost of failure in long-term experiments is high, so make experiments as fast and as iterative as possible whilst maintaining the utility of the potential learning from them. 
• They’re designed with a small ‘blast radius’ of failure. If they fail, how can we limit the undesired effects of failure? This might be financial cost, reputational damage, or even increased staff turnover if we’re experimenting with new ways of working.
• They’re based upon a hypothesis. Using past experience and knowledge, we should have reason to believe that what we’re doing will result in a desired outcome. Whilst there is occasionally value in the FAFO* approach, it’s fairly rare.
• They’re strategic. Our experiments should be aligned with our greater objectives, mission, and values. 

Designing Experiments

My first job title was “experimentalist”, where I spent a lot of time designing and conducting experiments. This approach to work has stayed with me throughout my career. That said, applying the principle of “everything is an experiment” requires some effort; designing experiments takes work, and so does designing work to be experimental. But putting this effort in can save a great deal of time and return a huge amount of value, even if it feels at the start like you just want to crack on and get it done, skipping the design part almost entirely. Being “action oriented” sounds seductive and bold, but spending a little time intelligently designing that action can amplify the value from it.

Using experimentation to learn isn’t a new idea. Francis Bacon devised the Scientific (Baconian) Method in 1620, and even then it was essentially a codification of existing practice. Iterative experimentation is one of the core Agile foundations, which has now been adopted across a wide range of domains. However, experimentation is often called different things by different people. 

Alternatives to “experiments” – “bets” and “probes”

In the product development space people often speak of “bets”. In this context “bets” are intended to mean rather the same thing as experiments. What this means is that by placing a “bet”, we’re wagering that if we do X, that it will result in Y, and we can bet a certain amount of money on it and expect a return of Z. I quite like this approach as it also suggests that there’s a sensible calculation of the amount of money that we’re comfortable to invest in this bet – we’re taking into account the risk involved, the time taken to execute the bet, the potential return, and making a resultant decision about how much money we’re willing to place on that bet. However, to me there are two key problems with the term: 

1. A bet doesn’t inherently suggest any sort of learning. A bet suggests that we’re going to put some money behind a gamble, then try it and see if it works out. It doesn’t explicitly say that we’re going to learn from the outcome of it and do something differently or better next time. I’m sure in most cases, good practice means that we will try to learn from it, but it doesn’t explicitly state that, and that’s a risk. 
2. Bets are associated with gambling. And I don’t necessarily think that gambling is something positive that we should be encouraging in the workplace. We may have people on our team (and we likely won’t know who they are) who are struggling with gambling behaviour or have struggled with gambling addictions and problematic behaviour in the past, and the language around bets may be triggering or painful for them. 

For these two reasons, I tend to avoid the term “bets” in a work context.

Safe-fail Probes

The other term that comes up sometimes with this is “probes”, especially in relation to the complex state of the Cynefin Framework. Probes are analogous to experiments in the complex domain. That makes sense: we probe into a system to work out what’s going on in there, and we probe to find out more information. The Cynefin framework describes “Safe-fail Probes” as small-scale experiments, self-contained, and used in uncertain, complex contexts. I don’t really have much against the word probe, except finding it ever-so-slightly discomforting as a word as it reminds me of stories of alien abduction and the euphemistic Ford Probe, which is just a terrible name for a car. 

While bets, probes, and experiments all aim to explore and understand outcomes of work, the language we use matters. Experiments explicitly incorporate learning and design, making them a powerful tool for continuous improvement. Bets, while practical and commercial, lack an inherent focus on learning and carry problematic associations with gambling. The term Safe-fail Probes can be substituted for experiments if you wish, though for now, I’ll stick with experiments. After all, I am an Experimentalist.

Some considerations to make when working with experiments:

• Psychological safety is a prerequisite for effective experimental design and learning. People need to feel able to contribute ideas, improvements, concerns, or surface weak signals of failure.
• Design your experiments with the assumption that they will fail. Some certainly will, we just don’t know which ones yet. Assuming failure allows us to design in the maximum learning potential.
• Run different types of experiments, but don’t assume that you can’t run the same experiment more than once, at different times, in different places, or with different people. 
• Don’t assume that because an experiment failed once, it will always fail. Maybe we just needed to get better at it. Obviously trying the same thing repeatedly is, in many cases, unwise, but don’t reject something out of hand simply because it failed the first time. 
• Be wary of causing cascading failures in tightly-coupled systems. Even a small experiment could tip over a point of criticality or cause a cascade of failures, resulting in a large failure from a small intervention. Can we decouple components before conducting the experiment, to reduce the blast radius mentioned above?
• Make sure that there are actually ways to observe and determine the results of the experiment before you start it. What are you monitoring and/or measuring? Imagine it failed – what questions would you ask, and does your experiment provide that information?
• Ask diverse groups of people to contribute to the design of experiments. Multiple perspectives will help in spotting flaws and improvements that one person cannot identify on their own.
• Consider whether there are any ethical issues regarding the experiment. Could it cause some external and undesired impact? Has everyone involved in it consented to be studied? (There’s often an implicit assumption that people working in organisations have consented to their work and behaviour being studied, but they wouldn’t always agree – especially if they don’t know the experiment is taking place.)
• Do we have our OODA hat on? If our competitors see this experiment, what will it lead them to think or do? Do we want to carry out this experiment covertly, or use it as a way to sow confusion amongst our competitors? Conversely, are there organisations and people that we could partner with to maximise and share the learning from it?

We try to put this into practice in our own work, and regularly challenge ourselves to design good experiments that help us learn and work better. In the spirit of openness, here are a couple we’ve tried:

Examples of Safe to Fail experiments in our practice at psychsafety.com:

• Experimenting in working with a virtual PA service. Our hypothesis was that they’d be able to handle some of the more operational and transactional work, enabling us to focus more on higher value tasks. As it turns out, it didn’t work for us. But we learned a lot from it, and the learning from it meant we were better able to design and organise our own work as a result.
• Running pilot workshops with trusted clients. By working with our trusted clients, in relatively well-known contexts, we can run pilot workshops at low cost and risk, which generate a lot of learning and value for us as well as our clients.
• Using a co-working space as an alternative to working from home for improving productivity. We’re still experimenting with this, but for certain types of work, we’ve found it really valuable. 

With experimental approaches to work, we learn and build confidence as we go. We want to design work so that it maximises learning and minimises the impact of failure. We want to learn fast, rather than waiting a long time for a result. After we’ve carried out some smaller experiments and increased our confidence in our approach, we can carry out larger experiments and utilise that earlier learning to better design them. Everything is an experiment, even designing the experiment itself. 

*FAFO: F**k Around and Find Out

Further Reading

Safe To Fail Experiments: https://psychsafety.co.uk/safe-to-fail/

Learning From Failure: https://psychsafety.co.uk/psychological-safety-85-learning-from-incidents/

Guardrails and Failure: https://psychsafety.co.uk/psychological-safety-84-guardrails-and-failure/

Making it safe(r) to fail in teaching: https://psychsafety.co.uk/making-it-safer-to-fail-in-teaching/

John Boyd and The OODA Loop: https://psychsafety.co.uk/john-boyd-and-the-ooda-loop/

Complexity: https://psychsafety.co.uk/complexity/

Psychological Safety: Ecotones & The Edge Effect: https://psychsafety.co.uk/psychological-safety-59-ecotones-the-edge-effect/

The importance of safe-to-fail wargames: https://psychsafety.co.uk/the-importance-of-safe-to-fail-wargames/

Photo by CHUTTERSNAP on Unsplash

Psychological Safety in Practice

A slightly shorter collection this week, as we’ve been a bit busy, from Scotland at the start of the week, London in the middle, and now we’re at INSEAD in France. 

Lying

This is a great paper by Jan Hayes, Sarah Maslen and Paul Schulman, examining lying at work, and shows how lying may start almost accidentally and/or by omission, and becomes entrenched when formal and informal incentives support it.

Grenfell: Building a disaster

One of the many disasters that come up in our workshops on failure and psychological safety is the tragic Grenfell disaster of 2017, when 72 people died. Concerns were raised before the disaster, and a lack of psychological safety also impeded the response. This is an excellent series presented by Kate Lamble on Radio 4, examining the causes and effects of the disaster.

Lessons from the Columbia Space Shuttle disaster

One of the classic organisational failure examples is that of the Columbia disaster of 2003. Not least because NASA had already suffered a shuttle disaster in 1987 – Challenger. We use both examples in some of our workshops in order to help people better understand organisational dysfunction, groupthink, hierarchy, power, normalisation of deviance, production pressure and more. This is a great HBR on Leadership episode with Amy Edmondson, discussing the lessons learned from both these catastrophes:

I “…believe one of the things they may have inadvertently learned from Challenger was that launch is dangerous. And I suspect that every time there was a successful launch in the aftermath of Challenger, engineers throughout the organization, and managers, had a deep sigh of relief: “We survived the launch,” and forgot to realize there’s every bit as big a risk upon re-entry.”

This week’s poem:

The Recipe is Simple, by Rupi Kaur, in “milk and honey”

the recipe is simple
dream big
and when you think the dream is big enough
triple it in size
once you’ve visualized
the weight of what you’re capable of
stretch it past the horizon of your mind
until you see into the future
sink into what’s possible
then come back and
get to work