9. Probabilities and dependencies

When thinking about probabilities, even in a qualitative sense without numbers, it’s critical to distinguish between absolute probabilities and conditional probabilities. Even if we think in terms of whether something is plausible rather than probable, as suggested by Aubrey Clayton, plausibility in safety usually depends on context and must be considered as conditional rather than absolute. For example, an instinctive reaction may be the right response to a condition in most cases, but is there a case where it’s exactly the wrong response because of slightly different conditions?

It’s also important to remember that redundancy works differently with people than with hardware or software. One may achieve full redundancy with independent dissimilar systems, but adding people or redundant steps to a task or procedure doesn’t achieve the same effect because those additions are not fully independent.

One reason is due to the diffusion of responsibility, a term from psychology that describes the fact that when multiple people are responsible for a task, it’s tempting for each one to pay less than full attention to it because they assume someone else will catch any errors. This also applies to humans and automation: there have been many accidents in which people assumed automation would handle a situation when it couldn’t.

Trying to strengthen a task by adding steps that a single person has to perform, like checking or double-checking that something was done right, is susceptible to the same problem: the assumption that, “I’ll check it again later so I don’t need to pay too much attention to it now,” or, “I’ve already checked it so I don’t need to pay as much attention this time.” This is where expectation bias and confirmation bias come in. Adding more opportunities to catch an error quickly reaches a point of diminishing returns because expectations cause those opportunities to be coupled rather than independent.

For all these reasons, when adding people, automation, or steps as barriers to catastrophic outcomes, it should be recognized that any dependencies between them will prevent them from being fully effective.