1. Introduction

I spent the last several years of my career in aviation human factors trying to answer the following questions: how can we better anticipate, analyze, and assess human-related risks when there are too little data for traditional statistical analysis? When we learn of a serious incident involving a surprising human behavior, how do we determine whether it presents a potential risk to safety?

By human-related risks, I mean unexpected human behaviors that either create hazards or fail to mitigate hazards that have otherwise emerged. For example, pilots are supposed to be trained to mitigate a wide range of hazards, including problems with the airplane or environmental threats. But sometimes, they may act in unexpected ways due to a variety of factors:

• an instinctive reaction may hijack the behavior away from the expected trained response;
• a key training or skill gap may prevent the pilot from responding as intended by the designers;
• an erroneous belief or assumption could cause a pilot to misinterpret the situation.

Any organization that designs, builds, or operates complex systems is susceptible to expecting the people on the front lines of using or operating them to do so as the designers intended. But incidents and accidents in all domains occur when people act in unexpected ways. When we see this occur in an incident, how do we evaluate a newly recognized risk to decide if it poses a potential safety issue? There are lots of methods and tools available to help with this, but sometimes they can mislead rather than inform.

The European Union Aviation Safety Agency (EASA) recognizes the importance of assumptions and evidence that may indicate deviations from them in a safety bulletin to airlines, which says in part: "To enable the in-depth analysis of in-service events involving human interventions, the assumptions, which have been made by the DAH (Design Assurance Holder, aka the manufacturer) when demonstrating compliance with the certification basis about the expected flight crew behaviour, need to be known in order to identify any deviations from these assumptions in the context of operation." The bulletin goes on to recommend that airlines report any unexpected pilot behaviors that may lead to hazards to the manufacturer because they can't fix what they don't know about.

This is a critical part of improving aviation safety because, in the vast majority of recent accidents, unexpected pilot behaviors were either causal or contributing.

So how do you know when a surprising behavior in an incident could potentially lead to an accident someday? How do you associate it with a safety threshold? In order to understand this better, I read a number of books on risk, probability, and prediction. Why prediction? Because any decision about whether a system or condition should be considered safe or potentially unsafe is a prediction. When a product is certified or approved, everything from airplanes to drugs to medical equipment to cars to power plants and beyond, that approval is itself a prediction that the product should be considered safe. So the literature on how people make predictions and what kinds of biases can affect their accuracy is relevant to assessing safety risks.

The books I read include:

• The Incerto series (Fooled by Randomness, The Black Swan, Antifragile, and Skin in the Game by Nassim Nicholas Taleb)
• Bernoulli’s Fallacy by Aubrey Clayton
• Expert Political Judgment and Super Forecasting by Philip Tetlock
• The Signal and the Noise and On the Edge by Nate Silver
• How to Measure Anything by Douglas Hubbard
• The Yellow Pad by Robert Rubin

Anyone who has read these books could come to the same conclusions I have. But since these books aren’t intended to specifically relate to safety, and I haven’t seen anyone else make the (I think valuable) connections to it, I do so here.

These essays relate to domains, particularly aviation, where serious incidents are rare, catastrophic events are exceedingly rare, and the standard for safety is exceedingly high. In aviation, that standard is typically 1E-9 (or one in a billion chance of a catastrophic outcome), and the known incidents that may be precursors to such an event are usually in the single digits if they exist at all. With such little statistical power and such a high standard of proof, other methods are needed. And since the human contribution to a serious event is often complex and unpredictable, this is where such methods are needed most.

I will not be referring to any incidents or information from my prior employers, so this will be a general philosophical treatment with no specific examples. However, I expect that an informed reader will think of their own examples.

The contents of this website can be downloaded as a pdf document.