In the early post-pandemic era, I was sitting in the synagogue (shul) one Saturday morning, at least 2 seats away from anyone else, my face covered by a mask. A hint of normalcy was beginning to return, and the Rabbi posed a question to the small number of congregants seated in front of him: Would scrapping the mask requirement bring more people back to shul? A hearty debate followed, one that I thought was fairly pointless and a clear case of survivorship bias.

What exactly is survivorship bias?

The attached image has been “rolled out” every time the concept of survivorship bias comes up, and for good reason; it is a great example. So, I won’t stray from the norm and will retell the story of Abraham Wald.

In World War II, bullet-riddled Allied planes were returning to base after encountering enemy anti-aircraft fire. Ground crews plotted the position of the bullet holes in order to determine areas of the heaviest damage (as per the image). The thinking was that if these areas could be reinforced, the planes and pilots would have a better chance of survival.

Then along came Abraham Wald, a Hungarian mathematician, who pointed out the flaw in this thinking. Wald assumed that damage must be more uniformly distributed and that the aircraft that did return or show up in the samples were hit in the less vulnerable parts.

Wald noted that the study only considered the aircraft that had survived their missions; the bombers that had been shot down were not present for the damage assessment. The holes in the returning aircraft, then, represented areas where a bomber could take damage and still return home safely.

This concept of considering only the successful observations in the dataset while ignoring the failures is known as survivorship bias.  

It occurs frequently. It is particularly prevalent in the ‘Self-Help’ section, where well-known “gurus” tell us which 7 habits we should follow to be “successful”, without researching the millions of people on the planet who also follow those habits but aren’t “successful”.

[Aside – There is a highly entertaining Mark Manson video discussing a similar theme]

Or when an eccentric billionaire tech entrepreneur, addresses a graduation ceremony and tells the graduating cohort they would have been better off not attending university or dropping out, like him and several other high-profile entrepreneurs. What the tech billionaire fails to include in his sample are the millions of non-graduates currently cleaning offices, collecting garbage, offering us fries with our meals, driving us from point A to point B, or labouring on building sites.

It is easy, and often very tempting, to find a particular trait in a small group and assume that its existence implies causality. But without widening the sample, there is very little to suggest it is true.

To go back to the Rabbi’s question, what made the debate moot? He was asking the “survivors”.