David McRaney offers a fascinating history of the Applied Mathematics Panel during World War II, which provided applied mathematical solutions to problems brought to them by military commanders. The blog post is a must read for anyone interested in the history of World War II, but it is especially useful in explaining "survivor bias" in addressing problems. McRaney uses the example of a very practical problems brought to the Panel by the Army--how to increase the survival of bombers:
The military looked at the bombers that had returned from enemy territory. They recorded where those planes had taken the most damage. Over and over again, they saw the bullet holes tended to accumulate along the wings, around the tail gunner, and down the center of the body. Wings. Body. Tail gunner. Considering this information, where would you put the extra armor? Naturally, the commanders wanted to put the thicker protection where they could clearly see the most damage, where the holes clustered. But Wald said no, that would be precisely the wrong decision. Putting the armor there wouldn’t improve their chances at all.
Do you understand why it was a foolish idea? The mistake, which Wald saw instantly, was that the holes showed where the planes were strongest. The holes showed where a bomber could be shot and still survive the flight home, Wald explained. After all, here they were, holes and all. It was the planes that weren’t there that needed extra protection, and they had needed it in places that these planes had not. The holes in the surviving planes actually revealed the locations that needed the least additional armor. Look at where the survivors are unharmed, he said, and that’s where these bombers are most vulnerable; that’s where the planes that didn’t make it back were hit.
Taking survivorship bias into account, Wald went ahead and worked out how much damage each individual part of an airplane could take before it was destroyed – engine, ailerons, pilot, stabilizers, etc. – and then through a tangle of complicated equations he showed the commanders how likely it was that the average plane would get shot in those places in any given bombing run depending on the amount of resistance it faced. Those calculations are still in use today.
. . .
Simply put, survivorship bias is your tendency to focus on survivors instead of whatever you would call a non-survivor depending on the situation. Sometimes that means you tend to focus on the living instead of the dead, or on winners instead of losers, or on successes instead of failures. In Wald’s problem, the military focused on the planes that made it home and almost made a terrible decision because they ignored the ones that got shot down.
It is easy to do. After any process that leaves behind survivors, the non-survivors are often destroyed or muted or removed from your view. If failures becomes invisible, then naturally you will pay more attention to successes. Not only do you fail to recognize that what is missing might have held important information, you fail to recognize that there is missing information at all.
Read it all here. Hat tip to Andrew Sullivan. Can you think of any contemporary military of public policy issues in which decisionmakers are displaying survivor bias?
Charles A. Blanchard
General Counsel
United States Air Force
