Nate Silver, The Signal and the Noise: Why So Many Predictions Fail–but Some Don’t, 2015, pp. 326-7
Beyond Results-Oriented Thinking
In the United States, we live in a very results-oriented society. If someone is rich or famous or beautiful, we tend to think they deserve to be those things. Often, in fact, these factors are self-reinforcing: making money begets more opportunities to make money; being famous provides someone with more ways to leverage their celebrity; standards of beauty may change with the look of a Hollywood starlet.
…As an empirical matter…success is determined by some combination of hard work, natural talent, and a person’s opportunities and environment–in other words, some combination of noise and signal. In the U.S., we tend to emphasize the signal component most of the time–except perhaps when it comes to our own shortcomings, which we tend to attribute to bad luck. We can account for our neighbors’ success by the size of their home, but we don’t know as much about the hurdles they had to overcome to get there.
When it comes to prediction, we’re really results-oriented. The investor who calls the stock market bottom is heralded as a genius, even if he had some buggy statistical model that just happened to get it right. The general manager who builds a team that wins the World Series is assumed to be better than his peers, even if, when you examine his track record, the team succeeded despite the moves he made rather than because of them…
Sometimes we take consideration of luck too far in the other direction, when we excuse predictions that really were bad by claiming they were unlucky…as a default, just as we perceive more signal than there really is when we make predictions, we also tend to attribute more skill than is warranted to successful predictions when we assess them later.
Part of the solution is to apply more rigor in how we evaluate predictions. The question of how skillful a forecast is can often be addressed through empirical methods; the long run is achieved more quickly in some fields than in others. But another part of the solution–and sometimes the only solution when the data is very noisy–is to focus more on process than on results…