The quality of forecasting is essential to any decision making process. Phil Tetlock, an eminent professor at Wharton, argues that forecasting is a skill which can be learned and mastered.
Various experiments conducted by Mr. Tetlock and his team over the years (including the Good Judgment Project, a forecasting tournament with a focus on geopolitics) provide evidence that non-experts beat subject-matter experts at predicting the future. In fact, a number of “superforecasters” could be identified within the non-expert population.
What are the qualities of these super-humans? In their style of thinking, they behave like “foxes” as opposed to “hedgehogs” (based on a definition by philosopher Isaiah Berlin well covered in an WSJ article available here):
As part of the forecasting process, hedgehogs, who tend to have deep domain expertise, rely upon a single central vision to explain the world’s functioning and direction of travel. They tend to simplify complex matters, with a certain degree of confidence. Their propensity to dogmatism creates a bias which affects their ability to predict the right outcome. Hedgehogs have been proven to think that they know more than what they actually do
Foxes operate differently—in a way which can be acquired through some learning and practice. They use diverse sources of information and analytical models. They are intellectually humble but are comfortable with complexity. They are “actively open-minded”, i.e. seeking out evidence and opinions that go against theirs through debates. Foxes behave in a non-ideological, pragmatic and flexible manner in the sense that they are ready to update beliefs in response to new evidence.
Despite a knowledge disadvantage, foxes consistently win when it comes to forecasting. But hedgehogs can show decisiveness, conviction and tend to be more popular, including in the media. Their prominence is due to a tendency to call for extreme outcomes with grandiloquence; and/or to the skillful avoidance of precise forecasting (incl. timing) despite appearances; and/or to the lack of post-event check and thus accountability. The result: a demonstrable inverse correlation between fame and prediction accuracy amongst pundits.
In the comedy series “How I met your Mother”, breakout character Barney Stinson abuses the system in a majestic way. He invents stats or probabilities (typically set at 83%) to add credibility to outrageous forecasts. No one ever tracks his predictions. He remains unchallenged.
Since none of this is helpful, P. Tetlock suggests that forecasters be required to come up with precise predictions (probability and timing) with a post outcome quality testing system. He argues that by doing so, hedgehogs would have a clear incentive to behave like foxes, including by consciously putting aside any cognitive pitfalls. As a positive by-product, extreme scenarios would become significantly scarcer.
Realistically, this will unlikely be enforced in the media. But Mr. Tetlock’s approach can provide an avenue for improvement for bank research (forecasting training, move from single point to scenario-based recommendations, enhanced accountability).
In the boardrooms, stricter discipline about monitoring the quality of assumptions, say, about any given market development, may help reduce the gap between “conservatives” and “expansionists” through the exclusion of extreme scenarios. This depolarization could facilitate the decision making process and potentially improve its quality.
Barney Stinson, we’re coming after you and your non-sense!
Sources:
Question Certainty, W Frick, Harvard Business Review, Oct 2015
The Power of Precise Predictions, P Tetlock & P Scoblic, The New York Times, Oct 2, 2015
Sharpening Your Forecasting Skills, M Mauboussin, Valuewalk, Sep 30, 2015
What is the Best Way Predict the Future, A Wileman, Management Today, Sep 29, 2015
The Fox vs the Hedgehog Forecaster, P Tetlock, History Squared, May 31, 2012
Improving Intelligence Forecasts, Research Roundup, Wharton article, Dec 20, 2011
Comments