Philip E. Tetlock

Prediction is hard, especially about the future. And despite the importance people and organizations lay on having a clear view of the future, we're not very good at prediction. The authors, Tetlock and Gardner, argue that the state of prediction is similar to the state of medicine, before randomized clinical trials. Sometime forecasters are right, but mostly they're wrong, and there's no way to separate the potentially useful treatments from quackish nonsense.

But there might be a better show more way. Superforecasters is a write up of the authors' Good Judgement Project, an IARPA (Intelligence Advanced Research Projects Agency) effort to systemically study predictions. The Good Judgement Project smoked the competition, including in-agency experts and prediction markets. Tetlock and Gardner used their background in psychology to find out what made their top 2%, the titular superforecasters, tick better than everyone else.

The first finding is that most of us are astonishingly bad at prediction. Generally, people have three settings for probability: impossible, certain, and maybe. Kahneman's System 1, the intuitive rush to judgment, is terrible at complex problems. Perhaps the biggest step is to slow down, and engage System 2, the rational and logical side of the mind. Beyond that, people with ideological blinders are lousy predictors. If everything has to fit into Marxist dialectics, or the immortal science of Friedman-Hayek Thought, you'll overlook evidence that contradicts the theory.

The Good Judgement project provide some necessary structure. Instead of weasel words (most of, highly likely) and vague timelines (in the near future), participants are given clear factual statements with a definite endpoint. There are thousands of predictions, and participants are allowed to update their initial assessments as they do more research.

Superforecasters are adept at seeing a problem in numerous ways, rather that focusing on grand theories. They tend to be comfortable with statistics. Participants in The Good Judgement Project self-selected as more intelligent and better educated than the population at large, but superforecasters aren't notably smarter or more credentialed than their less accurate peers. Instead, they have a bulldog tendency towards research, an ability to question their own assumptions, update beliefs, and think inside-out and outside-in.

Superforecasting is a fascinating and very useful book for anyone who is thinking about the future. show less

Perhaps I am biased in my view of predictability as a skill which accounts for a low rating for a book that others really liked. Tetlock's arguments and suggestions are sound and he does qualify them in the light of a disclaimer that very few people have really been successful in predicting outcomes from complex causes.

However, my concern is if the examples he's stated in his book can't really be justified as successful predictions. At close to 50% (though on the higher side), the odds of show more correct predictions are just based on chance. No doubt people may have analyzed the causes to determine the possible effect, but none of what is cited in the examples can either be considered as art or science. Meticulous analysis based on sound reasoning and statistical evidence or data will yield better decisions and that's common sense, not science.

Even in a deterministic world, predictability of outcomes is a game of chance considering the numerous hidden or unseen variables. Citing "chaos theory" does not make a hypothesis right.

As I mentioned at the start of my review, I am biased and have a closed view and read the book with the hope that my views and biases will be proven wrong; however, nothing in the book made a strong enough argument to change my views and beliefs. show less

A fascinating book to read, especially in the days after the unexpected Trump victory in the US Presidential race, despite the forecasts by "experts" that clearly predicted a Clinton win. Tetlock's introduction, written long before the election, seems appropriate to remember today: "There is plenty of room to stake out reasonable positions between the debunkers and the defenders of experts and their forecasts. On the one hand, the debunkers have a point. There are shady peddlers of show more questionable insights in the forecasting marketplace. There are also limits to foresight that may just not be surmountable. Our desire to reach into the future will always exceed our grasp. But debunkers go too far when they dismiss all forecasting as a fool's errand. I believe it is possible to see into the future, at least in some situations and to some extent, and that any intelligent, open-minded, and hardworking person can cultivate the requisite skills. Call me an 'optimistic skeptic.'" show less

I trained in the social sciences and so the ideas and stories in Philip Tetlock’s Superforecasting was a fascinating read for me.

Back in 2011, the research agency, IARPA, set up a series of forecasting tournaments in an effort to distill down best practices. This wasn’t a tournament to guess the weather or something seemingly trivial. This was a contest to predict events on a global scale, questions with geopolitical import, like what would happen in Syria or N. Korea. Teams in the show more tournament were given reams of questions by IARPA and over time the accuracy of their answers was recorded and assessed. Author Philip Tetlock led the winning team in that tournament. Called “Good Judgment Project,” Tetlock’s team outclassed the competition with consistently reliable estimates of events. How did they do it?

Superforecasting is a book that dismantles what forms an effective strategy for making predictions and offers a road map for better policymaking and geo-political wrangling.

Here are a few key findings discussed by Tetlock:

1. Expertise, especially specialization, is more often than not a liability. Tetlock observed that pundits and experts offered predictions that were less reliable than random guessing. (Yes, you could flip a coin and get better results than that expert with double PhDs.) Tetlock theorizes that these credentialed thinkers became so locked in by their own grand theories of history and current events that they ignore contrary evidence and complexity. These people often have a single “big idea” or narrative of the world that they adhere to (these beliefs are often based in political dogma) and tend to shoot for predictions that fit their grand theses. The are also stubbornly over-confident.

2. Not so surprisingly, political affiliations or beliefs have little impact on one’s predictive ability. It doesn’t matter where you are on the political spectrum. And the more extreme your beliefs (to the left or right), the less reliable you are at predicting events. Outlook doesn’t matter either. If you code as optimistic, you aren’t very reliable either; if you are pessimistic, you do even worse. (Lesson: Don’t listen to the doomsayers.)

3. Open-mindedness helps a lot. Tetlock argues that it’s those people who aren’t thinking in black and white, who acknowledge their limitations and do their research with an open mind, that fare the best. These are the people that look at the data—and lots of data—mindful of their own personal biases, and make sure to keep those biases in check.

4. Playing well with others helps, too. It’s an advantage to work with others and share data, opinions, and findings. Forecasters working in isolation—self-imposed or otherwise—rarely make good prognostications. That makes sense. Working in groups, especially diverse groups, exposes you to different points of view, which can sharpen forecasting.

5. General smarts and knowledge enhance predictive ability (dummies are not good predictors—ok, makes sense), but this sort of intelligence needs to be fluid, not static. People that are trapped by their own personal ideologies don’t fare well. People that do are willing to challenge their own thinking and make micro-changes to their forecasts as needed. In other words, they look at their ideas as testable theories, not prescriptive ideas set in stone.

6. Less ego in the thinking process leads to more reliable forecasting. Did telling the super forecasters that they did better than everyone else affect later performance? Remarkably, it didn’t. Their performance actually improved as they were reshuffled into groups to make more testable predictions. This last finding is noteworthy because it shows that predicative abilities can be analyzed and improved upon. Interventions at various levels—from our personal lives to government policymaking and foreign affairs can make a difference.

7. Superforecasters are logical people; they think numerically; they understand the basic and fundamental rules of probability.

8. Finally, superforecasters don’t try to make predictions on the huge problems; they take a huge problem and break it down into smaller, more manageable problems, making their forecasts on those smaller units. For example, they wouldn’t attempt to answer the question, “Will N. Korea launch a nuclear strike in the next three months?” Instead, they would look at various sub-questions that explore N Korea’s geopolitical relationships with its neighbors, its domestic state of affairs, its past behavior regarding nuclear weapons, and so on.

The main limitation of the Tetlock’s vision and what is beyond the scope of the book is what to do once reliable forecasts are made. If you predicted the bubble bursting in China’s economy, what then? The real trick is knowing how to proceed after a guess is made. What are the policy prescriptions?

Tetlock’s book is wonderfully readable and accessible, and I think that it will appeal to everyone from data nerds to the average reader looking for a ‘gee-whiz’ kind of read. I also think it’s a very important book. Forecasting isn’t some niche science. It’s an important part of formulating public policy responses, conducting foreign affairs and military action, and making financial and economic decisions. Getting it wrong is costly, and everyone is affected. Let’s start getting it right.

Math-y talk and quantitative discussions are simply presented. Tetlock recounts the evidence and offers a view that nicely reflects or fits that evidence. Tetlock also provides personal stories of the individual superforecasters, and so the writing has a strong story-centric rhythm. The “Ten Commandments for Aspiring Superforecasters” at the back of the book was a fun addition/primer. show less