Mastering the art of prediction: 3 lessons from ‘Superforecasting’ for Investors

“All of investing consists of dealing with the future.” ~ Howard Marks

The book Superforecasting discusses the findings from a research project called the Good Judgment Project, which studied the forecasting abilities of thousands of volunteers. In the Good Judgment Project, questions asked to forecasters typically involved predicting geopolitical events, economic trends, and international relations outcomes. Examples include:

- Will Argentina enter a recession within the next year?
- What will be the price of Gold by a certain date?
- Will a certain political leader remain in power by the end of the year?

All the participant’s accuracy of prediction was measured on a quantitative method called the Brier Scoring method, where a score of 1 is an accurate forecast and 0 means the worst possible forecast. The book highlights the traits and practices of the most successful forecasters, termed “superforecasters,” who were notably better at predicting global events than experts and even algorithms. In this article, I am sharing my top 3 takeaways and some lessons on how to be a better venture capital investor.

Superforecasting by Philip E. Tetlock & Dan Gardner

Kahneman cut to the chase: “Do you see them as different kinds of people, or as people who do different kinds of things?” My answer was, “A bit of both.” They score higher than average on measures of intelligence and open-mindedness, although they are not off the charts. What makes them so good is less what they are than what they do — the hard work of research, the careful thought and self-criticism, the gathering and synthesizing of other perspectives, the granular judgments and relentless updating.

Traits of Superforecasters:

In the book, various traits and themes(refer Notes 1) are identified as common among superforecasters. Here, I highlight the top three takeaways most applicable to venture capital investors.

1. Triage — Focus on mid-term time horizon: The authors talk about how, when trying to forecast, we should prioritize investment opportunities in the “Goldilocks zone” of uncertainty — avoiding both the overly obvious and the impossibly complex problem space. This means focusing on opportunities where due diligence and analytical rigor can uncover true potential and provide a competitive edge, rather than chasing highly unpredictable markets. This theme resonated with what I remember Buffet telling about ‘investing within your circle of competence’, & Sam Altman on ‘investing in seemingly bad, radical ideas.’ (refer Notes 2)

“Concentrate on questions in the Goldilocks zone of difficulty, where effort pays off the most.”

2. Fermi-zing — Decompose complex problems into sub-problems: For me, the single most tactical learning from this book was when the authors talked about how almost all the superforecasters when given seemingly intractable problems (or a ‘question’ to be precise) like ‘How many piano tuners are there in Chicago?’, they break down that bigger question into sub-questions like,
1. The number of Pianos in Chicago
2. Frequency with which Pianos are tuned
3. How long does it take to tune a piano (# of days)?
4. How many hours a year the average piano tuner works?

This method resonated with me since it is, in my opinion, one of the best analytical approaches to estimating the market size (TAM) for investment opportunities.

Fermi knew people could do much better and the key was to break down the question with more questions like “What would have to be true for this to happen?” Here, we can break the question down by asking, “What information would allow me to answer the question?”

3. Balance Between Confidence and Open-mindedness: A trait of super forecasters the authors keep talking about is their open-mindedness to information and opposing viewpoints. They subtly keep updating their probability of forecasting slightly with each new relevant information/ insight. In the context of the Good Judgment Project, this is totally feasible and makes sense, as the participants were allowed to update their forecast scores for each question until the deadline date. However, in the context of investing, having a conviction about a problem space and a startup’s solution is essential to tell a compelling story and get an investment approved; hence, there aren’t a lot of opportunities to go back on the investment decision(as the money is lock-in until the startup has a successful exit. There are also secondary markets where you could sell shares. But it’s not that big, and for the most part, Venture Investments are illiquid). I found this theme tangential, and the way I can relate to what a lot of corporate VC firms tell about ‘investing to learn’. I interpreted this theme as ‘Embrace uncertainty & invest to be in the journey for learning’ by Toyota Ventures. (refer notes 3)

“Superforecasters understand the risks both of rushing to judgment and of dawdling too long near “maybe.” They routinely manage the trade-off between the need to take decisive stands (who wants to listen to a waffler?) and the need to qualify their stands (who wants to listen to a blowhard?). They realize that long-term accuracy requires getting good scores on both calibration and resolution.”

~Rahul Sekar

P.S: If you were to ask me how likely I would recommend this book to anyone on a scale of 1 to 10, my response would be a 5.

Notes:

1.

A page from the book

2.
“We are the most successful when we fund things that other people don’t yet think are going to be immense deal but two years later become a big deal. And it’s really hard to predict that.” ~Sam Altman

“You want to tilt into the radical ideas.. but by their nature, you can’t predict what they will be.” ~Marc Andressen

3.
Two other themes in the book that I resonated with and found relevant for investors are to
1. Conduct post-mortems after each exit in a portfolio investment
2. Be intellectually honest always and engage in constructive criticism with your teammates.

4. Bonus: Notes from Michael Mauboussin’s paper on Pattern Recognition.

The white paper on pattern recognition was really enlightening. It describes environments governed by power-law distributions as ‘wicked,’ noting the difficulty in identifying clear causes for success. The article emphasizes that rapid learning and the development of our predictive models hinge on environments with swift feedback loops. Given the Venture Capital industry’s characteristic of long-term returns and extended feedback periods — often 7–10 years to judge the success of investments — it raises an intriguing question. Mauboussin defines an expert as someone whose predictive model is effective. So, considering our lengthy feedback loops, how can we as investors develop such a model and enhance our decision-making speed?

• Experts perceive patterns in their domains, solve problems qualitatively, and answer problems much faster and represent them at a deeper level when compared to novices.
• “There are a lot of areas where people who have experience think they’re experts, but the difference is that experts have predictive models, and people who have experience have models that aren’t necessarily predictive.”
• Experience leads to expertise only when there is learning guided by clear and timely feedback.
• Research shows that both intuition and expertise work in some settings and fail in others. Understanding where and why intuition and expertise are effective is essential for knowing when pattern recognition is effective.
• They found that intuitive expertise and pattern recognition tend to work well in stable environments where cause and effect are clear and participants can receive timely and accurate feedback.
• If stability and feedback are essential to successful pattern recognition, instability and unclear links between cause and effect show where pattern recognition fails. Robin Hogarth, a cognitive psychologist, distinguishes between “kind” and “wicked” environments. In kind environments, outcomes indicate the quality of the process and feedback is accurate and plentiful. In wicked environments, outcomes are a poor or misleading reflection of process because causal links are blurred.
• Expert agreement is one way to assess the validity of intuitive expertise. In kind environments, experts tend to agree on cues and the appropriate decisions that follow. For example, chess masters are likely to identify similar moves as attractive. In wicked environments, the views of experts often vary substantially. For instance, the one-year forecasts of the level of long-term interest rates by economists are not much different from random. Predictions of stock market returns by strategists and executives also tend to be poor.
• Pattern recognition often fails in complex and evolving environments. Ant colonies, cities, ecologies, economies, and stock markets are examples of complex adaptive systems.Properly identifying patterns within these systems is hard because cause and effect is not always clear. These systems also commonly exhibit non-linearity, where a small perturbation leads to a large outcome.
• Two prerequisites for acquiring intuitive expertise are a stable and linear environment and proper training with inputs that explain what works. Many, if not most, fundamental investors do not meet these basics. They tend to model corporate performance using what psychologists call the “inside view,” which focuses on the individual circumstances of a problem and draws heavily on personal understanding. For investors, this is a bottom-up method that considers the firm’s specific issues and is guided by the analyst’s experience.
• Outcomes that follow a power law are generally an indication of a wicked environment, where cause and effect are unclear and pattern recognition is hard.
• The bottom line is that fundamental investors can train their ability to recognize patterns under the correct conditions. Data that provide cues and causality, as well having a basis for timely and accurate feedback, are fundamental. The efficacy of pattern recognition is context dependent.
• Pattern recognition is more effective in stable environments where cause and effect are clear and participants are trained using timely and accurate feedback.
• Pattern recognition tends to fail in domains where causality and feedback are limited.
• All experts have experience but not all with experience are experts. The defining feature of an expert is having a predictive model that works. Ample research shows that expert predictions in social, political, and economic realms are poor.
• Investors who want to assess their skills at pattern recognition can maintain a journal and document their intuitions. Done properly, this allows for the measurement of calibration, or how well probabilistic forecasts match the frequency of outcomes. Over time, such an accurate self-assessment can help reveal where and when pattern recognition is accurate and adds value.