Philip E. Tetlock (@PTetlock) header
Philip E. Tetlock (@PTetlock) profile picture

Penn-Integrates-Knowledge (PIK) Professor, Wharton & School of Arts & Sciences. Likes = interesting; Retweets = very interesting; Interesting ≠ endorsement

Philadelphia, PA
Joined October 2011
Photo by Philip E. Tetlock (@PTetlock)Photo by Philip E. Tetlock (@PTetlock)Photo by Philip E. Tetlock (@PTetlock)Photo by Philip E. Tetlock (@PTetlock)Photo by Philip E. Tetlock (@PTetlock)Photo by Philip E. Tetlock (@PTetlock)Photo by Philip E. Tetlock (@PTetlock)Photo by Philip E. Tetlock (@PTetlock)Photo by Philip E. Tetlock (@PTetlock)Photo by Philip E. Tetlock (@PTetlock)
AI companies increasingly say they may soon reach recursive self-improvement (RSI). With RSI, AI capability progress could accelerate dramatically. This acceleration could happen inside AI companies before outsiders can notice. How close is RSI? Our new report reviews the evidence, turns key disagreements into forecasting questions, and gives initial LLM forecasts. 🧵
3
20
5
119
17,624
Philip E. Tetlock retweeted
Sad about Jensen. Shakespearean. But as de Gaulle said, “The graveyards are filled with indispensable men.”
300
1,822
212
18,259
1,347,410
Philip E. Tetlock retweeted
People who have been in research fields develop a felt sense that most "research" (in almost any field that isn't exactingly technical) is research theater, and many superstar careers enjoy star status not because of a unique connection to the truth but because of a unique connection with an audience. No one really has the time or stature to, from a position of objectivity and trust, bring popular researchers to heel. But it will get increasingly hard to hide from serious critique. The machines can do critique now, at the level of depth and rigor previously only possible for the academically deranged. We're going to have to revise a lot of what we "know" about the world that isn't really knowledge but is actually just received popular consensus - like floating voltages without grounding wires. The grounding wires are coming.
Tyler Cowen: 'I am not even sure what to say at this point. We need a radical revaluation of this work, and of the associated policy arguments. More generally, this is only a single step in the story of how AI is going to revise economic science, and indeed past research more generally. As I stress in many of my recent talks, there will be significant gains and losses in status, with the losses often outweighing the gains. I see this as one of those cases.'
58
243
64
1,824
219,145
Philip E. Tetlock retweeted
Barely over a year after I wrote this, AI is already commoditizing the formalizer - making the best intuitionists more valuable than ever.
The best intellectual collaborations tend to be between two types of people: the intuitionist and the formalizer Sometimes an intuitionist's idea is so radical that no formalizer will engage. Reciprocally, a formal framework may be so complex that it lends to no intuition 1/4
4
8
4
88
10,859
🚀 Can your LLM forecaster outperform the best human forecasters? ForecastBench is the leading LLM forecasting benchmark and the only one that compares AI forecasters with superforecasters. This month we're running a new survey of superforecasters—setting a new baseline for top human forecasters on ForecastBench. If you're building an AI forecaster, now is your chance to test your system against the very best human forecasters. Submission details in the thread 👇
4
2
24
2,145
AI experts largely agree about which policies would most reduce catastrophic AI risk. The Longitudinal Expert AI Panel (LEAP) brings together forecasts from a representative group of over 250 experts that policymakers look to when considering the future of AI: computer scientists, industry figures, top economists, policy experts, and AI risk experts. We asked panelists to forecast (1) how a menu of policies would affect AI risk; (2) which policies they each support; and (3) which policies they think will be implemented by 2030. Key findings: • Despite disagreeing on the level of AI risk, respondents broadly agree on policy • Respondents believe government policy can substantially reduce catastrophic AI risk • The most supported policy of those we asked about was the creation of an international body, including the US and China as members, with pre-release authorization power over frontier models • Some US-only policies are forecast to reduce catastrophic AI risk, but not as much as bilateral policies involving the US and China • Experts strongly oppose federal preemption and think it would increase AI risk • The policies experts expect to reduce risk the most are those they think are least likely to actually be implemented 🧵 More, with key insights below:
3
32
9
115
14,830
𝗨𝗽𝗱𝗮𝘁𝗲 #𝟭: 𝗦𝗵𝗮𝗿𝗶𝗻𝗴 𝗳𝗼𝗿𝗲𝗰𝗮𝘀𝘁𝘀 𝗳𝗿𝗼𝗺 𝗮 "𝗳𝗮𝘀𝘁 𝗔𝗜 𝗽𝗿𝗼𝗴𝗿𝗲𝘀𝘀" 𝘀𝘂𝗯𝘀𝗮𝗺𝗽𝗹𝗲 The majority of forecasters in our Longitudinal Expert AI Panel (LEAP) have consistently underrated AI progress on benchmarks and other capability indicators. For example, experts and superforecasters gave median forecasts of 10% and 5.4%, respectively, that a Millennium Prize Problem would be solved with AI assistance before the end of 2027. However, some LEAP panelists predict much faster AI progress. These forecasters have tended to be somewhat more accurate on near-term benchmark and capability forecasts. This subsample puts more weight on the possibility of transformative AI arriving soon than most LEAP forecasters, and we're interested in what this group thinks about other AI progress questions. In future LEAP reports, we'll break out forecasts from this "fast AI progress" sample. Our goal is to highlight this group’s forecasts for people who want to put more weight on them, and to explicitly track where this worldview is more and less accurate. For details on how we might select this group, see our full post (link below).
1
2
5
1,308
Philip E. Tetlock retweeted
No. Once AI can manage to simulate with high accuracy the result of simply rolling a pair of backgammon dice - knowing all of the initial conditions as perfectly as it possibly can - then we can have that conversation. Until then, no.
The logical endpoint of AI development in the social sciences is psychohistory from Foundation. A predictive world model which can simulate with high accurate future events and counterfactuals
1
14
1,741
We’ve run the most comprehensive series of studies on expert AI forecasts over the last four years. Today, we’re sharing an interim update on our findings about the accuracy of these forecasts. Our major findings are: 1. Experts, including top economists, computer scientists, and biologists, have dramatically underestimated AI capabilities progress each year. Superforecasters have underestimated progress to an even greater extent. 2. Experts have a more mixed forecasting track record on AI diffusion-related measures, with some major underestimates (forecasting AI revenue) and other forecasts on track to be accurate (the share of electricity used for AI). 3. Some notable cases of overestimating AI progress: how much mid-2025 AI models could help amateurs do biorisk-relevant laboratory tasks; the speed of rollout of self-driving cars. 4. It is too soon to say how forecasters have performed at predicting macro-scale impact on outcomes such as GDP growth, major AI harms, and averted deaths from disease. More 🧵
4
93
26
349
52,242
AI models now match superforecasters on ForecastBench, our LLM forecasting benchmark. (Although not by a statistically significant margin). This October, we're running the rematch: superforecasters return to the benchmark, and the top AI forecasters can take them on. 🧵 In this thread: • What this result means • Why we're bringing superforecasters back • How to compete
10
19
5
157
11,815
Philip E. Tetlock retweeted
Forecasting is about constructing a web of relationships between events. Future models & harnesses with superhuman forecasting capability will construct validated webs with more complexity than humans can manage.
The core contribution of this new FRI paper on AI-risk isn’t a number. It is a research program that, one step at a time, spans the chasm between what we need to know & what is knowable, between “will AI kill 10%+ of us by 2030?” and “how many Millennium Problems will frontier models solve in the next 4 weeks?”
2
3
14
3,253
Introducing the Automated AI Risk Outlook (AIRO): a dashboard of catastrophic risk forecasts from an ensemble of frontier AI models, updated weekly. Current ensemble forecasts of an AI-related catastrophe that kills at least 10% of the global population: • By 2030: 0.47% • By 2050: 6% This dashboard is a prototype, and forecasting catastrophic risk is extremely challenging. We see the current forecasts as tentative, but we expect them to improve over time as AI capabilities advance. Dashboard: airo.forecastingresearch.org… Launch white paper: forecastingresearch.org/pdf/…
4
13
4
64
10,504
Philip E. Tetlock retweeted
New preprint from @lightningrodai, this time with Philip Tetlock and Ville Satopää! We post-train 5 versions of the same LLM, changing only the scoring rule used as the RL reward. Similar aggregate scores, but very different BIN profiles. A good Brier score alone doesn't tell you if a forecaster can distinguish likely from unlikely events. One might discern well but lose if its probabilities run systematically too high. A less discerning one might score better by hugging the base rate. BIN splits forecast performance into bias, information, and noise. Bias is a systematic shift in the probabilities. Noise is random scatter. Information is real signal about which outcomes are more likely — the part you need a powerful LLM for. Different uses call for different profiles. Reward choice is one lever shaping which forecaster you get. Congrats to co-authors @indiequant @KSkotheim64001 @VSatopaa @PTetlock 🙌 Full paper: arxiv.org/abs/2608.28482
8
2
23
5,203
In his recent blog post, @slatestarcodex wrote that "the AI superforecasters are here." We run ForecastBench—a leading AI forecasting benchmark and the only one to compare AI forecasters to human superforecasters. So, how close are current AI systems to superforecasters? For the first time, several AI models are now statistically indistinguishable from superforecasters.
4
48
13
236
56,057
Philip E. Tetlock retweeted
Paging @PTetlock!
i remind folks that almost all the major falsifiable political predictions that they made in ai 2027 so far have turned out incorrect they thought china would consolidate almost all their labs in mid-2026 and the us government would not meaningfully respond to cyber capabilities
3
5
2
70
49,380
Philip E. Tetlock retweeted
"Debates about political bias in AI often conflate ideological leanings with epistemic error." @DavidRozado and @PTetlock offer a more coherent basis for evaluating and regulating AI systems in politically contested domains: link.springer.com/article/10…
1
3
1
12
1,348
Philip E. Tetlock retweeted
1. Have we been measuring AI political bias wrong? In a new paper @PTetlock and I argue that we might have. Studies have found that AIs tend to produce left-of-center responses to politically loaded questions. But ideological preferences are not the same as epistemic failure.
12
27
4
129
16,768
New in @Theory_Society, David Rozado (@DavidRozado) and Phil Tetlock (@PTetlock) argue that an AI system leaning politically left or right isn't bias on its own -- real bias is when that lean coincides with epistemic distortions like inconsistent evidentiary standards, failures of perspective-taking or faulty inferences. Check out their fascinating article below! link.springer.com/article/10…
1
7
2
29
5,758
Philip E. Tetlock retweeted
Seems @claudeai use Superforecaster manual by @PTetlock to ensure success.
5
34
3,204
Is it possible to spot a good forecast by its rationale? We used LLMs to score the reasoning behind 55,000+ forecasts and test the link between forecast accuracy and written rationales. We found that: • Causal reasoning is much more prevalent than statistical argumentation • It's easier to identify poor forecasters rather than excellent ones • Human ratings of rationale quality can be unreliable. 🧵A thread on the results:
2
11
2
47
5,177