AI existential risk probabilities are (still) too unreliable to inform policy

AI Existential‑Risk Probabilities: Why They’re Too Unreliable for Policy

The public and policymakers are increasingly hearing numbers like “a 10 % chance of human extinction within 30 years due to AI.” These figures are meant to signal urgency, but the underlying forecasts are built on shaky ground. In this draft we unpack why the probability estimates for AI‑driven existential risk (p(doom)) are not the robust, model‑based numbers that other policy domains rely on, and we outline practical steps for decision makers who must grapple with this uncertainty.

1. The Illusion of Quantification

Quantitative forecasts are powerful because they come with a justification—a model, a data set, or a logical deduction that explains how the number was derived. When that justification is missing, a probability is merely a guess. For most policy areas—insurance, climate, epidemiology—probabilities are anchored in well‑defined reference classes or physical laws. AI existential risk, however, lacks both:

• No reference class – Extinction from AI is a one‑off event. We cannot point to past occurrences that share the same causal structure. Attempts to use animal extinction, industrial revolutions, or mass‑casualty accidents as analogues fall far short because they ignore the unique technological and governance variables that drive AI development.
• No deductive model – Unlike asteroid impacts, where physics lets us extrapolate from observed small impacts to rare catastrophic ones, AI’s trajectory depends on unknowns in algorithmic progress, alignment research, and policy responses. Theoretical models that estimate computational requirements or loss‑of‑control thresholds are built on assumptions that are far less testable than physical laws.

Without a grounded basis, the numbers that appear in policy briefs are subjective probabilities—the forecaster’s personal judgment expressed numerically. These vary wildly across experts and are difficult to reconcile or validate.

2. Why Subjective Estimates Fail for Policy

Subjective probabilities can be useful for private actors who are willing to accept a high degree of uncertainty. For governments, however, the stakes are different:

• Legitimacy – Policy decisions must be explainable to the public. A 10 % figure that cannot be traced back to a transparent model erodes trust.
• Cost‑benefit balance – Implementing restrictive measures (e.g., limiting open‑source AI releases) imposes economic costs on firms and society. If the risk estimate is not credible, policymakers risk over‑ or under‑investing.
• Equity – The burden of precautionary measures is unevenly distributed. Without a defensible probability, it is hard to justify why certain groups should bear the costs.

In short, subjective estimates lack the justification that democratic governance demands.

3. Practical Implications for Policymakers

1. Treat p(doom) as a heuristic, not a mandate – Use the estimates to flag areas that need more research rather than to set hard policy thresholds.
2. Invest in research infrastructure – Fund interdisciplinary work that seeks to build better reference classes (e.g., historical case studies of rapid technological change) and more robust theoretical models (e.g., formal frameworks for alignment and governance).
3. Adopt a “big‑tent” safety approach – Encourage collaboration across the entire AI safety community, not just the existential‑risk niche. Broader safety research (robustness, interpretability, value alignment) yields more immediate benefits and is less speculative.
4. Transparent communication – When presenting risk figures, disclose the underlying assumptions, the range of expert opinions, and the degree of uncertainty. This transparency helps maintain public trust.

4. A Path Forward

The urgency of AI safety is undeniable, but policy must be built on reliable evidence. Until we develop validated models or discover a suitable reference class, p(doom) figures should be treated with caution. By focusing on research, broadening the safety agenda, and maintaining transparent communication, governments can responsibly navigate the uncertainty while still fostering innovation.

TL;DR: AI existential‑risk probabilities are largely subjective guesses that lack the model‑based justification needed for policy, so policymakers should treat them as heuristics and invest in research rather than rely on them for concrete decisions.