Within Objections
Why p(doom) estimates diverge so sharply
Small doubts at each step of a takeover story can compound into much lower estimates of extinction risk.
On this page
- The chain of assumptions in takeover scenarios
- Why multiplying uncertainty matters
- How sceptics and doomers read the same gaps
Page outline Jump by section
Introduction
One reason p(doom) estimates differ so dramatically is that people often disagree less about the final catastrophe than about the chain of assumptions required to reach it. P(doom) is an informal estimate of the probability that advanced AI causes an existential catastrophe such as human extinction or permanent loss of humanity’s ability to shape its future. It is not a measured statistic but a judgement under profound uncertainty. [ai-safety-atlas.com]ai-safety-atlas.comThe term has evolvedAppendix: Quantifying Existential Risks - Chapter 2P(doom) represents the subjective probability that artificial intelligence will cause…
In debates about AI loss of control, doom-focused researchers frequently describe a sequence of events: highly capable AI is developed, alignment fails, the system becomes strategically dangerous, humans lose control, and that loss of control leads to irreversible catastrophe. Critics often accept that each step is possible while arguing that uncertainty accumulates at every stage. When many uncertain claims are linked together, the resulting probability can become much smaller than any individual step initially appears. This process is often called uncertainty stacking. [arXiv]arxiv.orgarXiv Is Power-Seeking AI an Existential Risk?arXivIs Power-Seeking AI an Existential Risk?June 16, 2022…
The chain of assumptions in takeover scenarios
Most loss-of-control arguments are not built on a single prediction. They are built on multiple conditional claims.
A simplified takeover pathway might look like this:
- AI systems become far more capable than today’s models.
- Those systems become highly agentic, pursuing goals over long periods.
- Alignment techniques fail to keep their objectives compatible with human interests.
- Misaligned systems develop incentives to seek power or evade control.
- Human institutions fail to detect or stop the problem.
- The resulting loss of control becomes global and irreversible.
- Humanity is permanently disempowered or destroyed.
Researchers who are concerned about existential risk often view several of these steps as plausible enough that the combined risk remains substantial. Critics focus on the fact that each step introduces another layer of uncertainty. [arXiv]arxiv.orgarXiv Is Power-Seeking AI an Existential Risk?arXivIs Power-Seeking AI an Existential Risk?June 16, 2022…
This matters because the probability of a whole chain occurring depends on the probability of every major link. Even if no single step seems especially unlikely, the combined probability can fall quickly once multiple uncertain assumptions are multiplied together.
A simple illustration
Suppose someone assigns:
- 70% chance that highly capable AI is developed soon.
- 50% chance that alignment fails.
- 40% chance that failure leads to dangerous power-seeking behaviour.
- 30% chance that humans cannot regain control.
- 50% chance that such loss of control becomes existential.
Multiplying those probabilities yields roughly 2.1%, not 70%.
The numbers themselves are arbitrary, but the example illustrates why sceptics often ask doom advocates to specify the individual assumptions hidden inside a headline p(doom) estimate. A disagreement that appears to be about one number may actually be a disagreement about several different links in the chain. [arXiv]arxiv.orgarXiv Is Power-Seeking AI an Existential Risk?arXivIs Power-Seeking AI an Existential Risk?June 16, 2022…
Why multiplying uncertainty matters
The strongest versions of AI doom arguments often rely on events that have never yet been observed directly.
There is evidence for some precursor phenomena. Researchers have documented specification gaming, reward hacking, goal misgeneralisation and other cases where AI systems pursue unintended strategies. However, there is currently no public example of an AI system carrying out the kind of large-scale autonomous power-seeking envisioned in classic takeover scenarios. Reviews of the evidence therefore tend to conclude that the risk is concerning but not decisively established. [arXiv]arxiv.orgarXiv Is Power-Seeking AI an Existential Risk?arXivIs Power-Seeking AI an Existential Risk?June 16, 2022…
For sceptics, this creates an important distinction. Evidence that a system exploits a training loophole is not the same as evidence that a future system will conceal long-term goals, strategically deceive operators, seize critical infrastructure and permanently disempower humanity. Each transition requires additional inference. [arXiv]arxiv.orgarXiv Is Power-Seeking AI an Existential Risk?arXivIs Power-Seeking AI an Existential Risk?June 16, 2022…
This is where uncertainty stacking becomes central. If confidence falls slightly at every inferential step, overall p(doom) estimates can shrink dramatically. Critics argue that some discussions implicitly treat a long sequence of speculative assumptions as though it were a single prediction. When the assumptions are unpacked, the final probability often looks much smaller. [arXiv]arxiv.orgarXiv Is Power-Seeking AI an Existential Risk?arXivIs Power-Seeking AI an Existential Risk?June 16, 2022…
How sceptics and doomers read the same gaps
One of the most interesting features of the AI-risk debate is that both sides often agree about the evidence but interpret its implications differently.
The doomer interpretation
People with relatively high p(doom) estimates tend to argue that uncertainty should not automatically reduce concern.
Their reasoning is often that humanity has never before attempted to build systems that could potentially exceed human cognitive capabilities across many domains. The absence of direct evidence may therefore be exactly what one would expect before such systems exist. Waiting for decisive proof could mean waiting until the danger has already materialised. [planned-obsolescence.org]planned-obsolescence.orgScience and speculationby Ajeya CotraMay 1, 2026 — 1 May 2026 — We almost certainly won't be able to develop an evidence base about AI risks anywhere near as ro…
From this perspective, uncertainty does not necessarily imply safety. Instead, uncertainty may be a warning sign that society is entering poorly understood territory.
The sceptical interpretation
Critics usually agree that future AI could become extremely capable. Their objection is that uncertainty cuts both ways.
If there is little direct evidence for strategic deception, recursive self-improvement, durable power-seeking or irreversible takeover, then confidence in those outcomes should remain limited. Many sceptics argue that doom narratives often combine several speculative claims and then discuss the resulting scenario as though it were a single forecast. [arXiv]arxiv.orgarXiv Is Power-Seeking AI an Existential Risk?arXivIs Power-Seeking AI an Existential Risk?June 16, 2022…
Under this view, uncertainty stacking is not a minor technical adjustment. It is one of the strongest reasons to resist very high p(doom) estimates.
Why experts can end up far apart
The uncertainty-stacking problem helps explain why expert estimates vary by orders of magnitude.
Two researchers might agree on most facts yet differ sharply on a few key conditional questions:
- How likely is transformative AI this century?
- How difficult is alignment?
- How common is power-seeking behaviour?
- How effective will monitoring and control methods become?
- How capable will governments and organisations be at intervention?
Small differences at each stage can produce enormous differences in final estimates. Someone assigning relatively optimistic probabilities to five uncertain links may arrive at a p(doom) below 1%. Someone assigning moderately pessimistic probabilities to the same links may reach 20%, 30% or higher. [arXiv]arxiv.orgarXiv Is Power-Seeking AI an Existential Risk?arXivIs Power-Seeking AI an Existential Risk?June 16, 2022…
This helps explain why p(doom) figures often appear unstable or highly subjective. Researchers are not usually disagreeing about a single observable fact. They are disagreeing about a nested structure of forecasts, assumptions and unknowns. [CSET]cset.georgetown.eduCSETBeyond P(doom) for AI Risk: Quantifying Uncertainty…This issue brief explains why analysts and decision-makers need alternatives t…
The deeper disagreement: probability versus deep uncertainty
A further complication is that some analysts question whether a single probability is even the right tool.
The future of advanced AI may involve what economists and decision theorists call deep or Knightian uncertainty: situations where there is no reliable historical data from which to derive probabilities. Several researchers have argued that expressing beliefs solely through a single p(doom) number can create a misleading impression of precision. Alternative approaches focus on ranges of possibilities, conditional scenarios, or measures of confidence rather than point estimates alone. CSET [2lesswrong.com]lesswrong.comcommunicating effectively under knightian norms3 Apr 2023 — When Scott Alexander says "33% risk of AI doom" or Eliezer puts it at 90%, they are making estimates, and that is clearly a…
This criticism does not necessarily imply that existential risk is low. Instead, it argues that uncertainty itself is part of the story. A claim that there is a 30% chance of doom may sound precise, but the uncertainty surrounding that estimate may be almost as important as the estimate itself.
What uncertainty stacking actually tells us
Uncertainty stacking does not prove that AI doom is unlikely. Nor does it prove that high p(doom) estimates are mistaken.
What it does show is why debates about AI existential risk often become debates about intermediate assumptions rather than final outcomes. A takeover scenario may sound plausible when described as a narrative. But when broken into individual claims, each link must earn its own credibility.
For sceptics, this is one of the strongest objections to loss-of-control fears: many takeover stories require a long sequence of uncertain events, and multiplying those uncertainties can drive overall risk estimates down sharply. For doom advocates, the reply is that several links may be more likely than critics assume, and that even a relatively small probability of irreversible catastrophe deserves serious attention. [arXiv]arxiv.orgarXiv Is Power-Seeking AI an Existential Risk?arXivIs Power-Seeking AI an Existential Risk?June 16, 2022…
The result is not merely a disagreement about one number. It is a disagreement about how to reason when evidence is incomplete, unprecedented technologies are advancing rapidly, and every major step in the argument contains uncertainty of its own.
Amazon book picks
Further Reading
Books and field guides related to Why p(doom) estimates diverge so sharply. Use these as the next step if you want deeper reading beyond the article.
Human Compatible
Directly addresses AI control problems, uncertainty, and competing views about existential risk.
The Alignment Problem
Examines how different assumptions about alignment produce different risk assessments.
Superintelligence
Explores chains of assumptions leading from advanced AI to catastrophic outcomes.
Life 3.0
Surveys possible AI futures and the uncertainty surrounding long-term outcomes.
eBay marketplace picks
Marketplace Samples
Example marketplace items related to this page. Use the search link to explore similar finds on eBay.
Endnotes
-
Source: ai-safety-atlas.com
Title: The term has evolved
Link: https://ai-safety-atlas.com/chapters/v1/risks/appendix-quantifying-existential-risks/Source snippet
Appendix: Quantifying Existential Risks - Chapter 2P(doom) represents the subjective probability that [artificial]({{ 'artificial-goals/' | relative_url }}) intelligence will cause...
-
Source: arxiv.org
Title: arXiv Is Power-Seeking AI an Existential Risk?
Link: https://arxiv.org/abs/2206.13353Source snippet
arXivIs Power-Seeking AI an Existential Risk?June 16, 2022...
Published: June 16, 2022
-
Source: arxiv.org
Link: https://arxiv.org/abs/2310.18244 -
Source: planned-obsolescence.org
Title: Science and speculation
Link: https://www.planned-obsolescence.org/p/science-and-speculationSource snippet
by Ajeya CotraMay 1, 2026 — 1 May 2026 — We almost certainly won't be able to develop an evidence base about AI risks anywhere near as ro...
Published: May 1, 2026
-
Source: arxiv.org
Link: https://arxiv.org/abs/2512.04119Source snippet
arXivHumanity in the Age of AI: Reassessing 2025's Existential-Risk NarrativesDecember 1, 2025...
Published: December 1, 2025
-
Source: cset.georgetown.edu
Link: https://cset.georgetown.edu/publication/beyond-pdoom-for-ai-risk-quantifying-uncertainty-without-probability/Source snippet
CSETBeyond P(doom) for AI Risk: Quantifying Uncertainty...This issue brief explains why analysts and decision-makers need alternatives t...
-
Source: lesswrong.com
Title: communicating effectively under knightian norms
Link: https://www.lesswrong.com/posts/tG9BLyBEiLeRJZvX6/communicating-effectively-under-knightian-normsSource snippet
3 Apr 2023 — When Scott Alexander says "33% risk of AI doom" or Eliezer puts it at 90%, they are making estimates, and that is clearly a...
-
Source: arxiv.org
Link: https://arxiv.org/pdf/2503.07341Source snippet
The Economics of p(doom): Scenarios of Existential Risk...by J Growiec · 2025 · Cited by 10 — By contrast, the probability of AI doom—hu...
-
Source: arxiv.org
Title: What are the odds?
Link: https://arxiv.org/html/2510.23453v1Source snippet
Risk and uncertainty about AI...27 Oct 2025 — This work is a commentary of the article AI Survival Stories: a Taxonomic Analysis of AI E...
-
Source: arxiv.org
Link: https://arxiv.org/html/2505.04592v1Source snippet
AI [Governance]({{ 'governance/' | relative_url }}) to Avoid Extinction: The Strategic...Risks come from failure to control powerful AI systems, misuse of AI by malicious rog...
-
Source: Wikipedia
Link: https://en.wikipedia.org/wiki/P%28doom%29Source snippet
P(doom)In AI safety, P(doom) is the probability of existentially catastrophic outcomes (so-called "doomsday scenarios") as a result of...
-
Source: garymarcus.substack.com
Link: https://garymarcus.substack.com/p/d28/commentsSource snippet
p(doom) - by Gary Marcus - Marcus on AI27 Aug 2023 — The author does provide quantitative estimates for p(doom): "My best guess is that h...
Additional References
-
Source: linkedin.com
Link: https://www.linkedin.com/posts/garykucher_ai-aisafety-aialignment-activity-7438995662379130880-cFCSSource snippet
AI Existential Risk: Understanding p(Doom) EstimatesP(Doom) is shorthand for “probability of doom,” meaning the estimated chance that adv...
-
Source: x.com
Link: https://x.com/ajeya_cotra/status/1655243379637391360Source snippet
Ajeya CotraAjeya Cotra (@ajeya_cotra). 162 likes 9 replies. A common criticism of people who are trying to stop existential risk from pow...
-
Source: linkedin.com
Link: https://www.linkedin.com/posts/anlohn_beyond-pdoom-for-ai-risk-quantifying-uncertainty-activity-7457825915650203649-oKH8Source snippet
Alternative to Probability in AI Risk AssessmentThis was an excellent paper by Drew and CSET. I have similarly been frustrated with how p...
-
Source: abundance.institute
Link: https://abundance.institute/our-work/the-ai-technopanic-and-its-effectsSource snippet
The AI Technopanic and Its EffectsThe essay by Ajeya Cotra, who oversee Open Philanthropy's “Potential risks from advanced artificial int...
-
Source: psc-consultant.com
Title: ai 2027 a wake up call on advanced ai and existential risk
Link: https://www.psc-consultant.com/post/ai-2027-a-wake-up-call-on-advanced-ai-and-existential-riskSource snippet
AI-2027: A Wake-Up Call on Advanced AI and Existential...20 Oct 2025 — Explore AI-2027, a deeply researched scenario forecasting the ris...
-
Source: forum.effectivealtruism.org
Title: draft report on existential risk from power seeking ai
Link: https://forum.effectivealtruism.org/posts/78NoGoRitPzeT8nga/draft-report-on-existential-risk-from-power-seeking-aiSource snippet
report on existential risk from power-seeking AI28 Apr 2021 — I've written a draft report evaluating a version of the overall case for ex...
-
Source: abc.net.au
Title: whats your pdoom [ai researchers]({{ ‘expert-surveys/’ | relative_url }}) worry catastrophe
Link: https://www.abc.net.au/news/2023-07-15/whats-your-pdoom-ai-researchers-worry-catastrophe/102591340Source snippet
'What's your p(doom)?': How AI could be learning a...14 Jul 2023 — If a sophisticated AI was then motivated to defend itself, Ms Cotra a...
-
Source: alignmentforum.org
Link: https://www.alignmentforum.org/posts/pRkFkzwKZ2zfa3R6H/without-specific-countermeasures-the-easiest-path-toSource snippet
most concrete, detailed, clear, and comprehensive) story of existential risk from AI I know of (IMO). I expect I'll...Read more...
-
Source: reddit.com
Title: ai 2027 is the most realistic and terrifying
Link: https://www.reddit.com/r/collapse/comments/1kzqh53/ai_2027_is_the_most_realistic_and_terrifying/Source snippet
2027, humanity is basically sidelined. AI systems are so advanced and complex that even their creators don't fully underst...
-
Source: thebulletin.org
Title: stopping the clock on catastrophic ai risk
Link: https://thebulletin.org/premium/2025-12/stopping-the-clock-on-catastrophic-ai-risk/Source snippet
10 Dec 2025 — AI is already sufficiently robust that it introduces new global risks and exacerbates existing threats. Its development is...
Topic Tree