Within Conditional Risk
Why the Same p(doom) Can Mean Opposite Things
A small change in expected AI arrival or post-arrival danger can sharply alter the headline p(doom) figure.
On this page
- Breaking p(doom) into arrival and catastrophe probabilities
- Worked examples with different timelines and danger estimates
- Why similar totals can hide deep disagreement
Page outline Jump by section
Introduction
A single p(doom) figure can give a false impression of precision. In practice, estimates of AI existential risk are built from multiple uncertain assumptions, and relatively small changes in those assumptions can produce dramatically different headline probabilities. That is why two people who both report a 10% chance of AI-driven extinction may disagree about almost everything else: one may think transformative AI is unlikely to arrive soon but extremely dangerous if it does, while the other expects advanced AI to arrive almost certainly but believes safety measures will usually succeed. Understanding how these assumptions interact is more informative than focusing on the final percentage alone.[AI Impacts]aiimpacts.org2022 expert survey on progress in aiAI Impacts2022 Expert Survey on Progress in AI – AI ImpactsAugust 3, 2022…
This page focuses on the mechanics behind those changing estimates rather than on the broader debate over whether AI doom is likely. The key idea is that p(doom) is often the product of several probabilities, each of which reflects a different judgement about the future.
Breaking p(doom) Into Arrival and Catastrophe Probabilities
The simplest way to understand changing doom estimates is to separate two questions:
- How likely is transformative AI to be developed within the period being discussed?
- If it is developed, how likely is it to cause an existential catastrophe?
In probability terms:
Overall AI doom probability = probability that transformative AI arrives × probability of catastrophe given that it arrives
Although real analyses may include additional stages—such as whether unsafe systems are deployed, whether warning signs are recognised, or whether governments intervene—this two-stage model explains much of the variation between published estimates.[AI Impacts]aiimpacts.org2022 expert survey on progress in aiAI Impacts2022 Expert Survey on Progress in AI – AI ImpactsAugust 3, 2022…
The first probability depends mainly on beliefs about AI progress:
- whether current machine-learning approaches continue scaling;
- scientific breakthroughs;
- hardware improvements;
- engineering bottlenecks;
- investment and economic incentives.
The second depends on different assumptions:
- whether alignment problems prove technically solvable;
- whether highly capable systems become deceptively misaligned or difficult to supervise;
- whether governance keeps pace with capability growth;
- whether competitive pressure encourages premature deployment.
Changing either side changes the overall figure.
Worked Examples With Different Timelines and Danger Estimates
A few simple examples show why modest changes in assumptions can have large effects.
Chance transformative AI arrivesChance of catastrophe if it arrivesOverall p(doom)20%50%10%40%25%10%80%12.5%10%90%5%4.5%30%40%12%
Each row represents a different worldview.
The first assumes transformative AI may never arrive during the relevant period, but that losing control would be common if it did.
The third assumes transformative AI is almost inevitable but that safety efforts usually work.
Both can generate similar headline probabilities despite fundamentally different beliefs about technology and governance.
This is one reason many researchers encourage readers to ask “10% because of what?” rather than treating every p(doom) estimate as directly comparable.[80,000 Hours]80000hours.org80,000 Hours Why AI risks are the world’s most pressing problems | 80,000 Hours80,000 Hours Why AI risks are the world’s most pressing problems | 80,000 Hours
Small Assumption Changes Can Produce Large Probability Changes
The relationship between assumptions and overall risk is often non-linear.
Imagine someone initially believes:
- 50% chance transformative AI arrives this century.
- 20% chance of existential catastrophe if it arrives.
Overall p(doom) = 10%.
Now suppose new evidence convinces them that AI development is progressing faster than expected, increasing the arrival probability to 80%, while nothing else changes.
Overall p(doom) becomes 16%.
Alternatively, imagine capability forecasts stay unchanged, but new alignment techniques appear more promising, reducing conditional catastrophe risk from 20% to 8%.
Overall p(doom) falls to 4%.
Neither revision required changing every belief. One updated assumption altered the final estimate substantially.
This sensitivity explains why major advances in AI capabilities, unexpected technical obstacles, or evidence about control methods can all shift public p(doom) estimates even if underlying values remain unchanged.
Why Similar Totals Can Hide Deep Disagreement
Published estimates often conceal where disagreement actually lies.
Two researchers might both report a 5% overall probability while disagreeing about almost every important question.
One might believe:
- transformative AI before 2040 is highly likely;
- alignment remains unsolved;
- governments will respond effectively enough to avoid most catastrophes.
Another might instead believe:
- transformative AI is unlikely before the end of the century;
- if it eventually arrives without strong preparation, catastrophe is extremely difficult to avoid.
Their identical headline estimate masks opposing views about:
- AI timelines;
- technical alignment;
- institutional competence;
- international coordination;
- warning signs during development.
Conversely, people who broadly agree about AI timelines may report very different p(doom) values because they disagree about whether alignment research, evaluations, interpretability, monitoring, or deployment controls will scale alongside capabilities.[80,000 Hours]80000hours.org80,000 Hours Risks from power-seeking AI systems | 80,000 Hours80,000 Hours Risks from power-seeking AI systems | 80,000 Hours
What New Evidence Usually Changes
Not every piece of evidence should change every component of a doom estimate.
Different developments mainly affect different assumptions.
Evidence about AI capabilities tends to update beliefs about whether transformative AI will arrive. Examples include rapid benchmark improvements, falling training costs, or unexpected scientific breakthroughs.
Evidence about AI control tends to update the conditional probability after arrival. Examples include successful interpretability methods, stronger evaluations, reliable monitoring techniques, or, conversely, demonstrations of deceptive behaviour or failures of oversight.
Evidence about governance may affect both. Strong international coordination might reduce conditional catastrophe risk, while accelerating competitive races between organisations could increase it if safety work cannot keep pace.
Keeping these updates separate avoids treating every new AI result as evidence for or against every part of the overall argument.
Why Expert Surveys Still Show Wide Variation
Surveys of AI researchers consistently reveal large spreads in existential-risk estimates rather than convergence on a single number. In the 2022 expert survey conducted by AI Impacts, the median respondent assigned a 5% probability that advanced AI would produce an “extremely bad” long-term outcome such as human extinction, but responses ranged from zero to much higher values, with nearly half assigning at least a 10% chance.[AI Impacts]aiimpacts.org2022 expert survey on progress in aiAI Impacts2022 Expert Survey on Progress in AI – AI ImpactsAugust 3, 2022…
Those differences are not simply disagreements over arithmetic. They reflect different assumptions about:
- how quickly transformative AI will arrive;
- whether current approaches scale to general intelligence;
- whether alignment problems are fundamentally solvable;
- how capable future governance systems will be;
- whether dangerous deployment can be delayed if warning signs appear.
Because each estimate combines several uncertain judgements, similar-looking numbers often rest on entirely different reasoning.
The Main Lesson: Ask About the Assumptions, Not Just the Percentage
The most useful way to interpret any p(doom) estimate is to unpack the chain of reasoning behind it.
Instead of asking whether someone’s estimate is 5%, 10%, or 30%, it is often more revealing to ask:
- How likely do they think transformative AI is to arrive?
- What do they believe happens if it does?
- Which assumptions are driving most of the risk?
- What future evidence would make them revise their estimate?
Seen this way, p(doom) is less like a fixed prediction and more like a summary of several connected beliefs. Small changes in timelines, technical expectations, governance assumptions, or confidence in alignment methods can all shift the final number substantially, even when the underlying concern about AI existential risk remains the same.
Amazon book picks
Further Reading
Books and field guides related to Why the Same p(doom) Can Mean Opposite Things. Use these as the next step if you want deeper reading beyond the article.
Human Compatible
A leading artificial intelligence researcher lays out a new approach to AI that will enable us to coexist successfully with increasingly...
The Alignment Problem
Finalist for the Los Angeles Times Book Prize A jaw-dropping exploration of everything that goes wrong when we build AI systems and the m...
Superintelligence
This profoundly ambitious and original book picks its way carefully through a vast tract of forbiddingly difficult intellectual terrain.
Superforecasting
The international bestseller 'A manual for thinking clearly in an uncertain world. Read it.' Daniel Kahneman, author of Thinking, Fast an...
eBay marketplace picks
Marketplace Samples
Live-tested eBay searches with available results related to this page.
Selected fromartificial intelligence wall art oneBay.co.uk.
Current eBay listing
Artificial intelligence Framed Wall Art Poster Canvas Print Picture
Current eBay listing
Artificial intelligence Framed Wall Art Poster Canvas Print Picture
Endnotes
1.
Source: 80000hours.org
Title: 80,000 Hours Why AI risks are the world’s most pressing problems | 80,000 Hours
Link:https://80000hours.org/problem-profiles/artificial-intelligence/
2.
Source: 80000hours.org
Link:https://80000hours.org/podcast/episodes/carl-shulman-common-sense-case-existential-risks/
3.
Source: 80000hours.org
Title: 80,000 Hours Risks from power-seeking AI systems | 80,000 Hours
Link:https://80000hours.org/problem-profiles/risks-from-power-seeking-ai/
4.
Source: 80000hours.org
Title: a safe refuge in the southern hem
Link:https://80000hours.org/problem-profiles/nuclear-security/
Source snippet
Nuclear weapons | 80,000 HoursJune 1, 2024 — We’d guess the answer to that is yes — but mainly because of the technology we’ve developed...
Published: June 1, 2024
5.
Source: 80000hours.org
Title: The case for reducing existential risks
Link:https://80000hours.org/articles/existential-risks/
6.
Source: 80000hours.org
Link:https://80000hours.org/podcast/episodes/owen-cotton-barratt-epistemic-systems/
7.
Source: youtube.com
Title: AI Impacts Survey
Link:https://www.youtube.com/watch?v=xwJx_xqZI3Q
Source snippet
The threat of existential risk from AI...
8.
Source: aiimpacts.org
Title: 2022 expert survey on progress in ai
Link:https://aiimpacts.org/2022-expert-survey-on-progress-in-ai/
Source snippet
AI Impacts2022 Expert Survey on Progress in AI – AI ImpactsAugust 3, 2022...
Published: August 3, 2022
9.
Source: wiki.aiimpacts.org
Title: 2022 expert survey on progress in ai
Link:https://wiki.aiimpacts.org/ai_timelines/predictions_of_human-level_ai_timelines/ai_timeline_surveys/2022_expert_survey_on_progress_in_ai
Source snippet
AI Impacts2022 Expert Survey on Progress in AI [AI Impacts Wiki]...
10.
Source: wiki.aiimpacts.org
Title: quantitative estimates of ai risk
Link:https://wiki.aiimpacts.org/arguments_for_ai_risk/quantitative_estimates_of_ai_risk
11.
Source: aiimpacts.org
Title: Kruel AI Interviews – AI Impacts
Link:https://aiimpacts.org/kruel-[ai-survey
Additional References
12.
Source: youtube.com
Title: Top AGI Safety Researcher with 90% P(Doom) on the Trajectory to ASI
Link:https://www.youtube.com/watch?v=oOb9K1KIAyk
Source snippet
AI Impacts Survey with Katja Grace This video directly discusses the empirical findings and changing assumption variables from the major...
13.
Source: youtube.com
Link:https://www.youtube.com/watch?v=MMMqQm22LS0
Source snippet
Top AGI Safety Researcher with 90% P(Doom) on the Trajectory to ASI...
14.
Source: youtube.com
Title: P(doom): Probability that AI will destroy human civilization
Link:https://www.youtube.com/watch?v=xW0xjAMD60c
Source snippet
Nate Soares on P(Doom), Alien Superintelligence, Human Enhancement, and the Future of AI...
15.
Source: arxiv.org
Link:https://arxiv.org/abs/2502.14870
16.
Source: banthebots.org
Title: Last updated
Link:https://www.banthebots.org/explainers/ai-doomers
Source snippet
AI Doomers and p(doom): What the Fear Really MeansJuly 27, 2026 — AI DOOMERS AND P(DOOM): WHAT THE FEAR REALLY MEANS A plain-English guid...
Published: July 27, 2026
17.
Source: link.springer.com
Link:https://link.springer.com/article/10.1007/s43681-024-00475-w
Source snippet
By definition, an AI catastrophe would be very bad. This means that [catastrophic]({{ 'catastrophic-misuse/' | relative_url }}) AI risk can be high, i.e. severe, even if i...
18.
Source: ourworldindata.org
Link:https://ourworldindata.org/ai-timelines
19.
Source: sfu.ca
Link:https://www.sfu.ca/~smith/pdoom_interactive_standalone.html
20.
Source: sciencedirect.com
Link:https://www.sciencedirect.com/science/article/pii/S0264999326002476
21.
Source: journals.ub.uni-koeln.de
Link:https://journals.ub.uni-koeln.de/index.php/phai/article/download/2801/11838/23299?inline=1