Within P Doom
Are We Estimating Doom From Today or After AGI?
Someone can expect a low overall chance of AI doom while still believing superhuman systems would be extremely dangerous if built.
On this page
- The difference between conditional and unconditional risk
- Separating capability arrival from catastrophe risk
- Why identical numbers can encode different beliefs
Page outline Jump by section
Introduction
When people quote a p(doom) estimate, they often sound as though they are talking about a single probability. In reality, two very different questions are being conflated. One asks: from today, what is the chance that advanced AI eventually causes an existential catastrophe? The other asks: if humanity does build genuinely superhuman or transformative AI, how likely is that technology to lead to doom?
This distinction matters because people can agree about one question while disagreeing sharply about the other. Someone may believe there is only a small overall chance of AI causing extinction because they doubt transformative AI will be developed soon—or at all. Yet the same person could believe that, if such systems are built, the danger of losing control would be extremely high. Conversely, another person may be confident that transformative AI will arrive this century but believe alignment techniques, governance and safety engineering will keep existential risk low. Understanding this conditional-versus-unconditional distinction makes many apparent disagreements over p(doom) much easier to interpret.[AI Impacts]wiki.aiimpacts.org2023 expert survey on progress in aiAI Impacts2023 Expert Survey on Progress in AI [AI Impacts Wiki]August 17, 2023…
The Difference Between Conditional and Unconditional Risk
The distinction comes directly from probability theory.
An unconditional probability begins with the world as it exists today. It includes every major uncertainty between now and a possible catastrophe, including whether transformative AI is ever created.
A conditional probability starts later in the chain. It asks what happens assuming a specified condition is already true, such as:
- highly capable artificial general intelligence (AGI) exists;
- AI has reached or exceeded human-level performance across most cognitive tasks;
- superhuman AI systems are widely deployed.
The conditional estimate ignores uncertainty about whether that milestone will be reached. Instead, it focuses on what follows after it.
In simplified form:
Unconditional AI doom risk = probability that transformative AI is developed × probability of existential catastrophe given that it is developed
Real analyses usually include additional stages, such as governance failures, successful deployment, misuse and whether a catastrophe becomes irreversible. Even so, separating the first step from the later ones often clarifies where disagreements actually lie.[AI Impacts]wiki.aiimpacts.org2023 expert survey on progress in aiAI Impacts2023 Expert Survey on Progress in AI [AI Impacts Wiki]August 17, 2023…
Separating Capability Arrival From Catastrophe
Many public debates accidentally mix together two independent questions:
- Will transformative AI arrive?
- If it arrives, will humanity retain control?
These involve different evidence.
The first depends on questions such as:
- whether current machine learning approaches continue scaling successfully;
- engineering bottlenecks;
- economic incentives;
- hardware progress;
- scientific breakthroughs.
The second concerns different issues:
- whether advanced systems can be reliably aligned with human goals;
- whether deceptive or strategically aware behaviour emerges;
- whether governments and developers maintain effective oversight;
- whether dangerous capabilities spread uncontrollably;
- whether coordination failures produce unsafe deployment.
Because these are distinct uncertainties, changing one assumption can dramatically change overall p(doom) without changing beliefs about the other.
For example, someone who believes there is only a 20% chance of developing transformative AI this century but assigns a 50% chance of catastrophe if it occurs ends up with an unconditional extinction estimate of roughly 10%.
Someone else could believe there is a 90% chance of transformative AI arriving but only a 5% conditional chance of catastrophe. Their overall estimate is about 4.5%.
Although these individuals disagree strongly about timelines and alignment, their unconditional p(doom) values are surprisingly similar.
Why Identical Numbers Can Encode Different Beliefs
This is one reason published p(doom) figures can be misleading when presented without explanation.
Imagine two researchers who both report a 10% chance of AI extinction.
The first might believe:
- transformative AI is unlikely before 2100;
- if it is eventually built, humanity is in extreme danger.
The second might instead believe:
- transformative AI is almost inevitable within decades;
- alignment research and institutions are likely to succeed.
Both report 10%, but they disagree about almost every important mechanism.
Similarly, two people who share almost identical views about AI alignment may still report very different unconditional probabilities because they have different expectations about AI timelines.
The headline number therefore compresses several hidden assumptions into a single figure. Without unpacking those assumptions, comparisons between estimates can be more confusing than informative.[arXiv]arxiv.orgWhy do Experts Disagree on Existential Risk and P(doom)? A Survey of AI ExpertsJanuary 25, 2025…
Why This Matters When Reading Expert Surveys
Large surveys of AI researchers usually ask about unconditional risk over a specified period, such as the next 100 years.
For example, the 2023 Expert Survey on Progress in AI asked researchers about the probability that future AI advances would cause “human extinction or similarly permanent and severe disempowerment” within the next century. Different wording variants produced somewhat different results, illustrating how framing changes responses.[AI Impacts]wiki.aiimpacts.org2023 expert survey on progress in aiAI Impacts2023 Expert Survey on Progress in AI [AI Impacts Wiki]August 17, 2023…
These survey results should not be interpreted as direct measurements of conditional danger after AGI exists.
A researcher answering 5% might be expressing:
- scepticism that AGI arrives soon;
- confidence in future governance;
- confidence in technical alignment;
- optimism about human adaptation;[aiimpacts.org]aiimpacts.orgwhat do ml researchers think about ai in 2022what do ml researchers think about ai in 2022
- or some combination of all of these.
The survey number alone cannot distinguish among those possibilities.
For readers comparing expert opinions, this is one reason follow-up interviews and written explanations are often more informative than the headline percentage itself.
Why AI Doom Debates Often Talk Past Each Other
Many disagreements between AI doom advocates and sceptics are really disagreements about which probability is under discussion.
Some researchers primarily question whether current AI methods can ever produce transformative intelligence. If they assign a very low probability to that first step, their unconditional p(doom) will remain small even if they think an actual superintelligence would be extraordinarily dangerous.
Others accept rapid capability progress as likely and therefore concentrate on conditional questions such as:
- Can alignment scale to systems more intelligent than humans?
- Can deceptive behaviour be detected?
- Can developers reliably maintain human control?
- Will competitive pressures undermine safety?
Because these groups start from different assumptions, they may appear much farther apart than they actually are.
In many conversations, participants implicitly answer different questions without recognising it.
What Conditional Estimates Contribute
Conditional probabilities are particularly useful because they isolate the mechanisms that AI safety research aims to influence.
Alignment techniques, interpretability, evaluations, monitoring, secure deployment, governance arrangements and incident response primarily affect the probability of catastrophe given advanced AI exists, rather than the probability that AI research itself succeeds.
Thinking conditionally therefore helps clarify which interventions change which part of the overall risk equation.
For example:
- Better technical alignment mainly reduces conditional catastrophe risk.
- Slower capability development primarily changes the probability that highly capable AI is built within a given period.
- International coordination may affect both by slowing unsafe races while improving safety standards.
Breaking p(doom) into separate components makes it easier to see where evidence supports optimism and where important uncertainties remain.
Why Neither Probability Is the “Correct” One
Neither conditional nor unconditional risk is inherently superior.
An unconditional estimate is usually more relevant for public forecasting because it answers the practical question: from today, how likely is AI to produce existential catastrophe?
A conditional estimate is often more informative for analysing mechanisms because it asks: if transformative AI is achieved, how dangerous would that situation be?
Most serious discussions of AI extinction risk benefit from keeping both numbers conceptually separate. Doing so avoids treating disagreement over AI timelines as though it were disagreement over alignment, or vice versa. It also helps explain how two people can report similar p(doom) values while imagining fundamentally different futures, or report very different values despite agreeing on most technical questions about advanced AI.
Amazon book picks
Further Reading
Books and field guides related to Are We Estimating Doom From Today or After AGI?. Use these as the next step if you want deeper reading beyond the article.
Human Compatible
A leading artificial intelligence researcher lays out a new approach to AI that will enable us to coexist successfully with increasingly...
Superintelligence
This profoundly ambitious and original book picks its way carefully through a vast tract of forbiddingly difficult intellectual terrain.
Life 3.0
'This is the most important conversation of our time, and Tegmark's thought-provoking book will help you join it' Stephen Hawking THE INT...
The Alignment Problem
Finalist for the Los Angeles Times Book Prize A jaw-dropping exploration of everything that goes wrong when we build AI systems and the m...
eBay marketplace picks
Marketplace Samples
Live-tested eBay searches with available results related to this page.
Selected fromAI future poster oneBay.co.uk.
Endnotes
1.
Source: arxiv.org
Title: arXiv AI Survival Stories: a Taxonomic Analysis of AI Existential Risk
Link:https://arxiv.org/abs/2601.09765
2.
Source: arxiv.org
Link:https://arxiv.org/abs/2502.14870
Source snippet
Why do Experts Disagree on Existential Risk and P(doom)? A Survey of AI ExpertsJanuary 25, 2025...
Published: January 25, 2025
3.
Source: youtube.com
Link:https://www.youtube.com/watch?v=hliLDNdxkX0
Source snippet
AI Impacts Survey - The key implications, with Katja Grace...
4.
Source: youtube.com
Title: AI Impacts Survey
Link:https://www.youtube.com/watch?v=xwJx_xqZI3Q
Source snippet
Will AI end everything? A guide to guessing | Katja Grace | EAG Bay Area 23...
5.
Source: wiki.aiimpacts.org
Title: 2023 expert survey on progress in ai
Link:https://wiki.aiimpacts.org/ai_timelines/predictions_of_human-level_ai_timelines/ai_timeline_surveys/2023_expert_survey_on_progress_in_ai
Source snippet
AI Impacts2023 Expert Survey on Progress in AI [AI Impacts Wiki]August 17, 2023...
Published: August 17, 2023
6.
Source: blog.aiimpacts.org
Title: AI Impacts Blog FAQ: Expert Survey on Progress in AI methodology
Link:https://blog.aiimpacts.org/p/faq-expert-survey-on-progress-in
7.
Source: wiki.aiimpacts.org
Title: ai risk surveys
Link:https://wiki.aiimpacts.org/uncategorized/ai_risk_surveys
8.
Source: aiimpacts.org
Title: How bad a future do ML researchers expect? – AI Impacts
Link:https://aiimpacts.org/how-bad-a-future-do-ml-researchers-expect/
9.
Source: aiimpacts.org
Title: what do ml researchers think about ai in 2022
Link:https://aiimpacts.org/what-do-ml-researchers-think-about-ai-in-2022/
10.
Source: aiimpacts.org
Title: 2022 expert survey on progress in ai
Link:https://aiimpacts.org/2022-expert-survey-on-progress-in-ai/
Additional References
11.
Source: agirisk.com
Title: [Catastrophic]({{ ‘catastrophic-misuse/’ | relative_url }}) Scenarios — Concrete threat models from advanced AI | AGI Risk
Link:https://agirisk.com/scenarios
Source snippet
June 1, 2026 — A NOTE ON P(DOOM) AND HOW SERIOUSLY EXPERTS VARY "P(doom)" expresses extinction risk as a probability, and its defining fe...
Published: June 1, 2026
12.
Source: researchgate.net
Title: (PDF) Why do Experts Disagree on Existential Risk and P(doom)?
Link:https://www.researchgate.net/publication/389274220_Why_do_Experts_Disagree_on_Existential_Risk_and_Pdoom_A_Survey_of_AI_Experts
Source snippet
A Survey of AI ExpertsJanuary 25, 2025 — Preprint PDF Available WHY DO EXPERTS DISAGREE ON EXISTENTIAL RISK AND P(DOOM)? A SURVEY OF AI E...
Published: January 25, 2025
13.
Source: researchgate.net
Title: Why do experts disagree on existential risk?
Link:https://www.researchgate.net/publication/392941122_Why_do_experts_disagree_on_existential_risk_A_survey_of_AI_experts
Source snippet
A survey of AI expertsJune 23, 2025 — WHY DO EXPERTS DISAGREE ON EXISTENTIAL RISK? A SURVEY OF AI EXPERTS * June 2025 * AI and Ethics 5(6...
Published: June 23, 2025
14.
Source: spectrum.ieee.org
Title: ai existential risk survey 2667013207
Link:https://spectrum.ieee.org/amp/ai-existential-risk-survey-2667013207
Source snippet
the Prophecies of AI Doom - IEEE SpectrumJanuary 25, 2024 — WEIGHING THE PROPHECIES OF AI DOOM Researchers surveyed point to AI’s existen...
Published: January 25, 2024
15.
Source: youtube.com
Link:https://www.youtube.com/watch?v=Nbpw90WQmgU
Source snippet
This Harvard Professor Says AI Alignment Will BACKFIRE - Dr. Stephen Casper...
16.
Source: youtube.com
Title: Will AI end everything? A guide to guessing | Katja Grace | EAG Bay Area 23
Link:https://www.youtube.com/watch?v=j5Lu01pEDWA
Source snippet
Ep 13 - AI researchers expect AGI sooner w/ Katja Grace (Co-founder & Lead Researcher, AI Impacts)...
17.
Source: abc.net.au
Link:https://www.abc.net.au/news/2023-07-15/whats-your-pdoom-ai-researchers-worry-catastrophe/102591340
18.
Source: sfu.ca
Link:https://www.sfu.ca/~smith/pdoom_interactive_standalone.html
19.
Source: sciencedirect.com
Link:https://www.sciencedirect.com/science/article/pii/S0264999326002476
20.
Source: conjecture.dev
Link:https://www.conjecture.dev/research/conjecture-internal-survey-agi-timelines-and-probability-of-human-extinction-from-advanced-ai