Within Conditional Risk
What Does a 5% AI Extinction Estimate Mean?
Headline survey percentages usually mix beliefs about AI timelines, alignment, governance and adaptation rather than isolating one danger.
On this page
- How major surveys frame extinction and severe disempowerment
- Why wording and time horizons change responses
- What a single percentage cannot reveal about expert beliefs
Page outline Jump by section
Introduction
When headlines report that “AI experts put the risk of extinction at 5%”, it is easy to assume that researchers are expressing a single, well-defined probability. They are not. Most expert surveys combine several different judgements about the future: whether very advanced AI will be built, what kinds of systems are being imagined, how capable they become, whether humans can retain control, how governments respond, and what counts as an existential catastrophe.
That makes expert surveys valuable, but also easy to misinterpret. A quoted percentage is usually best understood as a summary of many assumptions rather than a direct measurement of one specific danger. This matters for debates about AI doom and p(doom), because people often compare survey numbers without noticing that the underlying questions differ in wording, time horizon and scope.
How major surveys frame extinction and severe disempowerment
The most widely cited evidence comes from the AI Impacts Expert Survey on Progress in AI, which periodically surveys researchers who have published at leading AI conferences. These surveys ask several related—but importantly different—questions rather than a single “probability of AI doom”.[AI Impacts]aiimpacts.org2022 expert survey on progress in aiAI Impacts2022 Expert Survey on Progress in AI – AI ImpactsAugust 3, 2022…
One commonly quoted finding is that the median respondent assigned around a 5% probability to “future AI advances causing human extinction or similarly permanent and severe disempowerment of the human species.” The survey deliberately paired extinction with another outcome—permanent severe disempowerment—rather than asking about extinction alone.[AI Impacts]aiimpacts.org2022 expert survey on progress in aiAI Impacts2022 Expert Survey on Progress in AI – AI ImpactsAugust 3, 2022…
That wording reflects an important idea in existential-risk research. Some researchers argue that humanity could permanently lose control over its future without literally becoming extinct. Examples discussed in the literature include:
- irreversible domination by autonomous AI systems;
- permanent loss of meaningful human decision-making;
- a stable future in which humanity survives biologically but no longer determines civilisation’s trajectory.
By including both extinction and severe disempowerment, the survey measures a broader class of existential outcomes than extinction alone.[AI Impacts]aiimpacts.org2022 expert survey on progress in aiAI Impacts2022 Expert Survey on Progress in AI – AI ImpactsAugust 3, 2022…
The same survey also asks about the overall long-run impact of advanced AI, ranging from “extremely good” to “extremely bad”. Those responses are conceptually different again. Someone could believe AI has a 5% chance of existential catastrophe while still believing overwhelmingly positive outcomes are more likely overall.[AI Impacts]aiimpacts.org2022 expert survey on progress in aiAI Impacts2022 Expert Survey on Progress in AI – AI ImpactsAugust 3, 2022…
Why wording and time horizons change responses
Small changes in survey wording can produce noticeably different answers because they ask subtly different questions.
For example, AI Impacts randomly assigned different respondents different versions of existential-risk questions. One version asked about:
future AI advances causing human extinction or similarly permanent severe disempowerment.
Another instead asked about:
human inability to control future advanced AI systems causing human extinction or similarly permanent severe disempowerment.
Surprisingly, the second version produced a higher median estimate (10%) than the broader first question (5%). Since inability to control AI is logically only one possible pathway to catastrophe, this seems counterintuitive. The survey authors caution against over-interpreting the difference because different random groups answered different versions, and the result could reflect sampling variation or differences in how respondents mentally interpreted the questions.[AI Impacts]aiimpacts.org2022 expert survey on progress in aiAI Impacts2022 Expert Survey on Progress in AI – AI ImpactsAugust 3, 2022…
Time horizon also matters.
Some survey variants ask about existential catastrophe ever, while others specify within the next 100 years. A probability over an unlimited future cannot be directly compared with one restricted to a century. Likewise, asking about “advanced AI” differs from asking about “high-level machine intelligence” or “artificial general intelligence”, because respondents may imagine different technological thresholds.[AI Impacts Blog]blog.aiimpacts.orgAI Impacts Blog FAQ: Expert Survey on Progress in AI methodologyAI Impacts Blog FAQ: Expert Survey on Progress in AI methodology
These framing differences mean apparently conflicting percentages may not represent genuine disagreement. Experts may simply be answering different questions.
Why a single percentage cannot reveal expert beliefs
A headline figure such as 5% conceals many independent beliefs.
An expert reaching that number might believe:
- advanced AI is highly likely to arrive this century;
- alignment remains technically difficult;
- governance will improve significantly;
- catastrophic misuse is possible but not dominant;
- existential catastrophe is unlikely despite serious challenges.
Another expert could reach exactly the same 5% by holding almost opposite assumptions:
- transformative AI may never be developed;
- if it is developed, loss of control could be extremely likely.
The identical numerical answer therefore hides very different models of the future.
This is especially relevant when comparing conditional and unconditional extinction risk. Many surveys do not explicitly separate:
- the probability that transformative AI is developed; and
- the probability of catastrophe if such AI exists.
As a result, survey responses often blend beliefs about technological progress with beliefs about alignment, institutions and human adaptation.
Surveys measure uncertainty as much as consensus
Another common misunderstanding is that surveys identify a consensus estimate.
In reality, AI expert surveys consistently show very wide disagreement.
The 2022 AI Impacts survey reported a median existential-risk estimate of 5%, but the mean was substantially higher because some respondents assigned much larger probabilities. Large fractions of respondents assigned at least a 10% chance to existential outcomes, while others assigned probabilities close to zero.[AI Impacts]aiimpacts.org2022 expert survey on progress in aiAI Impacts2022 Expert Survey on Progress in AI – AI ImpactsAugust 3, 2022…
The larger 2023 survey of more than 2,700 AI researchers found similar diversity. Most respondents expected advanced AI to be beneficial overall, yet sizeable minorities still assigned meaningful probabilities to extremely bad outcomes. Depending on the exact question, roughly 38%–51% assigned at least a 10% probability to outcomes as bad as human extinction.[arXiv]arxiv.orgarXiv Thousands of AI Authors on the Future of AIarXiv Thousands of AI Authors on the Future of AI
This spread is itself informative. Rather than demonstrating settled scientific agreement, the surveys suggest that substantial uncertainty remains even among experienced AI researchers.
What surveys cannot tell us
Expert surveys are useful evidence, but they cannot answer every question relevant to AI doom.
They generally do not reveal:
- exactly why each expert gave their probability;
- which failure scenarios they considered most plausible;
- whether they assumed rapid or slow AI progress;
- how much confidence they have in their own estimate;
- which interventions they believe would most reduce risk.
Nor should survey medians be interpreted as empirical measurements comparable to measured physical quantities. They are aggregated subjective judgements about unprecedented future events.
Methodological questions also matter. Response rates are incomplete, samples are drawn from particular research communities rather than all AI practitioners, and survey designers continually examine possible sources of selection bias. The organisers argue that invitation procedures and question randomisation reduce some important biases, but they also acknowledge unavoidable limitations common to expert elicitation.[AI Impacts Blog]blog.aiimpacts.orgAI Impacts Blog FAQ: Expert Survey on Progress in AI methodologyAI Impacts Blog FAQ: Expert Survey on Progress in AI methodology
How expert surveys should be interpreted in AI doom debates
Expert surveys provide an important snapshot of informed opinion, but they are best read as evidence about uncertainty rather than precise forecasts.
When someone quotes a figure such as “5% chance of AI extinction”, the most useful questions are:
- Exactly what question were experts asked?
- Was the estimate conditional or unconditional?
- Did it include severe disempowerment as well as extinction?
- What time horizon was specified?
- How much disagreement existed across respondents?
Those questions often matter more than the headline percentage itself.
Within debates over AI doom, the strongest use of expert surveys is therefore not to claim that scientists have measured the true probability of extinction. Instead, they demonstrate that a substantial number of leading AI researchers regard existential catastrophe as plausible enough to assign non-trivial probabilities, while also revealing deep disagreement about the assumptions, mechanisms and future pathways that those probabilities are intended to represent.
Amazon book picks
Further Reading
Books and field guides related to What Does a 5% AI Extinction Estimate Mean?. Use these as the next step if you want deeper reading beyond the article.
Human Compatible
A leading artificial intelligence researcher lays out a new approach to AI that will enable us to coexist successfully with increasingly...
The Alignment Problem
Finalist for the Los Angeles Times Book Prize A jaw-dropping exploration of everything that goes wrong when we build AI systems and the m...
Superforecasting
The international bestseller 'A manual for thinking clearly in an uncertain world. Read it.' Daniel Kahneman, author of Thinking, Fast an...
The Precipice
What existential threats does humanity face? And how can we secure our future?'The Precipice is a powerful book . . . Ord's love for huma...
eBay marketplace picks
Marketplace Samples
Live-tested eBay searches with available results related to this page.
Selected fromartificial intelligence poster oneBay.co.uk.
Endnotes
1.
Source: aiimpacts.org
Title: 2022 expert survey on progress in ai
Link:https://aiimpacts.org/2022-expert-survey-on-progress-in-ai/
Source snippet
AI Impacts2022 Expert Survey on Progress in AI – AI ImpactsAugust 3, 2022...
Published: August 3, 2022
2.
Source: blog.aiimpacts.org
Title: AI Impacts Blog FAQ: Expert Survey on Progress in AI methodology
Link:https://blog.aiimpacts.org/p/faq-expert-survey-on-progress-in
3.
Source: wiki.aiimpacts.org
Title: AI Impacts Surveys of experts on levels of AI Risk [AI Impacts Wiki]
Link:https://wiki.aiimpacts.org/uncategorized/ai_risk_surveys
4.
Source: arxiv.org
Title: arXiv Thousands of AI Authors on the Future of AI
Link:https://arxiv.org/abs/2401.02843
5.
Source: wiki.aiimpacts.org
Title: 2022 expert survey on progress in ai
Link:https://wiki.aiimpacts.org/ai_timelines/predictions_of_human-level_ai_timelines/ai_timeline_surveys/2022_expert_survey_on_progress_in_ai
Source snippet
AI Impacts2022 Expert Survey on Progress in AI [AI Impacts Wiki]...
6.
Source: lesswrong.com
Title: ai impacts 2023 expert survey on progress in ai
Link:https://www.lesswrong.com/posts/RkegCmCgjGhskiFvm/ai-impacts-2023-expert-survey-on-progress-in-ai
7.
Source: lesswrong.com
Title: ai impacts survey december 2023 edition
Link:https://www.lesswrong.com/posts/NfPxAp5uwgZugwovY/ai-impacts-survey-december-2023-edition
Published: december 2023
8.
Source: blog.aiimpacts.org
Title: 2023 [ai survey]({{ ‘survey-bias/’ | relative_url }}) of 2778 six things
Link:https://blog.aiimpacts.org/p/2023-ai-survey-of-2778-six-things?action=share
9.
Source: wiki.aiimpacts.org
Title: 2023 expert survey on progress in ai
Link:https://wiki.aiimpacts.org/ai_timelines/predictions_of_human-level_ai_timelines/ai_timeline_surveys/2023_expert_survey_on_progress_in_ai
10.
Source: aiimpacts.org
Title: How bad a future do ML researchers expect? – AI Impacts
Link:https://aiimpacts.org/how-bad-a-future-do-ml-researchers-expect/
11.
Source: aiimpacts.org
Title: what do ml researchers think about ai in 2022
Link:https://aiimpacts.org/what-do-ml-researchers-think-about-ai-in-2022/
12.
Source: aiimpacts.org
Link:https://aiimpacts.org/2022/08/
13.
Source: youtube.com
Title: Katja Grace on the Largest Survey of AI Researchers
Link:https://www.youtube.com/watch?v=qnqdTAO5OXw
Source snippet
AI Impacts Survey - The key implications, with Katja Grace...
14.
Source: youtube.com
Title: AI Impacts Survey
Link:https://www.youtube.com/watch?v=xwJx_xqZI3Q
Source snippet
Will AI end everything? A guide to guessing | Katja Grace | EAG Bay Area 23...
Additional References
15.
Source: survey160.com
Title: the publics concerns about ai and probability of doom
Link:https://www.survey160.com/methodological-research-blog/the-publics-concerns-about-ai-and-probability-of-doom
Source snippet
The Public’s Concerns about AI (and “Probability of Doom”) — Survey 160March 3, 2025 — THE PUBLIC’S CONCERNS ABOUT AI (AND “PROBABILITY O...
Published: March 3, 2025
16.
Source: researchgate.net
Title: Why do experts disagree on existential risk?
Link:https://www.researchgate.net/publication/392941122_Why_do_experts_disagree_on_existential_risk_A_survey_of_AI_experts
Source snippet
A survey of AI expertsJune 23, 2025 — WHY DO EXPERTS DISAGREE ON EXISTENTIAL RISK? A SURVEY OF AI EXPERTS * June 2025 * AI and Ethics 5(6...
Published: June 23, 2025
17.
Source: forum.effectivealtruism.org
Title: faq expert survey on progress in ai methodology
Link:https://forum.effectivealtruism.org/posts/c977usNg2AmF2m7ud/faq-expert-survey-on-progress-in-ai-methodology
Source snippet
Respondents are allocated randomly to one place in each horizontal set of blocks. DID ONLY A HANDFUL OF PEOPLE ANSWER EXTINCTION...
18.
Source: aiwiki.ai
Title: Existential risk from AI | AI Wiki
Link:https://aiwiki.ai/wiki/ai_existential_risk
Source snippet
SURVEY DATA The 2023 Expert Survey on Progress in AI (ESPAI), conducted by AI Impacts, collected responses from 2,778 AI researchers who...
19.
Source: youtube.com
Link:https://www.youtube.com/watch?v=017SSkPbJVs
Source snippet
Katja Grace AI Impacts survey expert AI risk Katja Grace—Slowing Down AI, Forecasting AI Risk The Inside View...
20.
Source: youtube.com
Title: Will AI end everything? A guide to guessing | Katja Grace | EAG Bay Area 23
Link:https://www.youtube.com/watch?v=j5Lu01pEDWA
Source snippet
Katja Grace—Slowing Down AI, Forecasting AI Risk...
21.
Source: youtube.com
Title: Katja Grace—Slowing Down AI, Forecasting AI Risk
Link:https://www.youtube.com/watch?v=rSw3UVDZge0
Source snippet
313 - Guest: Katja Grace, AI Impact Researcher, part 1...
22.
Source: sfu.ca
Link:https://www.sfu.ca/~smith/pdoom_interactive_standalone.html
23.
Source: cambridge.org
Link:https://www.cambridge.org/core/journals/political-analysis/article/synthetic-replacements-for-human-survey-data-the-perils-of-large-language-models/B92267DC26195C7F36E63EA04A47D2FE
24.
Source: researchgate.net
Link:https://www.researchgate.net/publication/389274220_Why_do_Experts_Disagree_on_Existential_Risk_and_Pdoom_A_Survey_of_AI_Experts