Within Question Wording
Why Does Loss of Control Wording Raise p(doom)?
Questions that mention losing control can produce higher estimates because they match how many researchers picture existential AI risk.
On this page
- The 5% versus 10% wording result
- Why concrete scenarios can feel more representative
- Sampling variation and limits on interpretation
Page outline Jump by section
Introduction
Questions that explicitly mention humans losing control of advanced AI often produce higher p(doom) estimates than otherwise similar questions about AI causing an existential catastrophe. This does not necessarily mean respondents become more pessimistic simply because of different wording. Instead, the wording appears to activate a more specific mental model of how many AI researchers think an existential catastrophe would occur in the first place.
The clearest evidence comes from the large 2023 Expert Survey on Progress in AI, where randomly assigned versions of closely related questions produced different median estimates. A general question about AI causing “human extinction or similarly permanent and severe disempowerment” produced a median estimate of 5%, while a version that specifically referred to human inability to control future advanced AI systems produced a median of 10%. Similar results appeared in the 2022 survey. These findings illustrate how survey wording can influence reported p(doom) values without necessarily reflecting a genuine change in underlying beliefs.[AI Impacts]wiki.aiimpacts.org2023 expert survey on progress in aiAI Impacts2023 Expert Survey on Progress in AI [AI Impacts Wiki]August 17, 2023…
The 5% versus 10% wording result
The most frequently discussed example comes from the AI Impacts 2023 expert survey, which deliberately used random assignment to test whether slightly different question wording affected responses.
Researchers asked different subsets of participants one of three closely related questions:
- What is the probability that future AI advances cause human extinction or similarly permanent and severe disempowerment?
- What is the probability that human inability to control future advanced AI systems causes the same outcome?
- What is the probability of the general outcome occurring within the next 100 years?
The median responses were:
Question wordingMedian estimateGeneral AI-caused extinction or permanent disempowerment5%Loss of human control causing extinction or permanent disempowerment10%Same outcome within 100 years5%
The striking feature is that the loss-of-control question is logically narrower than the general question. Since it refers to one particular pathway rather than all possible AI-related existential catastrophes, simple probability theory might suggest it should receive an equal or lower estimate. Instead, respondents gave it a higher median estimate.[AI Impacts]wiki.aiimpacts.org2023 expert survey on progress in aiAI Impacts2023 Expert Survey on Progress in AI [AI Impacts Wiki]August 17, 2023…
The same qualitative pattern appeared in the 2022 survey, strengthening the case that this was not an isolated statistical accident.[AI Impacts]wiki.aiimpacts.org2022 expert survey on progress in aiAI Impacts2022 Expert Survey on Progress in AI [AI Impacts Wiki]…
Why concrete loss-of-control scenarios can seem more representative
One explanation offered by the survey authors is the representativeness heuristic.
In psychology, people often estimate probabilities by asking themselves how closely a description matches the example they have in mind, rather than by carefully reasoning from all possible cases. A concrete description can therefore feel more plausible than a broader, more abstract one.
Applied to AI existential risk, many researchers who assign non-trivial p(doom) estimates appear to picture scenarios involving:
- advanced systems pursuing objectives different from human intentions;
- humans gradually or suddenly losing the ability to supervise or constrain those systems;
- AI acquiring enough capability or autonomy that corrective intervention becomes ineffective.
When respondents read “human inability to control future advanced AI systems”, the wording closely matches this commonly discussed pathway. As a result, it may feel like a better description of the risk they already believe exists. The survey authors explicitly note representativeness as one possible explanation for the higher median estimate.[AI Impacts]wiki.aiimpacts.org2022 expert survey on progress in aiAI Impacts2022 Expert Survey on Progress in AI [AI Impacts Wiki]…
This does not prove that respondents believe loss of control is the only existential pathway. Rather, it suggests that, for many experts, it is the pathway that comes most readily to mind when thinking about catastrophic AI risk.
Why “loss of control” is central to many p(doom) arguments
The wording effect also reflects the structure of much modern AI safety research.
Many technical discussions of existential risk focus on problems such as:
- systems pursuing goals that differ from human intentions (misalignment);
- deceptive or strategically hidden behaviour during training;
- AI systems becoming increasingly capable of operating without human supervision;
- failures of oversight once systems exceed human ability to understand or direct them.
These discussions often treat loss of human control as the central mechanism connecting advanced AI capabilities to existential outcomes.
Consequently, when survey respondents encounter wording that explicitly mentions control failure, they may interpret it as describing the scenario they have spent the most time analysing. The more general wording “future AI advances causing…” leaves open many other causal pathways—some respondents may find that wording less specific, less vivid or less closely aligned with their own thinking.[arXiv]arxiv.orgWhy do Experts Disagree on Existential Risk and P(doom)? A Survey of AI ExpertsJanuary 25, 2025…
Higher estimates do not necessarily mean greater pessimism
It would be a mistake to conclude that the 10% figure demonstrates experts are twice as worried as the 5% figure.
Several alternative explanations are plausible.
First, the surveys randomly assigned different question variants to different groups of respondents. Because each participant answered only one version, sampling variation alone can produce differences between medians. The survey authors explicitly caution against assuming the observed difference reflects a genuine change in beliefs.[AI Impacts]wiki.aiimpacts.org2022 expert survey on progress in aiAI Impacts2022 Expert Survey on Progress in AI [AI Impacts Wiki]…
Second, survey questions do not merely measure beliefs—they also shape which mental models respondents retrieve. Different wording can prompt respondents to think about different scenarios, different historical analogies or different technical arguments.
Third, probability judgements about unprecedented events are inherently noisy. When people estimate extremely uncertain futures, even experts often show framing effects across questions that are logically related. The survey authors identify framing sensitivity as one of the important limitations of expert forecasting in this area.[AI Impacts Blog]blog.aiimpacts.orgAI Impacts Blog FAQ: Expert Survey on Progress in AI methodologyAI Impacts Blog FAQ: Expert Survey on Progress in AI methodology
The role of “permanent and severe disempowerment”
Another important feature of these questions is that they refer not only to human extinction, but also to similarly permanent and severe disempowerment of the human species.
This wording broadens the endpoint beyond literal extinction. Many AI safety researchers argue that humanity could permanently lose meaningful control over civilisation without every person dying. Examples sometimes discussed include irreversible domination by advanced AI systems or lasting loss of humanity’s ability to shape its own future.
The survey designers adopted this broader formulation because they were interested in outcomes that effectively end humanity’s long-term agency, not solely biological extinction. The accompanying methodological discussion notes that this means respondents may have been imagining somewhat different catastrophic scenarios, making direct comparisons with surveys that mention only extinction more difficult.[AI Impacts Blog]blog.aiimpacts.orgAI Impacts Blog FAQ: Expert Survey on Progress in AI methodologyAI Impacts Blog FAQ: Expert Survey on Progress in AI methodology
Sampling variation and limits on interpretation
The loss-of-control wording effect is informative, but it should not be over-interpreted.
Several caveats matter:
- Different respondents answered different versions. Random assignment reduces systematic bias but does not eliminate ordinary sampling variation.
- The estimates remain highly uncertain. Individual responses span a very wide range, from effectively zero to very high probabilities.
- Question wording captures mental models as well as beliefs. A more concrete description may simply match how respondents naturally think about AI existential risk.
- One survey cannot establish causal beliefs. The results show how estimates change with wording, not whether loss of control is objectively more likely than alternative pathways.
For these reasons, the 5% versus 10% comparison is best understood as evidence about survey interpretation, not as definitive evidence that experts believe loss-of-control scenarios are exactly twice as probable as AI existential risk in general. The main lesson is methodological: p(doom) figures depend not only on who is surveyed, but also on precisely what question they believe they are answering.[aiimpacts.org]blog.aiimpacts.orgAI Impacts Blog FAQ: Expert Survey on Progress in AI methodologyAI Impacts Blog FAQ: Expert Survey on Progress in AI methodology
Amazon book picks
Further Reading
Books and field guides related to Why Does Loss of Control Wording Raise p(doom)?. Use these as the next step if you want deeper reading beyond the article.
Human Compatible
A leading artificial intelligence researcher lays out a new approach to AI that will enable us to coexist successfully with increasingly...
The Alignment Problem
Finalist for the Los Angeles Times Book Prize A jaw-dropping exploration of everything that goes wrong when we build AI systems and the m...
Life 3.0
'This is the most important conversation of our time, and Tegmark's thought-provoking book will help you join it' Stephen Hawking THE INT...
The Precipice
What existential threats does humanity face? And how can we secure our future?'The Precipice is a powerful book . . . Ord's love for huma...
eBay marketplace picks
Marketplace Samples
Live-tested eBay searches with available results related to this page.
Selected fromAI safety wall art oneBay.co.uk.
Endnotes
1.
Source: wiki.aiimpacts.org
Title: 2023 expert survey on progress in ai
Link:https://wiki.aiimpacts.org/ai_timelines/predictions_of_human-level_ai_timelines/ai_timeline_surveys/2023_expert_survey_on_progress_in_ai
Source snippet
AI Impacts2023 Expert Survey on Progress in AI [AI Impacts Wiki]August 17, 2023...
Published: August 17, 2023
2.
Source: wiki.aiimpacts.org
Title: 2022 expert survey on progress in ai
Link:https://wiki.aiimpacts.org/ai_timelines/predictions_of_human-level_ai_timelines/ai_timeline_surveys/2022_expert_survey_on_progress_in_ai
Source snippet
AI Impacts2022 Expert Survey on Progress in AI [AI Impacts Wiki]...
3.
Source: blog.aiimpacts.org
Title: AI Impacts Blog FAQ: Expert Survey on Progress in AI methodology
Link:https://blog.aiimpacts.org/p/faq-expert-survey-on-progress-in
4.
Source: aiimpacts.org
Title: 2022 expert survey on progress in ai
Link:https://aiimpacts.org/2022-expert-survey-on-progress-in-ai/
Source snippet
AI Impacts2022 Expert Survey on Progress in AI – AI Impacts...
5.
Source: arxiv.org
Link:https://arxiv.org/abs/2502.14870
Source snippet
Why do Experts Disagree on Existential Risk and P(doom)? A Survey of AI ExpertsJanuary 25, 2025...
Published: January 25, 2025
6.
Source: lesswrong.com
Title: ai impacts survey december 2023 edition
Link:https://www.lesswrong.com/posts/NfPxAp5uwgZugwovY/ai-impacts-survey-december-2023-edition
Published: december 2023
7.
Source: wiki.aiimpacts.org
Title: ai risk surveys
Link:https://wiki.aiimpacts.org/uncategorized/ai_risk_surveys
8.
Source: grjenkin.com
Title: 2022 expert survey on progress in ai
Link:https://grjenkin.com/articles/category/machine-learning/6892881/03/20/2023/2022-expert-survey-on-progress-in-ai
9.
Source: aiimpacts.org
Title: How bad a future do ML researchers expect? – AI Impacts
Link:https://aiimpacts.org/how-bad-a-future-do-ml-researchers-expect/
10.
Source: aiimpacts.org
Title: what do ml researchers think about ai in 2022
Link:https://aiimpacts.org/what-do-ml-researchers-think-about-ai-in-2022/
11.
Source: youtube.com
Title: AI Impacts Survey
Link:https://www.youtube.com/watch?v=xwJx_xqZI3Q
Source snippet
Surveying 2700+ AI Researchers on the Industry's Future with Katja Grace of AI Impacts...
Additional References
12.
Source: pmc.ncbi.nlm.nih.gov
Link:https://pmc.ncbi.nlm.nih.gov/articles/PMC12500108/
Source snippet
US officials’ views on the impacts and governance of AI: Evidence from 2022 and 2023 survey waves - PMCOctober 6, 2025 — Second, any surv...
Published: October 6, 2025
13.
Source: GOV.UK
Link:https://www.gov.uk/government/publications/[international
Source snippet
LOSS OF CONTROL KEY INFORMATION * Ongoing AI (artificial [intelligence]({{ 'hard-bottlenecks/' | relative_url }})) research is seeking to develop more capable ‘general-purpose AI (a...
14.
Source: GOV.UK
Title: Some experts are concerned that
Link:https://www.gov.uk/government/publications/frontier-ai-capabilities-and-risks-discussion-paper/frontier-ai-capabilities-and-risks-discussion-paper
Source snippet
AI: capabilities and risks – discussion paper - GOV.UKApril 28, 2025 — LOSS OF CONTROL Humans may increasingly hand over control of impor...
Published: April 28, 2025
15.
Source: youtube.com
Title: Will AI end everything? A guide to guessing | Katja Grace | EAG Bay Area 23
Link:https://www.youtube.com/watch?v=j5Lu01pEDWA
Source snippet
Dr. Mike Israetel Returns to DEBATE: AI's Gonna Kill Everyone vs. AI Will Make Everything Awesome...
16.
Source: forum.effectivealtruism.org
Title: faq expert survey on progress in ai methodology
Link:https://forum.effectivealtruism.org/posts/c977usNg2AmF2m7ud/faq-expert-survey-on-progress-in-ai-methodology
Source snippet
No. We published the near-identical 2016 survey in the Journal of AI Research, so the methodology had essentially...
17.
Source: youtube.com
Link:https://www.youtube.com/watch?v=hliLDNdxkX0
Source snippet
Is p(doom) bullsh*t?...
18.
Source: youtube.com
Title: Is p(doom) bullsh*t?
Link:https://www.youtube.com/watch?v=gyOGhnEnGkk
Source snippet
Will AI end everything? A guide to guessing | Katja Grace | EAG Bay Area 23...
19.
Source: youtube.com
Link:https://www.youtube.com/watch?v=WEMqmG1T00I
20.
Source: GOV.UK
Title: www.gov.uk Public attitudes to data and AI: Tracker survey (Wave 3)
Link:https://www.gov.uk/government/publications/public-attitudes-to-data-and-ai-tracker-survey-wave-3/public-attitudes-to-data-and-ai-tracker-survey-wave-3
21.
Source: bankofengland.co.uk
Title: Machine learning in UK financial services | Bank of England
Link:https://www.bankofengland.co.uk/report/2022/machine-learning-in-uk-financial-services