Why AI Doom Odds Change With the Question

Introduction

Headline p(doom) figures often appear to offer direct comparisons between experts, surveys and time periods. In reality, small changes in the wording of a survey question can produce noticeably different estimates, even when the same broad issue is being discussed. That does not necessarily mean respondents are being inconsistent or irrational. It often reflects the fact that questions are asking about different outcomes, different causal pathways or different time horizons.[AI Impacts]wiki.aiimpacts.org2023 expert survey on progress in aiAI Impacts2023 Expert Survey on Progress in AI [AI Impacts Wiki]August 17, 2023…Published: August 17, 2023

For anyone trying to interpret AI doom estimates, the wording of the question is therefore almost as important as the number itself. A reported “5% chance of doom” and a reported “10% chance of doom” may not represent a genuine change in opinion if one survey asked about all AI-related existential catastrophes while another asked specifically about humanity losing control of advanced AI. Understanding these differences helps explain why published p(doom) figures should be treated as summaries of judgement rather than precise measurements.[AI Impacts]wiki.aiimpacts.org2023 expert survey on progress in aiAI Impacts2023 Expert Survey on Progress in AI [AI Impacts Wiki]August 17, 2023…Published: August 17, 2023

Extinction, disempowerment and loss of control

The clearest demonstration of wording sensitivity comes from the large 2023 Expert Survey on Progress in AI, which questioned thousands of AI researchers using randomly assigned variants of closely related questions. Rather than asking every respondent exactly the same question, researchers deliberately changed the wording to see whether estimates remained stable.[AI Impacts]wiki.aiimpacts.org2023 expert survey on progress in aiAI Impacts2023 Expert Survey on Progress in AI [AI Impacts Wiki]August 17, 2023…Published: August 17, 2023

The three closely related formulations produced different median estimates:

• Future AI advances causing human extinction or similarly permanent and severe disempowerment: median 5%.
• Human inability to control future advanced AI systems causing human extinction or similarly permanent and severe disempowerment: median 10%.
• Future AI advances causing extinction or similarly permanent disempowerment within the next 100 years: median 5%.[AI Impacts]wiki.aiimpacts.org2023 expert survey on progress in aiAI Impacts2023 Expert Survey on Progress in AI [AI Impacts Wiki]August 17, 2023…Published: August 17, 2023

At first glance, the second question appears narrower. If it focuses only on failures of human control, it might seem as though respondents should assign a lower probability than they would to all possible AI-related existential catastrophes.

Instead, the opposite occurred: the median doubled from 5% to 10%. Similar patterns also appeared in the 2022 survey.[AI Impacts]aiimpacts.org2022 expert survey on progress in aiAI Impacts2022 Expert Survey on Progress in AI – AI ImpactsAugust 3, 2022…Published: August 3, 2022

Researchers caution against reading too much into that difference. Because different random subsets of respondents received different versions of the question, some variation is expected simply from sampling. However, the result also suggests that many experts mentally associate existential AI risk primarily with loss of human control, making that framing especially salient. The survey authors note that representativeness effects—where a more concrete description better matches respondents’ mental model—may also contribute.[AI Impacts]aiimpacts.org2022 expert survey on progress in aiAI Impacts2022 Expert Survey on Progress in AI – AI ImpactsAugust 3, 2022…Published: August 3, 2022

The inclusion of “permanent and severe disempowerment” alongside extinction also matters. Many researchers who discuss AI existential risk argue that humanity could permanently lose meaningful control over civilisation without every human dying. A survey that counts irreversible disempowerment as existential risk will generally capture a broader class of scenarios than one referring only to literal extinction. Readers should therefore avoid comparing surveys unless they define their endpoint in similar ways.[AI Impacts Blog]blog.aiimpacts.orgAI Impacts Blog FAQ: Expert Survey on Progress in AI methodologyAI Impacts Blog FAQ: Expert Survey on Progress in AI methodology

How time horizons alter responses

Time limits also change what respondents believe they are estimating.

A question asking whether AI will eventually cause an existential catastrophe leaves open the possibility that such an event could occur centuries in the future. A question restricted to “within the next 100 years” asks something different: respondents must combine their beliefs about technological progress, governance and catastrophic failure within a specified period.

The 2023 survey included this explicit 100-year version precisely to measure whether the time horizon affected responses. Interestingly, the median remained at 5%, suggesting that, for many respondents, most of the perceived existential risk already lay within roughly the next century rather than being pushed into a distant future. The average estimate declined somewhat compared with the unrestricted version, indicating that some respondents did place additional probability beyond the century mark.[AI Impacts]wiki.aiimpacts.org2023 expert survey on progress in aiAI Impacts2023 Expert Survey on Progress in AI [AI Impacts Wiki]August 17, 2023…Published: August 17, 2023

This illustrates an important statistical point. Two people who both report a 5% p(doom) may be imagining completely different timelines:

• one may think nearly all of the risk arrives before 2100;
• another may think the first century is relatively safe but expects danger to accumulate over much longer periods.

Without the question wording, those differences disappear behind a single number.

What wording sensitivity means for headline figures

Several practical lessons follow.

Different surveys are not automatically comparable. One poll may ask about extinction alone, another about extinction or permanent disempowerment, another about “extremely bad outcomes”, and another about loss of control. Treating all resulting percentages as though they measured the same quantity is misleading.[AI Impacts]wiki.aiimpacts.orgAI Impacts Surveys of experts on levels of AI Risk [AI Impacts WikiAI Impacts Surveys of experts on levels of AI Risk [AI Impacts Wiki

A single percentage compresses several separate judgements. Respondents must decide what counts as “doom”, which mechanisms matter, how likely advanced AI is to be developed, and over what period they should imagine the risk unfolding. Changing any one of those assumptions can change the final estimate.

Means and medians tell different stories. In these surveys the median typically remains around 5%, while the mean is substantially higher because a minority of respondents assign very large probabilities to existential catastrophe. Headlines quoting only one statistic can therefore obscure the underlying distribution of opinion.[AI Impacts]wiki.aiimpacts.org2023 expert survey on progress in aiAI Impacts2023 Expert Survey on Progress in AI [AI Impacts Wiki]August 17, 2023…Published: August 17, 2023

Question wording reveals conceptual disagreement, not just numerical disagreement. Some researchers mainly worry about uncontrollable autonomous systems, others about broader AI-driven societal collapse, and others about multiple interacting pathways. Their answers naturally shift depending on which scenario the survey emphasises.

Interpreting p(doom) with appropriate caution

The strongest conclusion from the available evidence is not that survey-based p(doom) estimates are unreliable, but that they are definition-sensitive.

The 2023 expert survey deliberately randomised question wording to expose this effect rather than conceal it. Across multiple formulations, the central finding remained broadly consistent: a substantial proportion of AI researchers assigned a non-trivial probability to existential or similarly irreversible outcomes from advanced AI. At the same time, the precise percentage varied depending on whether respondents were asked about extinction, permanent disempowerment, loss of human control or a specified time horizon.[aiimpacts.org]wiki.aiimpacts.org2023 expert survey on progress in aiAI Impacts2023 Expert Survey on Progress in AI [AI Impacts Wiki]August 17, 2023…Published: August 17, 2023

For readers comparing headline p(doom) figures, the safest approach is therefore to read the question before the number. A reported probability only has a clear meaning when its outcome, causal pathway and time horizon are specified. Without that context, apparently precise percentages can suggest a level of comparability that the underlying surveys were never designed to provide.

Amazon book picks

Further Reading

Books and field guides related to Why AI Doom Odds Change With the Question. Use these as the next step if you want deeper reading beyond the article.

Book

Human Compatible

By Stuart Russell

A leading artificial intelligence researcher lays out a new approach to AI that will enable us to coexist successfully with increasingly...

See on Amazon

Book

The Alignment Problem

By Brian Christian

Finalist for the Los Angeles Times Book Prize A jaw-dropping exploration of everything that goes wrong when we build AI systems and the m...

See on Amazon

Book

Superforecasting

By Philip Tetlock, Dan Gardner

The international bestseller 'A manual for thinking clearly in an uncertain world. Read it.' Daniel Kahneman, author of Thinking, Fast an...

See on Amazon

Book

The Precipice

By Toby Ord

What existential threats does humanity face? And how can we secure our future?'The Precipice is a powerful book . . . Ord's love for huma...

See on Amazon

Browse more on Amazon:Human CompatibleThe Alignment ProblemSuperforecasting

As an Amazon Associate I earn from qualifying purchases.

eBay marketplace picks

Marketplace Samples

Live-tested eBay searches with available results related to this page.

Shop location

UsingUSA USA USAUKCanadaAustraliaIreland

Selected fromartificial intelligence poster oneBay.co.uk.

Current eBay listing

Artificial intelligence is no a mat Framed Wall Art Poster Canvas Print Picture

GBP 29.95 | Shipping GBP 4.95NewSeller: silkframe (99.7% positive)Ships from: GB

View listing on eBay

Current eBay listing

A.I. Artificial Intelligence Movie Film Poster Art Print

GBP 4.99 | Free shippingNewSeller: prestigealbums (100.0% positive)Ships from: GB

View listing on eBay

Current eBay listing

A.I. Artificial Intelligence - Jude Law - One Sheet Cinema Poster

GBP 10.79 | Shipping GBP 5.15UsedSeller: supertramp84 (100.0% positive)Ships from: GB

View listing on eBay

Current eBay listing

A. I. Artificial Intelligence. Jude Law. Original UK Video Poster.

GBP 8.11 | Shipping GBP 3.38UsedSeller: griseuk (100.0% positive)Ships from: GB

View listing on eBay

Browse more oneBay.co.uk

Example items shown for inspiration; availability and pricing can change. Branchoria may earn a commission if you purchase through outbound eBay links.

Endnotes

1. Source: wiki.aiimpacts.org
Title: 2023 expert survey on progress in ai
Link:https://wiki.aiimpacts.org/ai_timelines/predictions_of_human-level_ai_timelines/ai_timeline_surveys/2023_expert_survey_on_progress_in_ai

Published: August 17, 2023

2. Source: blog.aiimpacts.org
Title: AI Impacts Blog FAQ: Expert Survey on Progress in AI methodology
Link:https://blog.aiimpacts.org/p/faq-expert-survey-on-progress-in

3. Source: wiki.aiimpacts.org
Title: AI Impacts Surveys of experts on levels of AI Risk [AI Impacts Wiki]
Link:https://wiki.aiimpacts.org/uncategorized/ai_risk_surveys

4. Source: aiimpacts.org
Title: 2022 expert survey on progress in ai
Link:https://aiimpacts.org/2022-expert-survey-on-progress-in-ai/

Published: August 3, 2022

5. Source: wiki.aiimpacts.org
Title: 2022 expert survey on progress in ai
Link:https://wiki.aiimpacts.org/ai_timelines/predictions_of_human-level_ai_timelines/ai_timeline_surveys/2022_expert_survey_on_progress_in_ai

6. Source: lesswrong.com
Title: ai impacts 2023 expert survey on progress in ai
Link:https://www.lesswrong.com/posts/RkegCmCgjGhskiFvm/ai-impacts-2023-expert-survey-on-progress-in-ai

7. Source: blog.aiimpacts.org
Title: 2023 [ai survey]({{ ‘survey-bias/’ | relative_url }}) of 2778 six things
Link:https://blog.aiimpacts.org/p/2023-ai-survey-of-2778-six-things?action=share

8. Source: aiimpacts.org
Title: How bad a future do ML researchers expect? – AI Impacts
Link:https://aiimpacts.org/how-bad-a-future-do-ml-researchers-expect/

9. Source: wiki.aiimpacts.org
Title: surveys of us public opinion on ai
Link:https://wiki.aiimpacts.org/responses_to_ai/public_opinion_on_ai/surveys_of_public_opinion_on_ai/surveys_of_us_public_opinion_on_ai

10. Source: aiimpacts.org
Title: what do ml researchers think about ai in 2022
Link:https://aiimpacts.org/what-do-ml-researchers-think-about-ai-in-2022/

11. Source: youtube.com
Title: AI Impacts Survey
Link:https://www.youtube.com/watch?v=xwJx_xqZI3Q

Additional References

12. Source: GOV.UK
Link:https://www.gov.uk/government/publications/[international

13. Source: spectrum.ieee.org
Title: ai existential risk survey 2667013207
Link:https://spectrum.ieee.org/amp/ai-existential-risk-survey-2667013207

Published: January 25, 2024

14. Source: aiwiki.ai
Title: Existential risk from AI | AI Wiki
Link:https://aiwiki.ai/wiki/ai_existential_risk

15. Source: youtube.com
Link:https://www.youtube.com/watch?v=Nbpw90WQmgU

16. Source: youtube.com
Title: P(Doom) Estimates Shouldn’t Inform Policy?? Liron Reacts to Sayash Kapoor
Link:https://www.youtube.com/watch?v=vV6JKQ6p918

17. Source: youtube.com
Title: Will AI end everything? A guide to guessing | Katja Grace | EAG Bay Area 23
Link:https://www.youtube.com/watch?v=j5Lu01pEDWA

18. Source: weforum.org
Title: How do experts view the development of AI for the future? | World Economic Forum
Link:https://www.weforum.org/agenda/2023/02/experts-ai-developing-over-the-coming-years/

19. Source: weforum.org
Title: How do experts view the development of AI for the future? | World Economic Forum
Link:https://www.weforum.org/stories/2023/02/experts-ai-developing-over-the-coming-years/

20. Source: arxiv.org
Title: arXiv Managing extreme AI risks amid rapid progress
Link:https://arxiv.org/abs/2310.17688

21. Source: arxiv.org
Title: arXiv Thousands of AI Authors on the Future of AI
Link:https://arxiv.org/abs/2401.02843