Does p(doom) Mean Extinction or Lost Control?

Introduction

A single p(doom) figure is only meaningful if you know what counts as “doom”. Some surveys ask about human extinction, while others ask about human extinction or similarly permanent and severe disempowerment. Those are not identical endpoints. A respondent who believes humanity could permanently lose control of civilisation without every human dying may give a much higher probability when disempowerment is included than when extinction alone is implied. Conversely, some readers mentally equate “AI doom” only with literal human extinction, leading them to misinterpret survey results.

This distinction matters because many modern discussions of AI existential risk deliberately define an existential catastrophe more broadly than the death of every human being. When comparing published p(doom) estimates, the first question should therefore be: what outcome was respondents actually asked to estimate? Surveys using different definitions cannot be merged into a single comparable number without losing important information.[AI Impacts]wiki.aiimpacts.org2023 expert survey on progress in aiAI Impacts2023 Expert Survey on Progress in AI [AI Impacts Wiki]August 17, 2023…Published: August 17, 2023

Does p(doom) Mean Extinction or Lost Control?

In everyday language, extinction means that humanity ceases to exist. Within parts of the AI safety literature, however, researchers often use a wider concept of existential catastrophe: any outcome that permanently destroys humanity’s long-term potential, even if some humans survive.

That broader framing includes scenarios where advanced AI systems permanently dominate political, economic or technological decision-making, leaving humans unable to regain meaningful control over civilisation. These are often described as forms of permanent disempowerment rather than extinction.

For survey designers, the distinction is important because respondents may picture different futures depending on the wording:

• Extinction only encourages respondents to think about scenarios where humans literally die out.
• Extinction or permanent disempowerment also includes futures where humans survive but permanently lose agency, power or control over civilisation.
• Loss of control wording may additionally direct respondents towards particular mechanisms, such as misaligned advanced AI systems escaping effective human oversight.

These differences affect what probability respondents believe they are estimating rather than merely how optimistic or pessimistic they are.[AI Impacts Blog]blog.aiimpacts.orgAI Impacts Blog FAQ: Expert Survey on Progress in AI methodologyAI Impacts Blog FAQ: Expert Survey on Progress in AI methodology

Literal Extinction Versus Permanent Disempowerment

The inclusion of permanent disempowerment reflects a longstanding argument within existential-risk research that an irreversible loss of humanity’s future need not require every human being to die.

Examples discussed in the literature include:

Whether these outcomes are plausible is heavily debated. Critics question whether such scenarios are technically achievable or politically realistic. Supporters argue that, if they occurred and could never be reversed, they would represent an existential catastrophe even without biological extinction. The disagreement is therefore often about what qualifies as an existential endpoint, not merely about its probability.[arXiv]arxiv.orgarXiv Is Power-Seeking AI an Existential Risk?arXiv Is Power-Seeking AI an Existential Risk?

Which Catastrophic Futures Each Definition Includes

Changing a survey’s endpoint changes the set of futures respondents are asked to consider.

Survey definitionTypically includesHuman extinctionDeath of humanity through AI-related causes.Extinction or permanent disempowermentExtinction plus irreversible loss of humanity’s ability to shape civilisation’s future.Loss of human control causing extinction or disempowermentA narrower causal pathway focused specifically on failures of human control over advanced AI.

This means two respondents can both answer “5%” while imagining entirely different futures. One may be thinking exclusively about literal extinction. Another may be assigning most of that probability to permanent loss of human control with surviving human populations.

As a result, identical numerical answers do not necessarily represent identical beliefs.[AI Impacts]wiki.aiimpacts.org2023 expert survey on progress in aiAI Impacts2023 Expert Survey on Progress in AI [AI Impacts Wiki]August 17, 2023…Published: August 17, 2023

Why the Wording Produces Different Survey Results

The AI Impacts expert surveys provide one of the clearest demonstrations that endpoint definitions matter.

In the 2023 survey, researchers randomly assigned closely related versions of existential-risk questions to different respondents. The median estimates differed:

• Future AI advances causing human extinction or similarly permanent and severe disempowerment: 5% median.
• Human inability to control future advanced AI systems causing human extinction or similarly permanent and severe disempowerment: 10% median.
• Future AI advances causing extinction or similarly permanent disempowerment within the next 100 years: 5% median.[AI Impacts]wiki.aiimpacts.org2023 expert survey on progress in aiAI Impacts2023 Expert Survey on Progress in AI [AI Impacts Wiki]August 17, 2023…Published: August 17, 2023

The 2022 survey showed the same broad pattern, with the loss-of-control wording again producing a higher median than the more general wording.[AI Impacts]wiki.aiimpacts.org2022 expert survey on progress in aiAI Impacts2022 Expert Survey on Progress in AI [AI Impacts Wiki]August 4, 2022…Published: August 4, 2022

Researchers caution against overinterpreting these differences. Because different random subsets of experts answered each version, sampling variation could explain part of the gap. They also suggest a psychological explanation: more concrete wording about human inability to control advanced AI may better match the mental model many respondents already associate with existential AI risk, making that scenario more salient when estimating probabilities.[AI Impacts]aiimpacts.org2022 expert survey on progress in aiAI Impacts2022 Expert Survey on Progress in AI – AI Impacts…

Why Surveys with Different Endpoints Cannot Be Merged

A common mistake is to compare p(doom) numbers from different surveys as though they all measured the same underlying quantity.

In reality, several dimensions may differ simultaneously:

Because each wording changes the event being estimated, combining results into a single average can be misleading. A reported 5% from one survey and 10% from another may reflect different definitions rather than a genuine disagreement about the likelihood of the same event.[AI Impacts]wiki.aiimpacts.org2023 expert survey on progress in aiAI Impacts2023 Expert Survey on Progress in AI [AI Impacts Wiki]August 17, 2023…Published: August 17, 2023

A Practical Rule for Reading p(doom) Surveys

When encountering an AI doom estimate, it helps to identify three features before interpreting the number:

Without those details, a single percentage can easily be misunderstood.

For this reason, many survey researchers deliberately publish the exact wording alongside headline results. The number alone is not the whole finding; the definition of “doom” is an essential part of what respondents were actually estimating.[AI Impacts Blog]blog.aiimpacts.orgAI Impacts Blog FAQ: Expert Survey on Progress in AI methodologyAI Impacts Blog FAQ: Expert Survey on Progress in AI methodology

Amazon book picks

Further Reading

Books and field guides related to Does p(doom) Mean Extinction or Lost Control?. Use these as the next step if you want deeper reading beyond the article.

Book

Human Compatible

By Stuart Russell

A leading artificial intelligence researcher lays out a new approach to AI that will enable us to coexist successfully with increasingly...

See on Amazon

Book

The Alignment Problem

By Brian Christian

Finalist for the Los Angeles Times Book Prize A jaw-dropping exploration of everything that goes wrong when we build AI systems and the m...

See on Amazon

Book

The Precipice

By Toby Ord

What existential threats does humanity face? And how can we secure our future?'The Precipice is a powerful book . . . Ord's love for huma...

See on Amazon

Book

Our Final Invention

By James Barrat

'I wish it was science fiction, but I know it's not' Jaan Tallinn, co-founder of Skype 'If you read just one book that makes you confront...

See on Amazon

Browse more on Amazon:Human CompatibleThe Alignment ProblemThe Precipice

As an Amazon Associate I earn from qualifying purchases.

eBay marketplace picks

Marketplace Samples

Live-tested eBay searches with available results related to this page.

Shop location

UsingUSA USA USAUKCanadaAustraliaIreland

Selected fromrobot control wall art oneBay.co.uk.

Current eBay listing

Control Is An Illusion Robot AI Art Framed Wall Art Poster Canvas Print Picture

GBP 8.99 | Shipping GBP 4.95NewSeller: nobleart (98.7% positive)Ships from: GB

View listing on eBay

Current eBay listing

Giant Sci-Fi Robot in High-Tech Control Room Poster 18x24in

GBP 16.87 | Shipping GBP 12.79NewSeller: 3dvoprinting (98.7% positive)Ships from: US

View listing on eBay

Current eBay listing

In Control Eindhoven Strip Robot Toy Toys Netherlands Doll Wall Art Print

GBP 15.99 | Shipping GBP 0.99NewSeller: assured-supplies (99.4% positive)Ships from: GB

View listing on eBay

Browse more oneBay.co.uk

Example items shown for inspiration; availability and pricing can change. Branchoria may earn a commission if you purchase through outbound eBay links.

Endnotes

1. Source: wiki.aiimpacts.org
Title: 2023 expert survey on progress in ai
Link:https://wiki.aiimpacts.org/ai_timelines/predictions_of_human-level_ai_timelines/ai_timeline_surveys/2023_expert_survey_on_progress_in_ai

Published: August 17, 2023

2. Source: blog.aiimpacts.org
Title: AI Impacts Blog FAQ: Expert Survey on Progress in AI methodology
Link:https://blog.aiimpacts.org/p/faq-expert-survey-on-progress-in

3. Source: arxiv.org
Title: arXiv Is Power-Seeking AI an Existential Risk?
Link:https://arxiv.org/abs/2206.13353

4. Source: wiki.aiimpacts.org
Title: 2022 expert survey on progress in ai
Link:https://wiki.aiimpacts.org/ai_timelines/predictions_of_human-level_ai_timelines/ai_timeline_surveys/2022_expert_survey_on_progress_in_ai

Published: August 4, 2022

5. Source: aiimpacts.org
Title: 2022 expert survey on progress in ai
Link:https://aiimpacts.org/2022-expert-survey-on-progress-in-ai/

6. Source: blog.aiimpacts.org
Title: 2023 [ai survey]({{ ‘survey-bias/’ | relative_url }}) of 2778 six things
Link:https://blog.aiimpacts.org/p/2023-ai-survey-of-2778-six-things?action=share

7. Source: wiki.aiimpacts.org
Title: ai risk surveys
Link:https://wiki.aiimpacts.org/uncategorized/ai_risk_surveys

8. Source: grjenkin.com
Title: 2022 expert survey on progress in ai
Link:https://grjenkin.com/articles/category/machine-learning/6892881/03/20/2023/2022-expert-survey-on-progress-in-ai

9. Source: aiimpacts.org
Title: How bad a future do ML researchers expect? – AI Impacts
Link:https://aiimpacts.org/how-bad-a-future-do-ml-researchers-expect/

10. Source: aiimpacts.org
Title: what do ml researchers think about ai in 2022
Link:https://aiimpacts.org/what-do-ml-researchers-think-about-ai-in-2022/

11. Source: youtube.com
Title: AI Impacts Survey
Link:https://www.youtube.com/watch?v=xwJx_xqZI3Q

12. Source: youtube.com
Link:https://www.youtube.com/watch?v=xW0xjAMD60c

Additional References

13. Source: doi.org
Link:https://doi.org/10.1007/s00146-024-01930-2

Published: April 14, 2024

14. Source: lesswrong.com
Title: FA Q: Expert Survey on Progress in AI methodology — Less Wrong
Link:https://www.lesswrong.com/posts/BEg5dct7nr2Yq3dJY/faq-expert-survey-on-progress-in-ai-methodology

15. Source: forum.effectivealtruism.org
Title: faq expert survey on progress in ai methodology
Link:https://forum.effectivealtruism.org/posts/c977usNg2AmF2m7ud/faq-expert-survey-on-progress-in-ai-methodology

16. Source: youtube.com
Title: Katja Grace on the Largest Survey of AI Researchers
Link:https://www.youtube.com/watch?v=qnqdTAO5OXw

17. Source: link.springer.com
Link:https://link.springer.com/article/10.1007/s00146-024-02134-4

18. Source: researchgate.net
Link:https://www.researchgate.net/publication/401691455_What_are_AI_researchers_worried_about

19. Source: doctrines.ai
Link:https://doctrines.ai/

20. Source: GOV.UK
Title: www.gov.uk Public attitudes to data and AI: Tracker survey (Wave 3)
Link:https://www.gov.uk/government/publications/public-attitudes-to-data-and-ai-tracker-survey-wave-3/public-attitudes-to-data-and-ai-tracker-survey-wave-3

21. Source: youtube.com
Title: Will AI end everything? A guide to guessing | Katja Grace | EAG Bay Area 23
Link:https://www.youtube.com/watch?v=j5Lu01pEDWA

22. Source: blog.biocomm.ai
Title: 2022 expert survey on progress in ai 03 august 2022
Link:https://blog.biocomm.ai/2022/08/03/2022-expert-survey-on-progress-in-ai-03-august-2022/
Published: august 2022