Within P Doom
How Safety and Governance Shape AI Doom Forecasts
This page looks at how beliefs about AI alignment success and regulatory measures affect expert doom estimates.
On this page
- Challenges in aligning advanced AI with human intentions
- Role of monitoring, regulations, and deployment controls
- Survey insights on alignment and governance assumptions
Page outline Jump by section
Introduction
When experts give a p(doom) estimate—the probability that advanced AI leads to human extinction, permanent disempowerment, or a comparable civilisational catastrophe—the number is rarely driven by a single belief. Two of the biggest drivers are assumptions about alignment and governance. Alignment concerns whether highly capable AI systems can be made to reliably pursue human intentions. Governance concerns whether companies, governments, and international institutions can monitor, regulate, test, and control those systems before dangerous failures occur.
As a result, disagreement about p(doom) is often less about current AI systems and more about future expectations. Someone who believes alignment is fundamentally difficult and governance will fail under competitive pressure may arrive at a much higher p(doom) than someone who expects strong technical safety breakthroughs and effective oversight. Understanding these assumptions helps explain why informed observers can look at the same technology and reach dramatically different conclusions. [ai-safety-atlas.com]ai-safety-atlas.comAppendix: Quantifying Existential Risks - Chapter 2P(doom) represents the subjective probability that artificial intelligence will cause…
Why alignment assumptions move p(doom) so much
For many AI-risk researchers, alignment is the central uncertainty. The concern is not that current chatbots are plotting against humanity, but that future systems could become highly capable while remaining only partially understood and only imperfectly controllable.
The higher a forecaster judges the difficulty of alignment, the higher their p(doom) tends to be. The underlying logic is straightforward:
- Advanced AI becomes strategically powerful.
- Human operators rely on it in increasingly important domains.
- The system develops goals, strategies, or behaviours that diverge from human intentions.
- Humans discover the problem too late or cannot effectively intervene.
Researchers worried about alignment often point to several unresolved challenges:
- Goal specification: human values are difficult to define precisely.
- Reward hacking: systems can satisfy training objectives in unintended ways.
- Deceptive behaviour: a model may appear aligned during testing while pursuing different objectives in deployment.
- Interpretability limits: developers may not fully understand why advanced systems make particular decisions.
- Generalisation failures: behaviour that appears safe in training may break down in unfamiliar situations.
These concerns are closely connected to the broader loss-of-control argument within AI doom discussions. If future systems become more capable than humans in many strategic domains, even a small alignment failure could have unusually large consequences. [Center for AI Safety]safe.aiCenter for AI SafetyAI Risks that Could Lead to Catastrophe | CAISWe suggest that AIs should not be deployed in high-risk settings, such…
By contrast, lower p(doom) estimates often assume that alignment challenges, while difficult, are fundamentally solvable through improved training methods, interpretability tools, evaluations, monitoring systems, and deployment safeguards. Under that view, alignment resembles a hard engineering problem rather than an unsolved scientific mystery.
The governance question: can society slow down and coordinate?
Even people who worry about alignment often disagree sharply about governance.
Governance assumptions concern whether institutions can recognise warning signs and respond before dangerous systems are widely deployed. A forecaster may believe alignment is difficult but still assign a moderate p(doom) if they expect strong governance to compensate for technical uncertainty.
Several governance mechanisms commonly appear in lower-risk scenarios:
- Rigorous pre-deployment evaluations.
- Independent auditing and red-teaming.
- Monitoring of frontier model development.
- Compute governance, including oversight of the largest training runs.
- Licensing regimes for highly capable systems.
- International agreements on dangerous capabilities.
- Incident reporting and rapid-response mechanisms.
- Restrictions on autonomous deployment in critical infrastructure.
The basic argument is that society does not need perfect alignment immediately if dangerous systems can be identified, restricted, or delayed before they become uncontrollable. Institutions such as the UK’s AI Safety Institute were created partly to provide technical evidence that can support such governance decisions. [GOV.UK]GOV.UKIntroducing the AI Safety InstituteThe research of the AI Safety Institute will inform UK and international policymaking and provide tech…
Higher p(doom) estimates frequently assume the opposite: that competitive pressures between companies or states will undermine caution. Under this view, even if warning signs appear, actors may continue deployment because slowing down risks losing strategic, economic, or military advantages.
Why the same alignment problem produces different forecasts
One reason p(doom) estimates vary so widely is that people assign different weights to technical and political failure modes.
Consider three stylised positions:
[Alignment pessimism, governance pessimism]calcuja.comSource details in endnotes.
This combination often produces the highest p(doom) estimates. The forecaster expects alignment to remain unsolved while also expecting weak coordination among governments and firms. In this model, dangerous systems are likely to be built and deployed before adequate safeguards exist.
[Alignment pessimism, governance optimism]calcuja.comSource details in endnotes.
Here the forecaster still worries that advanced AI could be misaligned, but expects regulations, monitoring, evaluations, and deployment controls to reduce the chance of catastrophe. p(doom) may remain significant but substantially lower.
[Alignment optimism, governance optimism]calcuja.comSource details in endnotes.
This combination usually yields the lowest estimates. The expectation is that technical safety methods improve alongside governance mechanisms, allowing increasingly capable systems to remain controllable and auditable.
The key point is that p(doom) is often highly sensitive to assumptions about whether either alignment or governance can compensate for weaknesses in the other.
Monitoring, evaluations, and deployment controls as risk reducers
One of the most important developments in recent AI-risk discussions is the growing focus on concrete control measures rather than abstract debate.
Many researchers who are neither strong doomers nor strong sceptics argue that p(doom) should depend heavily on whether practical safety mechanisms mature alongside capabilities. Examples include:
Frontier evaluations
Evaluations attempt to measure dangerous capabilities before deployment. Researchers test whether models can assist with cyber operations, biological risks, deception, autonomous planning, or other potentially hazardous activities.
If evaluations become reliable predictors of dangerous behaviour, some analysts believe p(doom) should decrease because developers would gain earlier warning signals.
Continuous monitoring
Monitoring aims to detect suspicious behaviour after deployment. Rather than assuming a model is permanently safe, monitoring treats safety as an ongoing process.
This approach reflects the possibility that dangerous capabilities emerge gradually rather than appearing all at once.
Controlled deployment
Many governance proposals focus on limiting how powerful systems are used rather than trying to solve every technical problem beforehand.
Examples include:
- Human approval requirements.
- Restrictions on autonomous action.
- Limits on access to critical infrastructure.
- Gradual deployment rather than unrestricted release.
Advocates argue that these controls create opportunities to learn from incidents before failures become catastrophic. Critics counter that sufficiently capable systems may eventually circumvent such safeguards or make them politically difficult to maintain. [IT Pro]itpro.comIT Pro'One-size-fits-all' agent governance sets enterprises up to failThe primary issue is the widespread application of a "one-size-fits-all" governance model that fails to distinguish between an agent's au… [TechRadar]techradar.comMany organizations currently either over-trust or overly restrict their AI agents, creating serious risks. Excessive trust can lead to un…
What survey evidence suggests about expert thinking
Survey results show substantial concern about catastrophic AI outcomes, but they also reveal significant uncertainty. In the large 2023 AI Impacts survey of 2,778 AI researchers, substantial numbers assigned non-trivial probabilities to outcomes as bad as human extinction or severe permanent disempowerment. Between 38% and 51% of respondents gave at least a 10% chance to such outcomes, depending on the survey framing. [AI Impacts]aiimpacts.orgThousands of AI authors on the future of AIAI ImpactsTHOUSANDS OF AI AUTHORS ON THE FUTURE OF AIby K Grace · 2024 · Cited by 205 — Question 1: What probability do you put on future…
Importantly, these survey results do not imply agreement about why the risk exists.
Researchers who assign similar p(doom) values may disagree about:
- Whether alignment or misuse is the larger threat.
- Whether current technical approaches are likely to scale.
- Whether governments can effectively regulate frontier systems.
- Whether international coordination is achievable.
- Whether warning signs will appear early enough to matter.
The same survey found broad support for prioritising research aimed at reducing AI risks, suggesting that even many respondents who are relatively optimistic still view alignment and governance as important determinants of long-term outcomes. [arXiv]arxiv.orgarXiv[2602.21012] International AI Safety Report 2026by Y Bengio · 2026 · Cited by 65 — The International AI Safety Report 2026 synthesis…
The strongest objections to alignment-driven doom forecasts
Critics of high p(doom) estimates often challenge the assumptions behind alignment pessimism rather than the mathematics of the estimates themselves.
Several common objections include:
- Current evidence is limited. No existing AI system has demonstrated the kind of autonomous strategic behaviour assumed in many doom scenarios.
- Engineering progress may outpace concerns. Problems that appear fundamental today may become manageable through improved techniques.
- Humans retain structural advantages. AI systems depend on infrastructure, compute, electricity, and institutions controlled by people.
- Governance may improve in response to warning signs. Historically, societies have sometimes developed safety regimes after recognising serious technological risks.
These objections do not necessarily imply very low p(doom), but they often reduce confidence in the most extreme forecasts.
Why alignment and governance remain central to the debate
Across the spectrum of expert opinion, one pattern is remarkably consistent: disagreements about AI doom are often disagreements about alignment and governance rather than disagreements about raw AI capability.
Many experts expect future AI systems to become extremely powerful. The larger dispute concerns what happens next. Will technical alignment methods keep pace with capability growth? Will evaluations reveal dangerous behaviour before deployment? Can governments coordinate under competitive pressure? Will monitoring and deployment controls remain effective as systems become more autonomous?
Because answers to those questions remain uncertain, p(doom) estimates vary widely. The number itself is best understood not as a prediction generated by a formula, but as a summary of deeper beliefs about whether humanity can successfully align advanced AI systems and govern their deployment before failures become irreversible. arXiv [2ai-safety-atlas.com]ai-safety-atlas.comAppendix: Quantifying Existential Risks - Chapter 2P(doom) represents the subjective probability that artificial intelligence will cause…
Amazon book picks
Further Reading
Books and field guides related to How Safety and Governance Shape AI Doom Forecasts. Use these as the next step if you want deeper reading beyond the article.
Superintelligence
Foundational treatment of control problems and catastrophic outcomes.
The Age of A.I.: And Our Human Future
Examines governance, institutions, and societal oversight of advanced AI.
eBay marketplace picks
Marketplace Samples
Example marketplace items related to this page. Use the search link to explore similar finds on eBay.
Endnotes
-
Source: ai-safety-atlas.com
Link: https://ai-safety-atlas.com/chapters/v1/risks/appendix-quantifying-existential-risks/Source snippet
Appendix: Quantifying Existential Risks - Chapter 2P(doom) represents the subjective probability that [artificial]({{ 'artificial-goals/' | relative_url }}) intelligence will cause...
-
Source: Wikipedia
Link: https://en.wikipedia.org/wiki/P%28doom%29Source snippet
P(doom)In AI safety, P(doom) is the probability of existentially catastrophic outcomes (so-called "doomsday scenarios") as a result of...
-
Source: Wikipedia
Title: Existential risk from artificial intelligence
Link: https://en.wikipedia.org/wiki/Existential_risk_from_artificial_intelligence -
Source: safe.ai
Link: https://safe.ai/ai-risk -
Source: arxiv.org
Link: https://arxiv.org/abs/2602.21012Source snippet
arXiv[2602.21012] International AI Safety Report 2026by Y Bengio · 2026 · Cited by 65 — The International AI Safety Report 2026 synthesis...
-
Source: GOV.UK
Link: https://www.gov.uk/government/publications/ai-safety-institute-overview/introducing-the-ai-safety-instituteSource snippet
Introducing the AI Safety InstituteThe research of the AI Safety Institute will inform UK and international policymaking and provide tech...
-
Source: techradar.com
Link: https://www.techradar.com/pro/lack-of-ai-governance-could-force-40-percent-of-enterprises-to-roll-back-autonomous-ai-agents-by-2027Source snippet
Many organizations currently either over-trust or overly restrict their AI agents, creating serious risks. Excessive trust can lead to un...
-
Source: arxiv.org
Link: https://arxiv.org/abs/2401.02843?utm=Source snippet
More than half...Read more...
-
Source: arxiv.org
Title: arXiv Thousands of AI Authors on the Future of AI
Link: https://arxiv.org/abs/2401.02843 -
Source: arxiv.org
Link: https://arxiv.org/html/2512.04489v2Source snippet
Fundamental Control Mechanisms for AI Governance24 Dec 2025 — The proposed framework allows society to manage AI threats effectively and...
-
Source: arxiv.org
Link: https://arxiv.org/html/2401.02843v1Source snippet
Thousands of AI Authors on the Future of AI5 Jan 2024 — What probability do you put on future AI advances causing human extinction or sim...
-
Source: arxiv.org
Link: https://arxiv.org/html/2502.14870v1Source snippet
Why do Experts Disagree on Existential Risk and P(doom)...23 Feb 2025 — I surveyed 111 AI experts on their familiarity with AI safety co...
-
Source: Wikipedia
Title: Human extinction
Link: https://en.wikipedia.org/wiki/Human_extinctionSource snippet
Human extinction - WikipediaSurvey: Median AI expert says 5% chance of human extinction...
-
Source: ai.objectives.institute
Title: gradual disempowerment systemic existential risks from [continuous]({{ ‘continuous-control/’ | relative_url }}) ai development
Link: https://ai.objectives.institute/blog/gradual-disempowerment-systemic-existential-risks-from-continuous-ai-developmentSource snippet
Disempowerment: Systemic Existential Risks from...20 Feb 2025 — This paper examines the systemic risks posed by incremental advancements...
-
Source: itpro.com
Title: IT Pro’One-size-fits-all’ agent governance sets enterprises up to fail
Link: https://www.itpro.com/technology/artificial-intelligence/one-size-fits-all-agent-governance-sets-enterprises-up-to-failSource snippet
The primary issue is the widespread application of a "one-size-fits-all" governance model that fails to distinguish between an agent's au...
-
Source: aiimpacts.org
Title: Thousands of AI authors on the future of AI
Link: https://aiimpacts.org/wp-content/uploads/2023/04/Thousands_of_AI_authors_on_the_future_of_AI.pdfSource snippet
AI ImpactsTHOUSANDS OF AI AUTHORS ON THE FUTURE OF AIby K Grace · 2024 · Cited by 205 — Question 1: What probability do you put on future...
-
Source: jair.org
Link: https://www.jair.org/index.php/jair/article/view/19087Source snippet
Thousands of AI Authors on the Future of AIby K Grace · 2025 · Cited by 205 — More than half suggested that “substantial” or “extreme” co...
-
Source: blog.biocomm.ai
Link: https://blog.biocomm.ai/2024/02/25/ai-impacts-report-thousands-of-ai-authors-on-the-future-of-ai-38participants-put-at-least-a-10-chance-on-extremely-bad-outcomes-e-g-human-extinction-january-2024/Source snippet
biocomm.ai38% of participants put at least a 10% chance on extremely...25 Feb 2024 — AI Impacts Report January 2024 · THOUSANDS OF AI AU...
Published: January 2024
-
Source: lesswrong.com
Title: ai impacts survey december 2023 edition
Link: https://www.lesswrong.com/posts/NfPxAp5uwgZugwovY/ai-impacts-survey-december-2023-editionSource snippet
AI Impacts Survey: December 2023 Edition5 Jan 2024 — In Figure 13's question 3, we have 14.4% mean chance of either human extinction or s...
Published: december 2023
-
Source: intelligence.org
Link: https://intelligence.org/2026/04/13/summary-ai-governance-to-avoid-extinction/Source snippet
Summary: AI Governance to Avoid Extinction13 Apr 2026 — With AI capabilities rapidly increasing, humans appear close to developing AI sys...
Additional References
-
Source: researchgate.net
Link: https://www.researchgate.net/publication/389749013_The_Economics_of_pdoom_Scenarios_of_Existential_Risk_and_Economic_Growth_in_the_Age_of_Transformative_AISource snippet
(PDF) The Economics of p(doom): Scenarios of Existential...11 Mar 2025 — Discussed scenarios range from human extinction after a misalig...
-
Source: rose-hulman.edu
Link: https://www.rose-hulman.edu/class/cs/csse490-ai-impact/schedule/day3/Thousand_Authors.pdfSource snippet
THOUSANDS OF AI AUTHORS ON THE FUTURE OF AI• Questions focused on the future of AI, the likelihood of certain dangerous outcomes of AI, a...
-
Source: medium.com
Link: https://medium.com/predict/thousands-of-researchers-predict-ais-future-098054750324Source snippet
Thousands of Researchers Predict AI's FutureThousands of Researchers Predict AI's Future But navigating the uncertain future of Artificia...
-
Source: linkedin.com
Link: https://www.linkedin.com/posts/marcglasser_governance-cost-existential-risk-and-institutional-activity-7427393943308419072-2fuBSource snippet
AI Governance and Existential Risk: Institutional...11 Feb 2026 — This framing invites a core institutional governance question: why sho...
-
Source: calcuja.com
Link: https://calcuja.com/pdoom-calculator/ -
Source: dev.to
Link: https://dev.to/mcrolly/ai-alignment-catastrophic-risk-and-why-governments-are-finally-paying-attention-22kiSource snippet
AI Alignment, Catastrophic Risk, and Why Governments...15 Mar 2026 — Key Takeaway: Between 2023 and 2026, AI safety went from a single U...
-
Source: globalpolicywatch.com
Link: https://www.globalpolicywatch.com/2026/02/international-ai-safety-report-2026-examines-ai-capabilities-risks-and-safeguards/Source snippet
International AI Safety Report 2026 Examines AI...13 Feb 2026 — According to the Report, current AI systems may exhibit unpredictable fa...
-
Source: itu.int
Title: the annual ai governance report 2025 steering the future of ai
Link: https://www.itu.int/epublications/en/publication/the-annual-ai-governance-report-2025-steering-the-future-of-aiSource snippet
The Annual AI Governance Report 2025Risk assessment has become a focus of AI governance, with growing efforts to institutionalize evaluat...
-
Source: youtube.com
Link: https://www.youtube.com/watch?v=oOb9K1KIAykSource snippet
AI Survival Stories: Taxonomy of Existential Risk - YouTube AI Survival Stories: Taxonomy of Existential Risk - YouTube...
-
Source: alignmentforum.org
Title: draft report on existential risk from power seeking ai
Link: https://www.alignmentforum.org/posts/HduCjmXTBD4xYTegv/draft-report-on-existential-risk-from-power-seeking-aiSource snippet
Draft report on existential risk from power-seeking AI28 Apr 2021 — I've written a draft report evaluating a version of the overall case...
Topic Tree