Within Loss of Control
How strong is the case against AI doom?
Sceptics argue that current evidence is weak, future capabilities are uncertain, and theoretical takeover stories may overstate real-world plausibility.
On this page
- Why current systems are not loss of control cases
- The main objections to power seeking arguments
- What evidence could settle part of the dispute
Page outline Jump by section
Introduction
The case for AI doom is often presented as a warning about future systems that become powerful enough, autonomous enough and misaligned enough to escape meaningful human control. But there is no consensus that this scenario is likely. Some of the strongest critics accept that advanced AI could create serious problems while rejecting the claim that loss of control is a plausible route to human extinction or permanent civilisational collapse.
The disagreement is not simply between people who care about safety and people who do not. Many sceptics accept that future AI systems could be dangerous. Their objection is that the strongest loss-of-control arguments rely heavily on theoretical reasoning, uncertain forecasts about future capabilities and assumptions about how highly capable AI systems would behave. They argue that present evidence is weak, that key steps in takeover scenarios remain unproven and that current systems do not look much like the agents described in classic AI doom arguments. [arXiv]arxiv.orgarXivA Review of the Evidence for Existential Risk from AI via Misaligned Power-SeekingOctober 27, 2023…
The result is an unusually deep dispute about evidence. Doom-focused researchers often argue that waiting for direct evidence could be dangerous because truly catastrophic failures may arrive only once systems become extremely capable. Critics reply that this makes the theory difficult to test and risks treating speculative possibilities as established probabilities.
Why current systems are not loss-of-control cases
One of the most common objections is straightforward: today’s AI systems do not resemble the systems that appear in classic takeover stories.
Large language models can generate text, write code, answer questions and perform increasingly complex tasks. Yet they still make obvious errors, hallucinate facts, struggle with long chains of reasoning and depend heavily on human prompting and infrastructure. Critics argue that these limitations matter because the loss-of-control scenario usually assumes systems with far more robust planning ability, situational awareness and strategic competence than anything publicly available today. [AI Security Institute]aisi.gov.ukmapping the limitations of current ai systemsAI Security InstituteMapping the limitations of current AI systems | AISI Work23 Oct 2025 — Most experts we interviewed agreed that failu…
Sceptics also note that current models are not generally trying to seize resources, conceal long-term agendas or acquire political power. They often fail at much simpler objectives. From this perspective, the jump from “imperfect chatbot” to “civilisation-threatening strategist” remains largely hypothetical. [LessWrong]lesswrong.comwhy ais aren t power seeking yetLessWrongWhy AIs aren't power-seeking yet10 Jan 2026 — Current AIs are getting increasingly general, but they're not self-promoting or am…
This does not prove future systems will remain limited. But critics argue that evidence from current systems cuts both ways. Doom advocates often point to examples of specification gaming, deceptive behaviour in controlled tests or reward hacking as early warning signs. Sceptics respond that these examples are usually narrow, artificial and far removed from the kind of robust, open-ended agency required for a genuine loss-of-control event. [arXiv]arxiv.orgarXivA Review of the Evidence for Existential Risk from AI via Misaligned Power-SeekingOctober 27, 2023…
A related criticism is that discussions of AI doom sometimes blur together very different concepts:
- A system giving misleading answers.
- A system exploiting flaws in training.
- A system strategically deceiving its operators.
- A system pursuing global domination.
Critics argue that evidence for the first three does not automatically provide strong evidence for the fourth.
The main objections to power-seeking arguments
Many loss-of-control scenarios rely on the idea of instrumental convergence: the claim that highly capable agents pursuing many different goals will tend to seek power, resources, self-preservation and freedom from interference because those behaviours help achieve a wide range of objectives. This is one of the central theoretical foundations of AI doom arguments. [arXiv]arxiv.orgarXivA Review of the Evidence for Existential Risk from AI via Misaligned Power-SeekingOctober 27, 2023…
Critics challenge this argument from several directions.
Goal-directedness may be less common than doom arguments assume
One objection is that future AI systems may not be strongly goal-directed agents in the relevant sense.
Many current systems behave more like tools than independent actors. They respond to prompts, perform bounded tasks and wait for further instructions. Critics argue that loss-of-control scenarios often assume future AI will naturally evolve into persistent agents with stable goals and long planning horizons, but this transition remains speculative. [Alignment Forum]alignmentforum.orgcounterarguments to the basic ai x risk caseAlignment ForumCounterarguments to the basic AI x-risk case14 Oct 2022 — Katja Grace provides a list of counterarguments to the basic cas…
Even if powerful AI systems emerge, sceptics question whether developers will deploy them as autonomous entities rather than as tightly constrained tools embedded within larger human institutions.
Intelligence does not automatically imply domination
Another challenge targets a common intuition behind AI doom: that a sufficiently intelligent system would inevitably outmanoeuvre humanity.
Critics argue that intelligence alone does not guarantee control over the physical world. Real-world power depends on manufacturing, energy, political influence, logistics, legal authority and access to infrastructure. Human societies are complex networks rather than single points of failure.
From this perspective, some takeover scenarios underestimate the practical difficulties of translating cognitive superiority into global control. A highly intelligent AI might still face severe constraints imposed by governments, competing organisations, physical bottlenecks and other AI systems. [arXiv]arxiv.orgarXivA Review of the Evidence for Existential Risk from AI via Misaligned Power-SeekingOctober 27, 2023…
Instrumental convergence may be overstated
Some critics accept that power-seeking is theoretically possible while arguing that its prevalence has been exaggerated.
A recurring criticism is that many instrumental-convergence arguments rely on abstract models of idealised agents rather than realistic machine learning systems. Real systems may contain conflicting objectives, fragmented goals, uncertainty about their environment or strong constraints imposed during training and deployment.
Researchers have also challenged specific arguments that attempt to show why deceptive or power-seeking systems should be expected by default. Critics argue that some of these arguments rely on assumptions about how training processes select goals that remain poorly established empirically. [EA Forum]forum.effectivealtruism.orga critique of ai takeover scenariosEA ForumA Critique of AI Takeover Scenarios31 Aug 2022 — In this article I will provide a brief critique of the way an 'AI takeover scena…
The disagreement is often less about whether power-seeking is possible and more about whether it should be considered the default expectation.
The problem of forecasting unknown capabilities
A deeper objection concerns the difficulty of predicting future AI systems at all.
Many loss-of-control arguments depend on forecasts about systems that do not yet exist. Critics argue that the further those forecasts extend beyond observed evidence, the less confidence anyone should have in detailed claims about future behaviour.
This criticism appears in several forms.
First, technological progress is often non-linear and surprising. Forecasts regularly fail in both directions. Some technologies arrive sooner than expected; others hit unexpected bottlenecks.
Second, today’s systems may be poor guides to future architectures. If AI changes substantially, both optimistic and pessimistic extrapolations from current models could prove wrong.
Third, critics argue that doom scenarios sometimes stack multiple uncertain assumptions together. A typical loss-of-control story may require:
- The creation of highly capable general systems.
- Strong autonomous agency.
- Persistent goal misalignment.
- Successful deception of human operators.
- Large-scale acquisition of power.
- Failure of competing institutions to intervene.
Each step may appear plausible in isolation. Critics argue that multiplying several uncertain propositions can produce much lower overall probabilities than some public discussions imply. [arXiv]arxiv.orgarXivA Review of the Evidence for Existential Risk from AI via Misaligned Power-SeekingOctober 27, 2023…
This is one reason debates over p(doom) often remain unresolved. Different estimates frequently reflect different judgements about long chains of uncertainty rather than disagreements about a single piece of evidence.
Are takeover stories too dependent on science-fiction intuitions?
Another criticism is that some popular AI doom narratives inherit assumptions from science fiction.
Stories about rogue superintelligences often involve coherent, singular entities pursuing grand objectives over long periods. Critics argue that real AI development may look much messier: multiple competing systems, extensive human supervision, fragmented incentives and continual software updates.
Some sceptics worry that vivid takeover scenarios create a false sense of concreteness. A detailed story can feel persuasive even when many of its key steps remain speculative.
This does not mean such scenarios are impossible. Rather, critics argue that narrative coherence should not be confused with empirical evidence. A scenario can be internally consistent while still having a low probability of occurring. [EA Forum]forum.effectivealtruism.orga critique of ai takeover scenariosEA ForumA Critique of AI Takeover Scenarios31 Aug 2022 — In this article I will provide a brief critique of the way an 'AI takeover scena…
A related concern is anthropomorphism. People often interpret AI behaviour through analogies with human motives such as ambition, self-preservation or desire for control. Critics argue that future AI systems could behave in ways that are alien, fragmented or difficult to compare with human psychology. This makes simple analogies potentially misleading. [The Guardian]theguardian.comThe Guardian AI consciousness is a red herring in the safety debateProfessor Virginia Dignum argues that interpreting AI behaviors—like resisting shutdowns—as signs of consciousness is misleading and anth…
The institutional objection: humans may remain in the loop
Some sceptics focus less on technical arguments and more on how AI is actually deployed.
Most important systems in modern society are embedded within organisations rather than operating independently. Governments, companies, regulators, military institutions and infrastructure operators already impose layers of oversight on high-risk technologies.
Critics argue that loss-of-control arguments sometimes assume a surprisingly passive human response. If warning signs appeared, developers could limit deployment, reduce autonomy, increase monitoring or impose legal restrictions. The world is not forced to hand unlimited authority to a single AI system.
This argument becomes especially important when discussing extreme outcomes such as human extinction. Even if advanced AI caused severe disruptions, sceptics argue that civilisation might adapt, contain problems or recover rather than disappearing entirely. [arXiv]arxiv.orgarXivA Review of the Evidence for Existential Risk from AI via Misaligned Power-SeekingOctober 27, 2023…
Doom-focused researchers often respond that competitive pressures could undermine caution. But sceptics note that this is itself an empirical question about institutions, incentives and governance rather than a direct consequence of machine intelligence alone.
What evidence could settle part of the dispute?
One reason the debate persists is that both sides can point to genuine uncertainties.
There are at least three areas where stronger evidence could narrow the disagreement.
Evidence for persistent strategic deception
Many loss-of-control scenarios rely on the possibility that AI systems will conceal their true objectives until they become difficult to stop.
If researchers repeatedly observed sophisticated deception, long-term strategic planning and deliberate circumvention of oversight in increasingly capable systems, sceptics would have stronger reasons to update toward concern.
Conversely, if capabilities continued improving while such behaviours remained absent, that would weaken some takeover arguments.
Evidence about autonomous power-seeking
A central unresolved question is whether advanced systems naturally develop incentives to gain resources, preserve themselves or resist intervention.
Researchers already study reward hacking, specification gaming and behaviour under evaluation. More realistic tests of long-horizon autonomy could provide clearer evidence about whether power-seeking emerges in practice rather than only in theory. [arXiv]arxiv.orgarXivA Review of the Evidence for Existential Risk from AI via Misaligned Power-SeekingOctober 27, 2023…
Evidence about capability limits
The strongest sceptical arguments often point to persistent weaknesses in current systems.
If AI repeatedly encounters hard limits in reasoning, planning, reliability or autonomous action, then some assumptions behind rapid loss-of-control scenarios become less compelling.
On the other hand, if systems continue overcoming apparent limitations while becoming increasingly agentic and capable of independent operation, critics would lose some of their strongest empirical support. [AI Security Institute]aisi.gov.ukmapping the limitations of current ai systemsAI Security InstituteMapping the limitations of current AI systems | AISI Work23 Oct 2025 — Most experts we interviewed agreed that failu…
Why the disagreement remains serious
The strongest objections to AI loss-of-control fears do not usually claim that advanced AI is obviously safe. Instead, they challenge the confidence with which some people move from theoretical possibility to existential prediction.
Critics argue that current evidence for AI takeover remains limited, that power-seeking arguments rely on contested assumptions, that intelligence does not automatically produce domination and that forecasts about future systems remain highly uncertain. They also emphasise that present AI systems show many weaknesses inconsistent with popular images of an emerging superintelligence. [arXiv]arxiv.orgarXivA Review of the Evidence for Existential Risk from AI via Misaligned Power-SeekingOctober 27, 2023… [Alignment Forum]alignmentforum.orgcounterarguments to the basic ai x risk caseAlignment ForumCounterarguments to the basic AI x-risk case14 Oct 2022 — Katja Grace provides a list of counterarguments to the basic cas…
At the same time, even many sceptics acknowledge that the question cannot be settled purely by observing today’s models. The core dispute concerns future systems that have not yet been built. As a result, the debate remains unusually dependent on theory, forecasting and judgement under uncertainty.
That is why arguments over AI doom often persist despite extensive technical discussion. The disagreement is not only about what AI can do today. It is about how much weight to place on warnings about what future systems might become before there is direct evidence either way. [arXiv]arxiv.orgarXivA Review of the Evidence for Existential Risk from AI via Misaligned Power-SeekingOctober 27, 2023…
Amazon book picks
Further Reading
Books and field guides related to How strong is the case against AI doom?. Use these as the next step if you want deeper reading beyond the article.
The Myth of Artificial Intelligence
Provides a substantial counterargument to strong AI-doom claims.
The Alignment Problem
Presents evidence and uncertainties surrounding alignment risks.
eBay marketplace picks
Marketplace Samples
Example marketplace items related to this page. Use the search link to explore similar finds on eBay.
Endnotes
-
Source: arxiv.org
Link: https://arxiv.org/abs/2310.18244Source snippet
arXivA Review of the Evidence for Existential Risk from AI via Misaligned Power-SeekingOctober 27, 2023...
Published: October 27, 2023
-
Source: arxiv.org
Title: arXiv Artificial Intelligence: Arguments for Catastrophic Risk
Link: https://arxiv.org/abs/2401.15487 -
Source: lesswrong.com
Title: why ais aren t power seeking yet
Link: https://www.lesswrong.com/posts/7ZH4oppNnTGtq4xXu/why-ais-aren-t-power-seeking-yetSource snippet
LessWrongWhy AIs aren't power-seeking yet10 Jan 2026 — Current AIs are getting increasingly general, but they're not self-promoting or am...
-
Source: arxiv.org
Title: arXiv Is Power-Seeking AI an Existential Risk?
Link: https://arxiv.org/abs/2206.13353Source snippet
arXivIs Power-Seeking AI an Existential Risk?June 16, 2022...
Published: June 16, 2022
-
Source: arxiv.org
Link: https://arxiv.org/pdf/2206.13353Source snippet
I proceed in two stages.Read mor...
-
Source: arxiv.org
Link: https://arxiv.org/html/2507.23330v1Source snippet
AI Must not be Fully Autonomous31 Jul 2025 — It has been argued that the autonomous self-improvement of ASI could become too advanced for...
-
Source: lesswrong.com
Title: draft report on existential risk from power seeking ai
Link: https://www.lesswrong.com/posts/HduCjmXTBD4xYTegv/draft-report-on-existential-risk-from-power-seeking-aiSource snippet
Draft report on existential risk from power-seeking AI28 Apr 2021 — I've written a draft report evaluating a version of the overall case...
-
Source: aisi.gov.uk
Title: mapping the limitations of [current ai]({{ ‘current-benchmarks/’ | relative_url }}) systems
Link: https://www.aisi.gov.uk/blog/mapping-the-limitations-of-current-ai-systemsSource snippet
AI Security InstituteMapping the limitations of current AI systems | AISI Work23 Oct 2025 — Most experts we interviewed agreed that failu...
-
Source: alignmentforum.org
Title: counterarguments to the basic ai x risk case
Link: https://www.alignmentforum.org/posts/LDRQ5Zfqwi8GjzPYG/counterarguments-to-the-basic-ai-x-risk-caseSource snippet
Alignment ForumCounterarguments to the basic AI x-risk case14 Oct 2022 — Katja Grace provides a list of counterarguments to the basic cas...
-
Source: forum.effectivealtruism.org
Title: a critique of ai takeover scenarios
Link: https://forum.effectivealtruism.org/posts/j7X8nQ7YvvA7Pi4BX/a-critique-of-ai-takeover-scenariosSource snippet
EA ForumA Critique of AI Takeover Scenarios31 Aug 2022 — In this article I will provide a brief critique of the way an 'AI takeover scena...
-
Source: forum.effectivealtruism.org
Title: EA Forum Counting arguments provide no evidence for AI doom
Link: https://forum.effectivealtruism.org/posts/qtDLZsM5rypkEHxYm/counting-arguments-provide-no-evidence-for-ai-doomSource snippet
EA ForumCounting arguments provide no evidence for AI doomFebruary 27, 2024 — 27 Feb 2024 — In this essay, we debunk the counting argumen...
Published: February 27, 2024
-
Source: theguardian.com
Title: The Guardian AI consciousness is a red herring in the safety debate
Link: https://www.theguardian.com/technology/2026/jan/06/ai-consciousness-is-a-red-herring-in-the-safety-debateSource snippet
Professor Virginia Dignum argues that interpreting AI behaviors—like resisting shutdowns—as signs of consciousness is misleading and anth...
-
Source: alignmentforum.org
Title: current ais seem pretty misaligned to me
Link: https://www.alignmentforum.org/posts/WewsByywWNhX9rtwi/current-ais-seem-pretty-misaligned-to-meSource snippet
15 Apr 2026 — Current misalignment is also evidence about how AI companies will operate—how sloppy they'll be (due to being in a huge rus...
Additional References
-
Source: researchgate.net
Link: https://www.researchgate.net/publication/397942549_Examining_popular_arguments_against_AI_existential_risk_a_philosophical_analysisSource snippet
Examining popular arguments against AI existential risk28 Nov 2025 — Many have argued, based on the Instrumental Convergence Thesis, that...
-
Source: 80000hours.org
Link: https://80000hours.org/problem-profiles/risks-from-power-seeking-ai/Source snippet
Risks from power-seeking AI systemsThis article looks at why AI power-seeking poses [severe risks]({{ 'risk-thresholds/' | relative_url }}), what current research reveals about the...
-
Source: wsj.com
Link: https://www.wsj.com/tech/ai/artificial-superintelligence-overestimation-3f954065Source snippet
A new study from Apple, "The Illusion of Thinking," along with corroborating research from Salesforce and academia, shows that current la...
-
Source: pauseai.info
Link: https://pauseai.info/ai-x-risk-skepticismSource snippet
They see no path for AI to gain unlimited intelligence and power to take over. It only...Read more...
-
Source: forum.effectivealtruism.org
Title: draft report on existential risk from power seeking ai
Link: https://forum.effectivealtruism.org/posts/78NoGoRitPzeT8nga/draft-report-on-existential-risk-from-power-seeking-aiSource snippet
report on existential risk from power-seeking AI28 Apr 2021 — I've written a draft report evaluating a version of the overall case for ex...
-
Source: reddit.com
Link: https://www.reddit.com/r/ControlProblem/comments/1fajsvw/my_critique_of_roman_yampolskiys_ai_unexplainable/Source snippet
nately we show that the AI control problem is not solvable...Read more...
-
Source: musingsandroughdrafts.com
Link: https://musingsandroughdrafts.com/2026/01/11/why-ais-arent-power-seeking-yet/Source snippet
Why AIs aren't power-seeking yet - musings and rough drafts11 Jan 2026 — Current AIs are getting increasingly general, but they're not se...
-
Source: compass.onlinelibrary.wiley.com
Link: https://compass.onlinelibrary.wiley.com/doi/10.1111/phc3.12964Source snippet
Intelligence: Arguments for Catastrophic Risk - Bales10 Feb 2024 — We review two influential arguments purporting to show how AI could po...
-
Source: blog.redwoodresearch.org
Title: current ais seem pretty misaligned
Link: https://blog.redwoodresearch.org/p/current-ais-seem-pretty-misalignedSource snippet
AIs seem pretty misaligned to me15 Apr 2026 — Current misalignment is also evidence about how AI companies will operate—how sloppy they'l...
-
Source: mechanize.work
Title: unfalsifiable stories of doom
Link: https://www.mechanize.work/blog/unfalsifiable-stories-of-doom/Source snippet
Mechanize, Inc.Unfalsifiable stories of doom25 Nov 2025 — A response to Yudkowsky and Soares's 'If Anyone Builds It, Everyone Dies'. Thei...
Topic Tree