Within Disempowerment
Could Competition Make Human Control Too Expensive?
Firms and governments may automate more authority because rivals that keep slower human processes risk falling behind.
On this page
- Why organisations feel pressured to delegate
- How competitive substitution could spread across institutions
- What could preserve meaningful human roles
Page outline Jump by section
Introduction
One proposed route to long-term human disempowerment does not depend on a dramatic AI takeover. Instead, it depends on competition. The concern is that businesses, governments and other institutions may gradually remove human judgement from important decisions because organisations that automate more aggressively become faster, cheaper or more effective than rivals. If keeping people “in the loop” becomes a competitive disadvantage, meaningful human oversight could steadily disappear even when nobody intended that outcome. This is sometimes described as a race to automate judgement rather than merely automate routine work. It is one of the central mechanisms discussed in gradual-disempowerment scenarios, where ordinary competitive pressures rather than malicious intent slowly reduce humanity’s practical influence over critical systems.[arXiv]arxiv.orgGradual Disempowerment: Systemic Existential Risks from Incremental AI DevelopmentJanuary 28, 2025…
Within AI doom and existential-risk debates, this mechanism matters because it offers an alternative to sudden “loss of control” stories. Instead of one catastrophic moment, control could erode through thousands of individually rational decisions. Whether that process could ultimately contribute to existential risk remains highly uncertain and strongly disputed, but it has become an increasingly important topic in AI governance and safety research.[arXiv]arxiv.orgGradual Disempowerment: Systemic Existential Risks from Incremental AI DevelopmentJanuary 28, 2025…
Why organisations feel pressured to delegate
Competitive markets often reward speed, lower costs and better predictions. If AI systems outperform people on enough tasks, organisations may conclude that retaining human review is simply too expensive.
This pressure already exists in limited forms. Automated fraud detection, logistics, advertising, financial trading, customer support and software development all illustrate situations where slower human decision-making can reduce competitiveness. The concern raised by AI-risk researchers is not that these individual applications are inherently catastrophic, but that the same incentives may eventually spread into progressively more important strategic, legal and governmental decisions.[arXiv]arxiv.orgGradual Disempowerment: Systemic Existential Risks from Incremental AI DevelopmentJanuary 28, 2025…
Several forces reinforce one another:
- Cost pressure. Human experts require salaries, training and time, while software can often be replicated cheaply.
- Speed pressure. AI systems may evaluate thousands of options in the time a committee reaches one decision.
- Competitive imitation. Once one successful organisation automates, rivals may feel compelled to follow regardless of their own preferences.
- Performance metrics. Organisations are often judged by measurable outputs such as profit, response time or efficiency rather than by how much human judgement they preserve.
None of these incentives requires anyone to believe that AI should replace people entirely. The mechanism depends only on local optimisation: each organisation making decisions that appear individually sensible.
How competitive substitution could spread across institutions
Supporters of the gradual-disempowerment hypothesis argue that competitive substitution can produce systemic effects that no individual actor intended.
Imagine two firms making similar decisions. One requires senior staff to approve every important recommendation. The other allows AI systems to make routine decisions automatically, with humans intervening only rarely.
If the second organisation consistently operates faster and more cheaply while producing acceptable results, competitors may copy its model. Over time, what began as an optional efficiency measure can become an industry expectation.
The same logic could apply beyond private companies:
- government agencies competing for efficiency targets;
- militaries attempting to shorten decision cycles;
- financial institutions responding to markets measured in milliseconds;
- healthcare systems facing staff shortages;
- legal systems under pressure to process growing caseloads.
No single organisation needs to believe that human judgement lacks value. Instead, each may conclude that extensive human review is no longer economically sustainable.
Researchers studying gradual disempowerment argue that these individually rational choices could accumulate across many sectors until human discretion becomes the exception rather than the norm.[arXiv]arxiv.orgGradual Disempowerment: Systemic Existential Risks from Incremental AI DevelopmentJanuary 28, 2025…
Why “a human in the loop” may not solve the problem
Many AI governance frameworks require human oversight. However, researchers increasingly distinguish between formal oversight and meaningful oversight.
A human reviewer who is expected to approve hundreds of AI-generated recommendations every hour may possess legal authority without having sufficient time, expertise or organisational support to exercise independent judgement. Human oversight can therefore become procedural rather than substantive.[springer.com]link.springer.comInstitutionalised distrust and human oversight of artificial intelligence: towards a democratic design of AI governance under the…
Researchers identify several recurring problems.
Automation bias. People often become more willing to accept recommendations from automated systems, especially when those systems perform well most of the time. This can reduce independent checking precisely because the technology appears trustworthy. The EU AI Act explicitly recognises automation bias as a risk that deployers should address.[arXiv]arxiv.orgAutomation Bias in the AI Act: On the Legal Implications of Attempting to De-Bias Human Oversight of AIFebruary 14, 2025…
Deskilling. If professionals increasingly rely on AI recommendations, they may gradually lose the practical experience needed to recognise subtle mistakes or intervene confidently. Oversight becomes weaker because the human capability it depends upon is slowly eroded.[World Economic Forum]weforum.orgWorld Economic ForumThe oversight paradox: Human control over AI may be eroding | World Economic ForumJuly 2, 2026…
Compressed decision times. Competitive environments frequently reward faster decisions. Even if humans technically retain approval authority, there may be insufficient time to investigate unusual cases before action is required.[Taylor & Francis Online]tandfonline.comTaylor & Francis OnlineHuman Oversight in AI-Driven Intelligence: Rhetoric, Reality, and the Risks of Automation: International Journal o…
These issues matter because human oversight is often proposed as the main safeguard against increasingly capable AI systems. If competitive pressures gradually weaken oversight itself, that safeguard may become less reliable.
Could this become an existential-risk pathway?
This mechanism differs from classic scenarios involving a single rogue superintelligence.
Instead, existential-risk advocates ask whether civilisation could become structurally dependent on AI systems whose objectives, optimisation strategies or interactions are no longer meaningfully directed by humans.
The concern is not simply widespread automation. It is the possibility that economic, political and administrative systems become organised around machine optimisation because doing so consistently outcompetes alternatives. If reversing that dependence later proves economically or politically impossible, humanity’s ability to redirect civilisation could gradually diminish.[arXiv]arxiv.orgGradual Disempowerment: Systemic Existential Risks from Incremental AI DevelopmentJanuary 28, 2025…
This argument contains several important uncertainties.
First, there is little evidence today that society is undergoing civilisation-scale disempowerment. Current AI systems remain heavily dependent on human infrastructure, maintenance and supervision.
Second, historical technological revolutions often generated new forms of work and governance rather than permanently eliminating human authority.
Third, many decisions involve values, legitimacy, negotiation and accountability rather than simple optimisation, making complete automation difficult even if technically possible.
For these reasons, the competitive-automation pathway remains a forward-looking hypothesis rather than an established trend.
The strongest objections
Critics argue that the competitive race argument often underestimates both institutional resilience and the continuing value of human judgement.
Several counterarguments appear repeatedly.
Human judgement often creates competitive value. Trust, creativity, negotiation, political legitimacy and responsibility remain difficult to automate fully. Organisations may discover that customers, regulators and voters actively prefer visible human accountability.
Regulation can change incentives. Competition is shaped by rules. Safety standards, liability law and procurement requirements can make meaningful human oversight part of competition rather than an obstacle to it. The EU AI Act, for example, requires human oversight for many high-risk AI systems rather than leaving the issue entirely to market forces.[Springer]link.springer.comInstitutionalised distrust and human oversight of artificial intelligence: towards a democratic design of AI governance under the…
Automation is frequently partial rather than total. Empirical studies of AI adoption commonly find that work is reorganised rather than completely transferred to machines, with humans concentrating on higher-value or exceptional cases instead of disappearing entirely.[DROPS]drops.dagstuhl.deDROPSChallenges of Human Oversight: Achieving Human Control of AI-Based SystemsDROPSChallenges of Human Oversight: Achieving Human Control of AI-Based Systems
These objections do not necessarily reject gradual-disempowerment concerns. Instead, they question whether competitive incentives alone are strong enough to eliminate meaningful human authority across society.
What could preserve meaningful human roles?
Researchers increasingly argue that preserving human judgement requires changing institutional incentives rather than merely inserting a nominal reviewer into automated workflows.
Important proposals include:
- designing systems where humans retain genuine authority to reject or override AI outputs;
- ensuring reviewers have enough time, expertise and organisational independence to exercise judgement;
- measuring successful oversight instead of simply measuring automation rates;
- preserving human expertise through continued practice rather than allowing complete deskilling;
- requiring audit trails that record when and why humans disagree with AI recommendations;
- creating regulatory standards that reward meaningful oversight rather than symbolic compliance.[springer.com]link.springer.comDesigning meaningful human oversight in AI | AI and Ethics | Springer Nature Link…
Recent research increasingly emphasises that effective oversight depends on more than simply placing a person somewhere in the decision chain. Humans must possess the knowledge, authority, incentives and practical opportunity to exercise independent judgement. Otherwise, “human control” risks becoming a legal label attached to decisions that are, in practice, almost entirely automated.[nature.com]nature.comOpen source on nature.com.
Why this mechanism matters in AI doom debates
Competitive automation occupies an unusual position within AI existential-risk discussions because it does not rely on spectacular failures or deliberate machine rebellion.
Instead, it asks whether ordinary economic incentives could gradually remove humans from increasingly important decisions until meaningful control becomes difficult to recover. The mechanism therefore connects everyday organisational behaviour with much larger questions about humanity’s long-term ability to direct technological civilisation.
Whether this trajectory ultimately contributes to existential risk remains uncertain. It depends on future AI capabilities, institutional responses, regulatory choices and whether societies deliberately preserve meaningful human judgement even when doing so carries short-term economic costs. That uncertainty explains why competitive automation has become an increasingly important part of discussions about gradual human disempowerment rather than a settled prediction about AI’s future.[arxiv.org]arxiv.orgGradual Disempowerment: Systemic Existential Risks from Incremental AI DevelopmentJanuary 28, 2025…
Amazon book picks
Further Reading
Books and field guides related to Could Competition Make Human Control Too Expensive?. Use these as the next step if you want deeper reading beyond the article.
Power and Progress: Our Thousand-Year Struggle Over Technolog...
A bold reinterpretation of economics and history revealing why technology does not inevitably lead to shared prosperity, and how we must...
Race Against the Machine
Examines how information technologies are affecting jobs, skills, wages, and the economy.
The Technology Trap: Capital, Labor, and Power in the Age of...
A look at the history of technological progress and how it has radically shifted the distribution of economic and political power among s...
The Second Machine Age
A New York Times Bestseller. A “fascinating” (Thomas L. Friedman, New York Times) look at how digital technology is transforming our work...
eBay marketplace picks
Marketplace Samples
Live-tested eBay searches with available results related to this page.
Selected fromfactory robot model oneBay.co.uk.
Endnotes
1.
Source: arxiv.org
Link:https://arxiv.org/abs/2501.16946
Source snippet
Gradual Disempowerment: Systemic Existential Risks from Incremental AI DevelopmentJanuary 28, 2025...
Published: January 28, 2025
2.
Source: drops.dagstuhl.de
Title: DROPSChallenges of Human Oversight: Achieving Human Control of AI-Based Systems
Link:https://drops.dagstuhl.de/storage/04dagstuhl-reports/volume15/issue06/25272/html/DagRep.15.6.189/DagRep.15.6.189.html
3.
Source: link.springer.com
Link:https://link.springer.com/article/10.1007/s00146-023-01777-z
Source snippet
Institutionalised distrust and human oversight of artificial [intelligence]({{ 'hard-bottlenecks/' | relative_url }}): towards a democratic design of AI governance under the...
4.
Source: link.springer.com
Link:https://link.springer.com/article/10.1007/s43681-026-01147-7
Source snippet
Designing meaningful human oversight in AI | AI and Ethics | Springer Nature Link...
5.
Source: arxiv.org
Link:https://arxiv.org/abs/2502.10036
Source snippet
Automation Bias in the AI Act: On the Legal Implications of Attempting to De-Bias Human Oversight of AIFebruary 14, 2025...
Published: February 14, 2025
6.
Source: arxiv.org
Title: arXiv Human Oversight of Artificial Intelligence and Technical Standardisation
Link:https://arxiv.org/abs/2407.17481
7.
Source: nature.com
Link:https://www.nature.com/articles/s41746-026-02971-1
8.
Source: link.springer.com
Link:https://link.springer.com/article/10.1007/s41469-025-00194-4
9.
Source: nature.com
Link:https://www.nature.com/articles/s42256-025-00986-z
10.
Source: gradual-disempowerment.ai
Link:https://gradual-disempowerment.ai/
11.
Source: weforum.org
Link:https://www.weforum.org/stories/artificial-intelligence/oversight-paradox-human-control-ai/
Source snippet
World Economic ForumThe oversight paradox: Human control over AI may be eroding | World Economic ForumJuly 2, 2026...
Published: July 2, 2026
12.
Source: tandfonline.com
Link:https://www.tandfonline.com/doi/abs/10.1080/08850607.2026.2637905
Source snippet
Taylor & Francis OnlineHuman Oversight in AI-Driven Intelligence: Rhetoric, Reality, and the Risks of Automation: [International]({{ 'shared-testing/' | relative_url }}) Journal o...
13.
Source: doi.org
Link:https://doi.org/10.1002/mar.70136
14.
Source: tandfonline.com
Link:https://www.tandfonline.com/doi/full/10.1080/07421222.2025.2561390
Additional References
15.
Source: youtube.com
Title: Dan Hendrycks
Link:https://www.youtube.com/watch?v=arqYqHX13eM
Source snippet
Race to the bottom AI automation market competition existential risk I can change your mind about the AI hype NeetCodeIO...
16.
Source: oecd.org
Link:https://www.oecd.org/en/publications/governing-with-artificial-intelligence_795de142-en/full-report/how-artificial-intelligence-is-accelerating-the-digital-government-journey_d9552dc7.html
17.
Source: oecd.org
Link:https://www.oecd.org/en/publications/2025/06/governing-with-artificial-intelligence_398fa287/full-report/implementation-challenges-that-hinder-the-strategic-use-of-ai-in-government_05cfe2bb.html
18.
Source: youtube.com
Title: The Dark Side of Competition in AI | Liv Boeree | TED
Link:https://www.youtube.com/watch?v=WX_vN1QYgmE
Source snippet
Risks of Agentic AI: What You Need to Know About Autonomous AI...
19.
Source: youtube.com
Title: Risks of Agentic AI: What You Need to Know About Autonomous AI
Link:https://www.youtube.com/watch?v=v07Y4fmSi6Y
Source snippet
Dan Hendrycks - Avoiding an AGI Arms Race...
20.
Source: nist.gov
Title: [adversarial]({{ ‘concealment/’ | relative_url }}) machine learning taxonomy and terminology attacks and mitigations 0
Link:https://www.nist.gov/publications/adversarial-machine-learning-taxonomy-and-terminology-attacks-and-mitigations-0
21.
Source: youtube.com
Link:https://www.youtube.com/watch?v=bVV91jt56-o
Source snippet
Gradual Disempowerment by Nora Ammann...
22.
Source: nber.org
Link:https://www.nber.org/papers/w34259
23.
Source: ico.org.uk
Link:https://ico.org.uk/for-organisations/uk-gdpr-guidance-and-resources/artificial-intelligence/guidance-on-ai-and-data-protection/how-do-we-ensure-individual-rights-in-our-ai-systems/?q=machine%2Blearning
24.
Source: sciencedirect.com
Link:https://www.sciencedirect.com/science/article/pii/S0747563225002481