Within Automation Race
Can Regulation Stop the Race to Remove People?
Liability, safety standards and procurement rules can make substantive human oversight a requirement rather than a commercial disadvantage.
On this page
- How regulation can reshape competitive incentives
- What effective human oversight duties need to require
- Where rules may fail or become box ticking
Page outline Jump by section
Introduction
If competition rewards organisations that remove people from important decisions, then preserving meaningful human judgement can become an economic disadvantage. Within debates about AI doom and existential risk, one proposed response is to change the rules of competition rather than simply asking companies to behave differently. The idea is that liability law, safety regulation, procurement standards and sector-specific rules can make accountable human oversight part of the cost of doing business instead of an optional expense.
This approach does not assume that regulation alone can prevent long-term loss of human control. Rather, it attempts to reshape incentives so that organisations are not commercially punished for keeping trained people responsible for consequential decisions. Whether that is sufficient remains disputed, but it has become a central governance proposal in discussions about competitive automation and gradual human disempowerment.
How regulation can reshape competitive incentives
The competitive pressure described in AI-risk discussions arises because individual organisations may benefit from replacing human judgement even if society as a whole would prefer that important decisions remain accountable. Economists describe this as an incentive problem rather than simply a technology problem.
Regulation attempts to change those incentives in several ways:
- Liability: Organisations remain legally responsible for decisions even when AI systems contribute to them, reducing the incentive to treat automation as a way to shift responsibility.
- Mandatory oversight: Certain categories of AI cannot legally operate without effective human supervision.
- Safety standards: Organisations must demonstrate that oversight is genuine rather than merely documented.
- Public procurement: Governments can require human accountability as a condition for purchasing AI systems, rewarding suppliers that design for oversight rather than full autonomy.
- Auditing and record keeping: Regulators can require organisations to preserve evidence showing how human reviewers assessed or overrode AI recommendations.
Instead of making human judgement an internal cost, these mechanisms make it part of regulatory compliance. If competitors must satisfy the same requirements, removing people from critical decisions no longer provides the same commercial advantage.
For AI doom researchers concerned about gradual loss of control, this matters because competitive races are driven by incentives. Changing those incentives may reduce pressure to automate away meaningful human authority.
What effective human-oversight duties need to require
One lesson from AI governance over the past several years is that simply requiring “a human in the loop” is often inadequate. Researchers distinguish between nominal oversight—where a person technically approves decisions—and substantive oversight, where that person genuinely exercises judgement.
The European Union’s AI Act illustrates this distinction. For high-risk AI systems, human oversight must be designed into both the system and its deployment. Oversight personnel should understand the system’s capabilities and limitations, recognise the risk of excessive reliance on AI outputs (automation bias), interpret outputs appropriately, override recommendations when necessary, and intervene or stop system operation where appropriate. Oversight measures must also match the risks and degree of autonomy involved.[AI Act Service Desk]ai-act-service-desk.ec.europa.euAI Act Service Desk Article 14: Human oversight | AI Act Service DeskAI Act Service DeskArticle 14: Human oversight | AI Act Service DeskJune 13, 2024…
From the perspective of maintaining competitively viable human judgement, several characteristics matter more than the simple presence of a reviewer.
Humans must have real authority
Oversight is meaningful only if reviewers can reject, delay or reverse AI outputs without suffering systematic organisational penalties.
If employees are expected to approve AI recommendations almost automatically because questioning them slows operations or reduces performance metrics, the oversight role becomes symbolic. In practice, commercial incentives can quietly overwhelm formal governance rules.
Reviewers need sufficient understanding
A human who cannot recognise when a model is failing cannot provide effective supervision.
The AI Act therefore links oversight to competence rather than mere presence. People responsible for supervision must understand relevant limitations, unexpected behaviour and situations in which the AI should not be trusted.[AI Act Service Desk]ai-act-service-desk.ec.europa.euAI Act Service Desk Article 14: Human oversight | AI Act Service DeskAI Act Service DeskArticle 14: Human oversight | AI Act Service DeskJune 13, 2024…
Systems should support intervention
Oversight depends partly on interface design.
Reviewers need access to information explaining what the system has done, why it reached a recommendation where possible, and how to interrupt or override its operation safely. Without these capabilities, humans may formally remain responsible while lacking practical control.
Accountability should remain with organisations
One objective of oversight rules is preventing responsibility from disappearing into complex technical systems.
If regulators, courts and procurement authorities continue holding organisations accountable for AI-assisted decisions, companies retain incentives to invest in capable reviewers rather than treating human involvement as unnecessary bureaucracy.
Why procurement rules matter as much as regulation
Governments are often among the largest purchasers of advanced technology. Procurement requirements can therefore shape markets even without banning particular systems.
For example, public contracts may require suppliers to demonstrate:
- documented human review procedures;
- audit trails showing when people accepted or rejected AI advice;
- staff training for oversight roles;
- mechanisms for escalating uncertain cases to human experts;
- evidence that reviewers are not pressured into automatic approval.
These requirements affect competition because suppliers unable to provide accountable oversight may lose access to valuable contracts.
Advocates argue that procurement can influence industry behaviour more rapidly than legislation alone, particularly where governments purchase AI for healthcare, infrastructure, administration or other critical public services.
Liability can make human judgement economically valuable
Liability rules influence organisational behaviour by determining who bears the cost when AI-assisted decisions cause serious harm.
If firms remain responsible regardless of how autonomous their systems become, eliminating human review does not eliminate legal exposure. Instead, organisations must decide whether reduced staffing genuinely outweighs increased litigation, regulatory penalties and reputational damage.
From an existential-risk perspective, this does not directly solve long-term alignment problems. It does, however, reduce one mechanism through which competition might otherwise encourage progressively weaker human supervision across increasingly important systems.
Where oversight rules may fail
Supporters of governance measures generally acknowledge that regulation is not automatically effective. Several recurring failure modes appear in both academic work and policy discussions.
Box-ticking compliance. Organisations may satisfy formal documentation requirements without changing real decision-making. Employees sign approval forms while AI outputs are rarely questioned.
Automation bias. Even well-trained reviewers tend to over-trust computer recommendations, particularly when systems appear highly accurate. Legal requirements cannot eliminate this psychological tendency by themselves. The EU AI Act explicitly recognises this risk by requiring oversight measures that address over-reliance on AI outputs.[AI Act Service Desk]ai-act-service-desk.ec.europa.euAI Act Service Desk Article 14: Human oversight | AI Act Service DeskAI Act Service DeskArticle 14: Human oversight | AI Act Service DeskJune 13, 2024…
Commercial pressure. Managers may reward speed and throughput even while formally requiring human review. Staff can therefore experience conflicting incentives.
Insufficient expertise. Oversight becomes ineffective if reviewers lack time, technical understanding or organisational authority.
Scaling problems. As AI systems make larger numbers of decisions, organisations may struggle to provide genuinely independent review for every consequential output.
Researchers examining compliance with human-oversight requirements have argued that these problems make evaluation difficult. Measuring whether oversight genuinely improves decisions is substantially harder than verifying whether documentation exists, creating a risk that regulation encourages procedural compliance rather than meaningful supervision.[arXiv]arxiv.orgHow to Test for Compliance with Human Oversight Requirements in AI Regulation?April 4, 2025…
Can regulation stop the race to remove people?
Within AI doom debates, opinions differ considerably.
Supporters argue that competitive races are partly products of institutional rules. If governments require meaningful human accountability in high-impact applications, organisations no longer gain as much advantage from eliminating people entirely. The objective is not to prevent automation but to preserve human responsibility where mistakes could have systemic consequences.
Critics question whether regulation can keep pace with rapidly advancing AI capabilities. Firms operating in jurisdictions with weaker requirements may obtain competitive advantages, enforcement resources may prove inadequate, and organisations may learn to satisfy oversight obligations without preserving genuine human agency. Some also worry that overly burdensome regulation could slow beneficial innovation without materially improving safety.[ft.com]ft.comThe legislation categorizes AI systems by risk levels, with high-risk systems facing the most stringent regulations. Cleve supports AI re…
From the narrower perspective of existential-risk governance, oversight rules should therefore be understood as one layer of defence rather than a complete solution. They primarily address competitive incentives that encourage removing humans from important decisions. They do not by themselves resolve deeper questions about advanced AI alignment, deceptive behaviour, highly autonomous systems or international coordination.
Nevertheless, if gradual human disempowerment depends on markets consistently rewarding the replacement of accountable judgement, then regulations, liability regimes and procurement standards that preserve substantive human authority may reduce one of the mechanisms through which that disempowerment could otherwise become self-reinforcing.
Amazon book picks
Further Reading
Books and field guides related to Can Regulation Stop the Race to Remove People?. Use these as the next step if you want deeper reading beyond the article.
The Coming Wave
"We are approaching a critical threshold in the history of our species. Everything is about to change. Soon you will live surrounded by A...
Human Compatible
A leading artificial intelligence researcher lays out a new approach to AI that will enable us to coexist successfully with increasingly...
Power and Progress
A bold reinterpretation of economics and history revealing why technology does not inevitably lead to shared prosperity, and how we must...
Atlas of AI
The hidden costs of artificial intelligence--from natural resources and labor to privacy, equality, and freedom "This study argues that [...
eBay marketplace picks
Marketplace Samples
Live-tested eBay searches with available results related to this page.
Selected fromtechnology policy pin oneBay.co.uk.
Endnotes
1.
Source: arxiv.org
Link:https://arxiv.org/abs/2504.03300
Source snippet
How to Test for Compliance with Human Oversight Requirements in AI Regulation?April 4, 2025...
Published: April 4, 2025
2.
Source: arxiv.org
Title: arXiv Human Oversight of Artificial Intelligence and Technical Standardisation
Link:https://arxiv.org/abs/2407.17481
3.
Source: ft.com
Link:https://www.ft.com/content/6cc7847a-2fc5-4df0-b113-a435d6426c81
Source snippet
The legislation categorizes AI systems by risk levels, with high-risk systems facing the most stringent regulations. Cleve supports AI re...
4.
Source: en.ai-act.io
Link:https://en.ai-act.io/article/high-risk-ai-systems/requirements-for-high-risk-ai-systems/human-oversight
5.
Source: ai-act-service-desk.ec.europa.eu
Title: AI Act Service Desk Article 14: Human oversight | AI Act Service Desk
Link:https://ai-act-service-desk.ec.europa.eu/en/ai-act/article-14
Source snippet
AI Act Service DeskArticle 14: Human oversight | AI Act Service DeskJune 13, 2024...
Published: June 13, 2024
6.
Source: digital-strategy.ec.europa.eu
Title: eu Navigating the AI Act | Shaping Europe’s digital future
Link:https://digital-strategy.ec.europa.eu/en/faqs/navigating-ai-act
Source snippet
Before placing a high-risk AI system on the EU market or otherwise putting it into service, providers must subject it to a conformity a...
7.
Source: regulation-ai.eu
Title: article 14
Link:https://www.regulation-ai.eu/en/articles/article-14/
8.
Source: eur-lex.europa.eu
Title: eu Regulation
Link:https://eur-lex.europa.eu/legal-content/EN/TXT/?qid=1402647301340&uri=CELEX%3A32024R1689
9.
Source: eur-lex.europa.eu
Title: eu Regulation
Link:https://eur-lex.europa.eu/legal-content/EN/TXT/?hl=en-US&uri=CELEX%3A32024R1689
10.
Source: ai-act-service-desk.ec.europa.eu
Title: eu Recital 73 | AI Act Service Desk
Link:https://ai-act-service-desk.ec.europa.eu/en/ai-act/recital-73
11.
Source: ai-act-service-desk.ec.europa.eu
Title: eu Recital 91 | AI Act Service Desk
Link:https://ai-act-service-desk.ec.europa.eu/en/ai-act/recital-91
12.
Source: ai-act-service-desk.ec.europa.eu
Title: eu Annex III | AI Act Service Desk
Link:https://ai-act-service-desk.ec.europa.eu/en/ai-act/annex-3
13.
Source: europarl.europa.eu
Title: eu Texts adopted
Link:https://www.europarl.europa.eu/doceo/document/TA-9-2023-0236_EN.html
Additional References
14.
Source: nature.com
Link:https://www.nature.com/articles/s41746-026-02971-1
Source snippet
July 23, 2026 — Meaningful oversight of medical AI beyond human in the loop Download PDF Download PDF * Comment * Open access *...
Published: July 23, 2026
15.
Source: mdpi.com
Link:https://www.mdpi.com/2079-8954/14/7/849
Source snippet
July 17, 2026 — Background: Open Access Article OPERATIONALISING HUMAN-CENTRED AI GOVERNANCE UNDER THE EU AI ACT: A GOVERNANCE FRAMEWORK...
Published: July 17, 2026
16.
Source: youtube.com
Title: Your AI Agent Just Acted on Its Own. Who’s Liable — You or the Machine?
Link:https://www.youtube.com/watch?v=UBnPRMd0HiU
Source snippet
Human Oversight in AI | Why People Must Stay in the Loop #2026...
17.
Source: service.betterregulation.com
Link:https://service.betterregulation.com/document/2026/08/02/742206
Source snippet
"14 Human oversight | Regulation 2024/1689/EU - Artificial Intelligence Act (EU AI Act) | Better RegulationToday — [https://service.betterr..."](https://service.betterr...")...
18.
Source: ico.org.uk
Link:https://ico.org.uk/for-organisations/uk-gdpr-guidance-and-resources/artificial-intelligence/guidance-on-ai-and-data-protection/how-do-we-ensure-individual-rights-in-our-ai-systems/?q=meeting
19.
Source: euai-act.com
Link:https://www.euai-act.com/articles/high-risk-ai-systems-requirements
20.
Source: youtube.com
Title: When to Use AI, Rules, or Humans
Link:https://www.youtube.com/watch?v=RZ7VCg-_NL0
Source snippet
What Is Algorithmic Accountability in the Context of AI Systems?...
21.
Source: youtube.com
Title: Why We Need AI Oversight
Link:https://www.youtube.com/watch?v=3bRr0wC2zdw
Source snippet
Your AI Agent Just Acted on Its Own. Who's Liable — You or the Machine?...
22.
Source: GOV.UK
Title: www.gov.uk PP N 017: Improving transparency of AI use in procurement (HTML)
Link:https://www.gov.uk/government/publications/ppn-017-improving-transparency-of-ai-use-in-procurement/ppn-017-improving-transparency-of-ai-use-in-procurement-html
23.
Source: nist.gov
Title: trustworthy ai managing risks artificial intelligence
Link:https://www.nist.gov/speech-testimony/trustworthy-ai-managing-risks-artificial-intelligence