Within Disempowerment
What Happens When Humans Forget How to Override AI?
Human oversight can become fragile when organisations stop practising, teaching or retaining the skills needed to challenge automated systems.
On this page
- How assistance can become dependency
- Evidence from aviation, medicine and cognitive offloading
- Ways to retain independent human competence
Page outline Jump by section
Introduction
The idea that advanced AI could gradually reduce human control does not require machines to seize power overnight. A slower pathway is that people and institutions become so dependent on AI systems that they lose the practical ability to operate, verify or replace them. In this view, the greatest danger is not simply automation but deskilling: the erosion of human expertise through long-term reliance on increasingly capable systems.
Within debates about AI doom and existential risk, this matters because human oversight is only meaningful if humans retain the knowledge, judgement and confidence needed to challenge AI when it is wrong. If governments, companies, hospitals and critical infrastructure gradually stop exercising those capabilities, formal human authority could remain while effective control becomes increasingly difficult to reclaim. Today’s evidence comes mainly from aviation, medicine and cognitive psychology rather than from advanced AI itself, so it does not prove civilisation-wide disempowerment. However, it illustrates mechanisms that many AI safety researchers believe deserve close attention.[PubMed]pubmed.ncbi.nlm.nih.govComplacency and bias in human use of automation: an attentional integration - PubMed…
How assistance can become dependency
Most successful automation begins by improving performance. Flight-management computers reduce pilot workload. Clinical decision-support systems help doctors detect disease. Generative AI drafts documents and software. These tools often increase average productivity and reduce routine mistakes.
The concern begins when people increasingly monitor rather than actively perform a task. Skills that are rarely exercised become harder to recover, especially during emergencies when automation fails or encounters situations outside its training.
Researchers studying human interaction with automation have described several recurring mechanisms:
- Automation bias: people become more likely to accept computer recommendations, even when independent evidence points elsewhere.
- Automation complacency: operators monitor automated systems less carefully because they expect them to work.
- Cognitive offloading: memory, reasoning or problem-solving is increasingly delegated to external tools instead of being practised internally.
- Training erosion: fewer opportunities remain for novices to develop expertise because machines perform much of the work they would previously have learned by doing.[PubMed]pubmed.ncbi.nlm.nih.govComplacency and bias in human use of automation: an attentional integration - PubMed…
None of these effects automatically produces catastrophe. In many settings automation improves overall outcomes despite introducing new risks. The question for existential-risk discussions is whether these well-established patterns could eventually scale from individual workplaces to entire institutions responsible for governing increasingly capable AI.
What aviation teaches about keeping humans in the loop
Commercial aviation provides one of the longest-running real-world experiments in highly reliable automation.
Modern aircraft are vastly safer than earlier generations, and autopilot systems have contributed significantly to that improvement. Yet aviation safety researchers have repeatedly found that extensive automation creates new challenges rather than eliminating the need for human expertise.
The US Federal Aviation Administration warns that automation overreliance has contributed to accidents in which pilots became insufficiently proficient at manually flying aircraft or failed to recognise when automated systems behaved unexpectedly. Safety guidance emphasises maintaining proficiency both with and without automation, precisely because operators may eventually need to take over during unusual situations.[Federal Aviation Administration]faa.govOpen source on faa.gov.
Human-factors research has documented related phenomena over several decades. Rather than replacing human judgement, sophisticated automation often shifts people’s role towards supervising systems that function correctly almost all the time. Ironically, this can make the remaining human interventions rarer but more demanding: when operators are finally needed, they may have had little recent practice performing the task manually. Bainbridge described this as one of the classic “ironies of automation”, an idea that continues to shape modern AI human-factors research.[PubMed]pubmed.ncbi.nlm.nih.govPub Med Ironies of artificial intelligenceIronies of artificial intelligence - PubMed…
The aviation analogy has limits. Today’s AI systems differ substantially from autopilots in architecture and capability. Nevertheless, aviation demonstrates that making humans nominally responsible is not enough if routine operation steadily weakens the skills required for exceptional situations.
Medicine shows how expertise can weaken despite better average performance
Healthcare offers another useful case because AI increasingly assists diagnosis while clinicians remain legally and ethically responsible for patient care.
Recent reviews find growing evidence for three related problems.
First, clinicians may become susceptible to automation bias, accepting incorrect AI recommendations even when their own initial judgement was correct.
Second, AI can reduce opportunities to practise difficult diagnostic skills. If algorithms consistently perform initial screening, trainees may encounter fewer challenging cases during education.
Third, maintaining expertise becomes harder when important clinical reasoning is increasingly delegated to software. Several empirical studies reviewed in 2025 and 2026 found measurable examples of incorrect AI advice influencing expert judgement, while broader reviews concluded that concerns about diagnostic deskilling now have experimental support rather than being purely theoretical.[nih.gov]pubmed.ncbi.nlm.nih.govArtificial intelligence in medicine: a scoping review of the risk of deskilling and loss of expertise among physicians - PubMedMarc…
One frequently discussed example comes from pathology. Changes associated with human papillomavirus primary screening dramatically reduced traditional cytology workloads, shrinking opportunities for training and experience. While this transition cannot be attributed solely to AI, reviewers use it to illustrate how automation and workflow redesign together can reduce opportunities for maintaining specialist expertise.[PubMed]pubmed.ncbi.nlm.nih.govArtificial intelligence in medicine: a scoping review of the risk of deskilling and loss of expertise among physicians - PubMedMarc…
A more direct AI example emerged from research on AI-assisted colonoscopy. Investigators reported that clinicians who routinely used AI assistance showed reduced performance during procedures conducted without AI after several months. The authors suggested that dependence on AI may have weakened visual search habits, although outside experts cautioned that workload and other factors could also have contributed. The study is important precisely because it illustrates both the evidence and the uncertainty: it points towards possible deskilling but does not establish that AI inevitably causes it across medicine.[TIME]time.comNew Study Suggests Using AI Made Doctors Less Skilled at Spotting CancerConducted across four endoscopy centers in Poland as part of the AI in Colonoscopy for Cancer Prevention (ACCEPT) trial, researchers foun…
Cognitive offloading is real, but its long-term effects remain uncertain
Humans have always offloaded cognition. Writing reduced the need to memorise long texts. Calculators reduced routine arithmetic. Satellite navigation changed how many people navigate unfamiliar places.
Generative AI differs because it can increasingly perform complex reasoning, drafting, coding and analysis rather than simply storing or retrieving information.
Researchers distinguish between productive offloading, which frees attention for more valuable work, and harmful offloading, where people cease practising skills they may later need independently. The balance depends on how AI is integrated into everyday work.
Evidence for widespread long-term cognitive decline caused by AI remains limited. Existing studies often involve short laboratory tasks, educational settings or particular professions. Nevertheless, there is growing evidence that heavy reliance on AI increases cognitive offloading and may reduce independent critical evaluation unless users deliberately verify outputs.[arXiv]arxiv.orgLarge Language Models and User Trust: Consequence of Self-Referential Learning Loop and the Deskilling of Healthcare ProfessionalsMa…
For AI doom arguments, the concern is cumulative rather than individual. A society where millions of professionals rely on AI for reasoning could eventually find that fewer people retain the expertise needed to detect subtle failures or rebuild essential capabilities after disruption.
Why institutional dependence matters more than individual convenience
Individual deskilling becomes more significant when it occurs simultaneously across many organisations.
Imagine an institution where:
- AI drafts most policy recommendations.
- Human staff primarily approve outputs.
- Training increasingly assumes AI assistance.
- Retired experts are not replaced because automation appears cheaper.
- Independent verification becomes rare because AI is usually correct.
Each decision may appear economically rational. Yet after years of optimisation, the organisation may struggle to function if AI systems fail, become unavailable or begin producing systematically misleading recommendations.
This institutional dependence differs from ordinary software reliance. If essential expertise disappears across an entire profession, rebuilding it may require years rather than days. Organisations can lose not only practical skills but also the ability to recognise when those skills have been lost.
Within gradual human disempowerment scenarios, this creates a feedback loop. Better AI encourages greater delegation, greater delegation reduces human competence, weaker human competence makes oversight less reliable, and declining oversight encourages still greater dependence.
Whether this dynamic would ever reach civilisation-wide scale remains highly uncertain. Present-day institutions continue to rely heavily on human expertise, and many sectors deliberately preserve manual procedures precisely because automation can fail.
Does this strengthen AI doom arguments?
Deskilling alone is not an existential risk.
A world with widespread AI dependence could remain prosperous, democratic and ultimately controllable if people preserve enough independent competence to replace or correct automated systems when necessary.
However, many researchers concerned about advanced AI view deskilling as an important enabling condition rather than the primary danger.
If future AI systems became substantially more capable, deceptive or strategically autonomous, effective human oversight would depend on experienced operators who could:
- recognise unusual behaviour,
- verify AI outputs independently,
- intervene during unexpected failures,
- redesign institutions when automation proves inadequate.
Those abilities cannot be improvised during a crisis if they have gradually disappeared over decades.
Critics of AI doom argue that this concern may underestimate human adaptability. History contains many examples of new technologies replacing some skills while creating new forms of expertise. Modern pilots, for example, know far more about managing complex automated systems than pilots from previous generations. Likewise, clinicians increasingly develop skills in evaluating AI outputs rather than performing every diagnostic task manually.
This objection is important because technological change rarely produces simple one-way deskilling. Many occupations simultaneously lose some competencies while gaining others. The central unresolved question is whether future AI capabilities might eventually outpace the development of equally robust human oversight skills.[PubMed]pubmed.ncbi.nlm.nih.govPub Med Ironies of artificial intelligenceIronies of artificial intelligence - PubMed…
How institutions can preserve independent human competence
Research from aviation, medicine and human-factors engineering points towards practical ways of reducing institutional dependence without abandoning useful AI.
Common recommendations include:
- Regular manual practice, ensuring professionals periodically perform important tasks without AI assistance.
- Independent verification, requiring humans to reach an initial judgement before viewing AI recommendations where practical.
- Training for failure cases, exposing staff to scenarios where automation behaves incorrectly rather than only demonstrating successful operation.
- Maintaining deep expertise, preserving specialist career paths even when automation reduces routine workload.
- Meaningful accountability, ensuring people remain responsible not merely for approving AI outputs but for understanding and challenging them.
- Designing AI to support rather than replace judgement, making systems explain uncertainty and alternative possibilities instead of presenting overly authoritative answers.[faa.gov]faa.govOpen source on faa.gov.
These measures reflect a broader principle increasingly discussed in AI safety: resilience depends not only on building trustworthy AI but also on preserving trustworthy human institutions capable of questioning, correcting and, if necessary, overriding it.
Amazon book picks
Further Reading
Books and field guides related to What Happens When Humans Forget How to Override AI?. Use these as the next step if you want deeper reading beyond the article.
Human Compatible
A leading artificial intelligence researcher lays out a new approach to AI that will enable us to coexist successfully with increasingly...
The Second Machine Age
A New York Times Bestseller. A “fascinating” (Thomas L. Friedman, New York Times) look at how digital technology is transforming our work...
The Glass Cage
At once a celebration of technology and a warning about its misuse, The Glass Cage will change the way you think about the tools you use...
The Shallows
'Boldly reactionary... What looks like feast, Carr argues, may be closer to famine' Sunday Times 'Chilling' The Economist In this ground-...
eBay marketplace picks
Marketplace Samples
Live-tested eBay searches with available results related to this page.
Selected fromautomation poster oneBay.co.uk.
Endnotes
1.
Source: time.com
Title: New Study Suggests Using AI Made Doctors Less Skilled at Spotting Cancer
Link:https://time.com/7309274/ai-lancet-study-artificial-[intelligence
Source snippet
Conducted across four endoscopy centers in Poland as part of the AI in Colonoscopy for Cancer Prevention (ACCEPT) trial, researchers foun...
2.
Source: arxiv.org
Link:https://arxiv.org/abs/2403.14691
Source snippet
Large Language Models and User Trust: Consequence of Self-Referential Learning Loop and the Deskilling of Healthcare ProfessionalsMa...
3.
Source: arxiv.org
Link:https://arxiv.org/abs/2503.03924
4.
Source: youtube.com
Title: The Glass Cage: Automation and Us | Nicholas Carr | Talks at Google
Link:https://www.youtube.com/watch?v=Mt8ooCms4sE
Source snippet
Automation Complacency - AI Induced Abdication of Medical Decision Making...
5.
Source: youtube.com
Title: Automation Complacency
Link:https://www.youtube.com/watch?v=-cJ4zfcpvag
Source snippet
Peter Brodeur, MD – AI in Healthcare: What Helps, What Hurts, and What Comes Next...
6.
Source: pubmed.ncbi.nlm.nih.gov
Link:https://pubmed.ncbi.nlm.nih.gov/21077562/
Source snippet
Complacency and bias in human use of automation: an attentional integration - PubMed...
7.
Source: pubmed.ncbi.nlm.nih.gov
Title: Pub Med Ironies of artificial intelligence
Link:https://pubmed.ncbi.nlm.nih.gov/37534468/
Source snippet
Ironies of artificial intelligence - PubMed...
8.
Source: faa.gov
Link:https://www.faa.gov/newsroom/safety-briefing/cfitautomation-overreliance
9.
Source: pubmed.ncbi.nlm.nih.gov
Link:https://pubmed.ncbi.nlm.nih.gov/41890350
Source snippet
Artificial intelligence in medicine: a scoping review of the risk of deskilling and loss of expertise among physicians - PubMedMarc...
10.
Source: pmc.ncbi.nlm.nih.gov
Link:https://pmc.ncbi.nlm.nih.gov/articles/PMC13015734/
Source snippet
PubMed Central (PMC)Artificial intelligence in medicine: a scoping review of the risk of deskilling and loss of expertise among physician...
11.
Source: pubmed.ncbi.nlm.nih.gov
Link:https://pubmed.ncbi.nlm.nih.gov/42445869/
Source snippet
Generative Artificial Intelligence in Healthcare: Automation Bias, Deskilling, and Cognitive Implications - A Systematic Review - P...
12.
Source: pmc.ncbi.nlm.nih.gov
Link:https://pmc.ncbi.nlm.nih.gov/articles/PMC13360813/
14.
Source: pubmed.ncbi.nlm.nih.gov
Link:https://pubmed.ncbi.nlm.nih.gov/25886768/
15.
Source: faa.gov
Title: Office of Aerospace Medicine Technical Reports | Federal Aviation Administration
Link:https://www.faa.gov/data_research/research/med_humanfacs/oamtechreports/1990s/1997/9724
16.
Source: faa.gov
Title: Office of Aerospace Medicine Technical Reports | Federal Aviation Administration
Link:https://www.faa.gov/data_research/research/med_humanfacs/oamtechreports/1990s/1997/9706
17.
Source: sciencedirect.com
Title: Ironies of automation
Link:https://www.sciencedirect.com/science/article/abs/pii/0005109883900468
18.
Source: sciencedirect.com
Title: Ironies of automation
Link:https://www.sciencedirect.com/science/article/pii/0005109883900468
Additional References
19.
Source: youtube.com
Title: What Happens to Your Brain When You Use AI Every Day
Link:https://www.youtube.com/watch?v=syYspHSnXGc
Source snippet
The Glass Cage: Automation and Us | Nicholas Carr | Talks at Google explores the fundamental mechanisms of cognitive offloading, skill de...
20.
Source: ora.ox.ac.uk
Link:https://ora.ox.ac.uk/objects/uuid%3A3e31de3c-ef13-40d4-9bb9-d5983f0f8201
Source snippet
Oxford University Research Archive...
21.
Source: youtube.com
Link:https://www.youtube.com/watch?v=6sp03fwKi1Y
Source snippet
Chapter 8 Overview: "Automation & Human-Automation Interaction"...
22.
Source: nature.com
Link:https://www.nature.com/articles/s41746-026-02971-1
23.
Source: digital.ahrq.gov
Link:https://digital.ahrq.gov/key-topics/clinical-decision-support/clinical-practice-improvement-and-redesign-how-change-workflow-can-be-supported-clinical-decision
24.
Source: sciencedirect.com
Link:https://www.sciencedirect.com/science/article/pii/S2451958826001764
25.
Source: cambridge.org
Link:https://www.cambridge.org/core/journals/the-british-journal-of-psychiatry/article/artificial-intelligence-and-deskilling-in-medicine/641F0B39DC9494C7B9561F63E718EAC6
26.
Source: sciencedirect.com
Link:https://www.sciencedirect.com/science/article/pii/S2949820126000123
27.
Source: frontiersin.org
Link:https://www.frontiersin.org/journals/psychology/articles/10.3389/fpsyg.2019.00225/full
28.
Source: sciencedirect.com
Link:https://www.sciencedirect.com/science/article/pii/S1071581908000724