Within AI Deskilling
What Happens When Humans Must Suddenly Take Over?
When AI handles routine work, the rare moments requiring human takeover can become harder precisely because people have had too little recent practice.
On this page
- The irony of reliable automation
- How monitoring weakens active skill
- Lessons from aviation for advanced AI oversight
Page outline Jump by section
Introduction
Highly reliable AI systems create a paradox. The better they become at handling routine work, the less often human operators need to intervene. That usually improves day-to-day performance, but it can also leave people less prepared for the rare moments when the system fails, behaves unexpectedly, or encounters a situation outside its capabilities. Human-factors researchers have studied this pattern for decades in aviation and other highly automated industries, where it is often described as the out-of-the-loop performance problem or one of the ironies of automation.[Sage Journals]journals.sagepub.comSage JournalsThe Out-of-the-Loop Performance Problem and Level of Control in Automation - Mica R. Endsley, Esin O. Kiris, 1995…
Within debates about AI doom and existential risk, this mechanism matters because many proposed AI safety strategies assume that humans can intervene if an advanced system begins behaving dangerously. If operators rarely exercise the skills needed to supervise, challenge or override increasingly capable AI, then formal human authority may remain while practical human control weakens. This concern does not prove that advanced AI will become uncontrollable, but it identifies a plausible mechanism by which meaningful oversight could erode precisely because AI appears to work so well most of the time.
The irony of reliable automation
At first glance, more reliable automation should make human supervision easier. If AI makes fewer mistakes than people, operators spend less time correcting errors and more time monitoring successful performance.
The problem is that monitoring is a very different cognitive activity from doing. Actively controlling a complex system requires continuous decision-making, prediction and feedback. Watching a system that almost always succeeds requires sustained vigilance but little active engagement. Human attention is poorly suited to this role. Decades of research have found that people become less effective at detecting rare failures when those failures occur infrequently, particularly after long periods of successful automation.[Sage Journals]journals.sagepub.comSage JournalsThe Out-of-the-Loop Performance Problem and Level of Control in Automation - Mica R. Endsley, Esin O. Kiris, 1995…
This creates a paradox:
- Better automation reduces routine workload.
- Reduced workload means fewer opportunities to practise difficult skills.
- Less practice weakens both technical ability and situational awareness.
- When an unusual failure finally occurs, it often demands faster and better human performance than before automation existed.
Instead of replacing human expertise, highly reliable automation can concentrate the remaining human role into the rarest and most demanding situations.
How monitoring weakens active skill
The loss of preparedness is not simply a matter of forgetting procedures. Several overlapping psychological mechanisms contribute.
First, manual skills decay through lack of use. Skills ranging from flying an aircraft to diagnosing complex technical faults improve through repeated practice. If AI performs nearly every routine case, operators accumulate far fewer opportunities to maintain those abilities.
Second, situational awareness becomes shallower. Operators who continuously make decisions build an internal understanding of what the system is doing and why. Passive monitoring encourages people to observe outcomes rather than reconstruct the underlying process. When automation suddenly fails, rebuilding that mental picture takes valuable time. Human-factors research on the out-of-the-loop problem identifies reduced situational awareness as one of the principal reasons operators struggle during automation failures.[Sage Journals]journals.sagepub.comSage JournalsThe Out-of-the-Loop Performance Problem and Level of Control in Automation - Mica R. Endsley, Esin O. Kiris, 1995…
Third, rare failures produce surprise. Humans respond poorly when unexpected events interrupt long periods of normal operation. The initial seconds after an automation failure may be spent identifying what has happened rather than correcting it. In time-critical systems, that delay can matter more than the original malfunction.
Finally, confidence can become miscalibrated. Successful automation encourages appropriate trust most of the time, but repeated success may also make operators less inclined to question unusual outputs or recognise subtle signs that the system is entering an unfamiliar state.
None of these mechanisms imply that people inevitably become incapable. Their strength depends on training, system design, organisational culture and how frequently operators remain actively engaged.
Lessons from aviation for advanced AI oversight
Commercial aviation provides one of the best-studied examples because modern aircraft combine extremely high reliability with periods when pilots may suddenly need to take manual control.
Automation has made aviation dramatically safer overall, but safety organisations have repeatedly warned against automation overreliance. The US Federal Aviation Administration advises that pilots maintain proficiency both with and without automation because automated systems can reduce manual flying skills and contribute to unexpected situations if crews are not prepared to intervene.[Federal Aviation Administration]faa.govOpen source on faa.gov.
Several broader lessons emerge from aviation research.
Reliable automation does not eliminate training needs. As systems improve, maintaining human expertise often requires deliberate practice rather than relying on everyday experience.
Emergency interventions are unusually demanding. Operators typically regain control only when conditions have already become abnormal, increasing workload at exactly the moment their recent experience is least relevant.
Understanding the automation matters as much as operating it. Pilots are trained not only to fly manually but also to understand the modes, assumptions and limitations of automated flight systems so that unexpected behaviour can be recognised early.[Federal Aviation Administration]faa.govOpen source on faa.gov.
These lessons are frequently cited by researchers discussing advanced AI because future operators may occupy a similar supervisory role: overseeing systems that function correctly almost all the time until an unusual failure requires rapid human judgement.
Why this matters more for advanced AI than ordinary software
Many conventional computer failures are obvious. A program crashes, produces an error message or stops responding.
Advanced AI systems may fail differently. A highly capable model could continue producing fluent, apparently reasonable outputs while gradually drifting away from the operator’s intentions. Detecting such failures may require expertise rather than simply noticing that the software has stopped working.
This changes the oversight challenge. If humans spend years mainly approving AI-generated analyses, plans or recommendations, they may lose both the habits and confidence needed to independently verify difficult cases. When an unusual situation finally requires independent reasoning, operators may have little recent experience performing the task unaided.
For researchers concerned about AI doom, this creates a possible feedback loop. More capable AI encourages greater institutional dependence, greater dependence reduces opportunities to maintain human expertise, and reduced expertise makes it harder to recognise or correct increasingly capable AI when exceptional situations arise. This is a mechanism rather than evidence that such an outcome will occur, but it explains why some safety researchers worry that “keeping a human in the loop” may not guarantee meaningful human control if the human has become largely a passive supervisor.
How strong is the evidence?
The underlying human-factors evidence is considerably stronger than the claim that it will scale to advanced AI.
There is broad agreement that prolonged reliance on automation can contribute to skill decay, reduced vigilance and out-of-the-loop performance in domains such as aviation, industrial control and other safety-critical environments. These findings have been replicated across decades of research and continue to influence safety guidance.[sagepub.com]journals.sagepub.comSage JournalsThe Out-of-the-Loop Performance Problem and Level of Control in Automation - Mica R. Endsley, Esin O. Kiris, 1995…
What remains uncertain is whether the same effects would become severe enough to affect governance of frontier AI systems.
Several important objections are raised:
- Human expertise does not inevitably disappear if organisations deliberately schedule regular practice, simulation and independent verification.
- AI systems might eventually support rather than replace human understanding by providing explanations, uncertainty estimates or interactive training.
- Some tasks may genuinely become safer when humans intervene less often, because human error is itself a major source of accidents.
- Institutions can redesign work so that operators remain actively engaged instead of merely monitoring automated outputs.
These objections mean that deskilling should be viewed as a manageable risk rather than an unavoidable consequence of automation.
What would reduce the takeover gap?
If rare interventions become the critical safety function, organisations may need to train specifically for those rare events rather than relying on everyday experience.
Measures often proposed include:
- Regular manual practice even when automation performs well.
- Simulated failures that require operators to diagnose unfamiliar AI behaviour.
- Rotating responsibilities so experts continue performing core tasks directly.
- Interfaces that expose the AI’s reasoning process, uncertainty and operating assumptions instead of presenting only final answers.
- Independent cross-checks for high-consequence decisions rather than requiring humans merely to approve AI recommendations.
- Clear procedures defining when automation should be disengaged and when escalation is mandatory.
The common theme is that human oversight should remain an active skill, not a ceremonial approval step.
Within the wider debate about AI doom, the concern is therefore not simply that advanced AI might make mistakes. It is that sustained success could gradually reduce the practical ability of people and institutions to respond effectively when those rare but potentially crucial mistakes eventually occur.
Amazon book picks
Further Reading
Books and field guides related to What Happens When Humans Must Suddenly Take Over?. Use these as the next step if you want deeper reading beyond the article.
The Glass Cage
At once a celebration of technology and a warning about its misuse, The Glass Cage will change the way you think about the tools you use...
The Checklist Manifesto
THE GAME-CHANGING BOOK FROM THE BESTSELLING AUTHOR OF BEING MORTAL Today we find ourselves in possession of stupendous know-how, which we...
The Human Factor
Technology innovation is progressing so quickly that we have fallen behind our ability to manage it. Our world is filled with objects tha...
Sources of Power
Anyone who watches the television news has seen images of firefighters rescuing people from burning buildings and paramedics treating bom...
eBay marketplace picks
Marketplace Samples
Live-tested eBay searches with available results related to this page.
Selected fromaviation poster oneBay.co.uk.
Endnotes
1.
Source: journals.sagepub.com
Link:https://journals.sagepub.com/doi/10.1518/001872095779064555
Source snippet
Sage JournalsThe Out-of-the-Loop Performance Problem and Level of Control in Automation - Mica R. Endsley, Esin O. Kiris, 1995...
2.
Source: faa.gov
Link:https://www.faa.gov/newsroom/safety-briefing/cfitautomation-overreliance
3.
Source: Wikipedia
Title: Out-of-the-loop performance problem
Link:https://en.wikipedia.org/wiki/Out-of-the-loop_performance_problem
4.
Source: faa.gov
Link:https://www.faa.gov/sites/faa.gov/files/MayJun2025.pdf
Source snippet
Federal Aviation AdministrationMay/June 2025...
Published: June 2025
5.
Source: faa.gov
Title: mayjune 2025 faa safety briefing magazine
Link:https://www.faa.gov/newsroom/safety-briefing/mayjune-2025-faa-safety-briefing-magazine
Source snippet
May/June 2025 FAA Safety Briefing Magazine | Federal Aviation AdministrationMay 5, 2025 — MAY/JUNE 2025 FAA SAFETY BRIEFING MAGAZINE MayJ...
Published: june 2025
6.
Source: journals.sagepub.com
Link:https://journals.sagepub.com/doi/abs/10.1177/15553434231222059
7.
Source: faa.gov
Title: Fly Safe Fact Sheets | Federal Aviation Administration
Link:https://www.faa.gov/newsroom/safety-briefing/faa-safety-briefing-fact-sheets
8.
Source: faa.gov
Title: Pilot Proficiency Training | Federal Aviation Administration
Link:https://www.faa.gov/newsroom/safety-briefing/pilot-proficiency-training
9.
Source: faa.gov
Title: Fly Safe: Prevent Loss of Control Accidents | Federal Aviation Administration
Link:https://www.faa.gov/newsroom/fly-safe-prevent-[loss-control
10.
Source: doi.org
Title: From Here to Autonomy
Link:https://doi.org/10.1177%2F0018720816681350
11.
Source: doi.org
Link:https://doi.org/10.1002/PRS.680160304
12.
Source: faa.gov
Title: Safety Briefing | Federal Aviation Administration SAFETY BRIEFING Left Nav
Link:https://www.faa.gov/taxonomy/term/301?page=17
13.
Source: doi.org
Title: Humans and Automation: Use, Misuse, Disuse, Abuse
Link:https://doi.org/10.1518/001872097778543886
14.
Source: ouci.dntb.gov.ua
Link:https://ouci.dntb.gov.ua/en/works/4OzxyWb7/
15.
Source: journals.sagepub.com
Link:https://journals.sagepub.com/doi/abs/10.1518/001872095779064555
16.
Source: journals.sagepub.com
Link:https://journals.sagepub.com/doi/pdf/10.1518/001872095779064555
17.
Source: sciencedirect.com
Title: Ironies of automation
Link:https://www.sciencedirect.com/science/article/pii/0005109883900468
18.
Source: sciencedirect.com
Title: Ironies of automation
Link:https://www.sciencedirect.com/science/article/pii/0005109883900468/pdf?_valck=1&md5=63dee2ce59531b69680a4af83c53cd3d&pid=1-s2.0-0005109883900468-main.pdf
19.
Source: andrewclark.co.uk
Title: ironies of automation
Link:https://andrewclark.co.uk/all-media/ironies-of-automation
Additional References
20.
Source: legalclarity.org
Title: The Automation Paradox: Legal Liability and System Risk
Link:https://legalclarity.org/the-automation-paradox-legal-liability-and-system-risk/
Source snippet
June 15, 2026 — THE AUTOMATION PARADOX: LEGAL LIABILITY AND SYSTEM RISK The more we rely on automated systems, the harder it...
Published: June 15, 2026
21.
Source: faasafety.gov
Title: Activities, Courses, Seminars & Webinars
Link:https://www.faasafety.gov/SPANS/event_details.aspx?caller=%2FSPANS%2Fevents%2FEventList.aspx&eid=133570
Source snippet
Event Details and Registration - FAA - FAASTeam - FAASafety.govDecember 5, 2024 — Event Details and Registration...
Published: December 5, 2024
22.
Source: faasafety.gov
Link:https://www.faasafety.gov/SPANS/noticeView.aspx?nid=14400
Source snippet
Notices - FAA - FAASTeam - FAASafety.govMay 7, 2025 — Notices Image: FAASTeam FAASTeam Notice Type: FAA Newsletters Notice Date: Wednesda...
Published: May 7, 2025
23.
Source: youtube.com
Title: Air France 447: The 4-Minute Mystery That Killed 228
Link:https://www.youtube.com/watch?v=lw7t5x27LAY
Source snippet
SREcon19 Asia/Pacific - Ironies of Automation: A Comedy in Three Parts...
24.
Source: faasafety.gov
Title: Event Details and Registration
Link:https://www.faasafety.gov/spans/event_details.aspx?eid=133666&pf=1
Source snippet
You may register by clicking the "Register" link...
25.
Source: researchgate.net
Link:https://www.researchgate.net/publication/238726310_The_Out-of-the-Loop_Performance_Problem_and_Level_of_Control_in_Automation
26.
Source: tandfonline.com
Link:https://www.tandfonline.com/doi/abs/10.1207/s15327108ijap0301_1
27.
Source: youtube.com
Title: EP7: Human Decision Making in the AI Era
Link:https://www.youtube.com/watch?v=NovRPsEOHyw
Source snippet
Air France 447: The 4-Minute Mystery That Killed 228...
28.
Source: frontiersin.org
Link:https://www.frontiersin.org/journals/neuroscience/articles/10.3389/fnins.2017.00541/full
29.
Source: sciencedirect.com
Title: Out-of-the-loop (OOL) Performance Problem: Characterization and Compensation
Link:https://www.sciencedirect.com/science/article/pii/B9780128119266000750