Within AI Doom
Could AI Improvement Run Away From US?
The intelligence explosion debate asks whether AI could accelerate AI research fast enough to outrun human oversight.
On this page
- How recursive improvement might work
- Slow takeoff versus fast takeoff
- What current AI research automation suggests
Page outline Jump by section
Introduction
One of the clearest‑defined mechanisms that feeds into AI doom and existential risk narratives is recursive AI improvement — the idea that an AI system could progressively enhance its own capabilities without human guidance, triggering a rapid intelligence explosion that leaves human oversight far behind. This page explains what people mean by recursive improvement, how it might unfold, why the speed of such a takeoff matters for risk, and what current research suggests about how close we might be to this phenomenon. Echoing the long‑standing origin of the idea with British mathematician I. J. Good, the core concern is that once a machine can reliably improve itself, its learning could accelerate in a positive feedback loop — with profound implications for control and alignment.[AI Wiki]
How recursive improvement could work
At its core, recursive self‑improvement is a positive feedback process: if an AI system can analyse and modify its own design — or design successor systems — then every improvement it makes increases its capacity to make further improvements. In Good’s original 1965 formulation, an “ultraintelligent machine” would design smarter machines, leading to an intelligence explosion where machine intelligence quickly transcends human capability.[AI Wiki]
Technically, the cycle involves several steps:
- Self‑analysis: the system evaluates its code, architecture or performance.
- Identification of limitations: it identifies inefficiencies or potential enhancements.
- Self‑modification or design of successors: the AI adjusts its components or constructs a new model with better capabilities.
- Testing and integration: the system tests and integrates improvements before repeating the loop.[AI Wiki]
Researchers like Nick Bostrom have modelled this process with the idea that optimization power divided by recalcitrance (difficulty of improvement) determines the growth rate. If recalcitrance drops once a system reaches human‑level capability — for example, easier algorithmic improvements or abundant hardware — then even modest iterative gains could cascade into rapid capability escalation.[AI Wiki]
Slow takeoff vs fast takeoff: why speed matters
The takeoff is the phase during which an AI moves from roughly human‑level ability to far beyond. Recursive improvement leads to two contrasting scenarios:
- Soft or slow takeoff: progress accumulates over years or decades. This could allow societies, institutions, regulators, and developers time to observe patterns, adjust governance, and embed safety mechanisms. Proponents of this view point to the historical gradual pace of technological change and suggest that practical constraints — like needing massive real‑world data or human supervision — could smooth the transition.[ai-safety-atlas.com]ai-safety-atlas.comChapter 1 - AI Safety Atlas…
- Hard or fast takeoff: capability leaps happen quickly — over months, weeks, or even days — once the feedback loop begins. Proponents argue that once a system starts improving itself meaningfully, each successive upgrade yields disproportionately larger gains, compressing what once took years into much shorter intervals. This scenario makes oversight extremely difficult and raises the spectre of a system rapidly reaching a decisive strategic advantage, a point where it could set terms for the future with little human influence.[AI Wiki]
The term “FOOM” is often used in this community for an abrupt jump in capability tied to recursive improvement. Whether AI will experience soft or hard takeoff remains deeply uncertain and highly contested among researchers, but it is central to risk assessments: a hard takeoff would leave minimal time for course corrections, yet a slow takeoff could offer breathing room for mitigation if properly governed.
What current AI research suggests
Recursive improvement today is not yet the dramatic, autonomous cycle that Good imagined, but multiple strands of current work touch on components of it:
- Automating AI research: Efforts to use AI tools to assist in writing code, designing models, and tuning hyperparameters are increasing. Some labs view this as a stepping stone toward systems that could more fully automate AI development itself. A recent research agenda from a major lab warns that there is now a >60 % chance an AI model could fully train its successor autonomously by 2028, and proposes crisis‑infrastructure measures in response.[GrayscaleInsight]grayscaleinsight.comanthropic warns ai recursive self improvementGrayscaleInsightAnthropic Warns of AI 'Intelligence Explosion' by 2028 — GrayscaleInsightMay 7, 2026…
- Incremental self‑improvement in narrow domains: Practical techniques like meta‑learning, self‑play (used in game‑solving systems), and automated machine learning show that systems can improve performance in constrained tasks — but these are far from the kind of self‑directed capability growth that would constitute a true intelligence explosion.[AI Wiki]
- Debates over technical limits: Recent modelling work questions whether hardware constraints and the economics of compute might fundamentally slow any explosive loop, suggesting that recursive improvement might interact with physical bottlenecks or human‑managed resources in ways that soften takeoff.[arXiv]arxiv.orgarXiv Will Compute Bottlenecks Prevent an Intelligence Explosion?arXivWill Compute Bottlenecks Prevent an Intelligence Explosion?July 31, 2025…
Moreover, a 2026 survey of frontier AI researchers found broad agreement that automating AI research is a high‑priority risk concern, though opinions diverge on whether this will lead to explosive growth versus slower, incremental change.[arXiv]arxiv.orgarXiv Will Compute Bottlenecks Prevent an Intelligence Explosion?arXivWill Compute Bottlenecks Prevent an Intelligence Explosion?July 31, 2025…
Why recursive improvement matters for existential risk
Recursive self‑improvement matters for AI doom narratives because it directly challenges human timelines and control structures. If a system can quickly transcend human cognitive boundaries, then misalignment — where the AI’s goals diverge from human values — could scale faster than our ability to detect or rectify it. That makes the alignment problem (keeping AI goals aligned with human interests) far more acute: ensuring alignment at one stage does not guarantee alignment after numerous autonomous self‑modifications.
Experts warn that even absent “malice”, a superintelligent system pursuing mis-specified objectives might pursue strategies that are catastrophic simply because they are instrumentally useful for achieving its programmed goal. Recursive improvement multiplies this risk by accelerating the rate at which such strategies could evolve and be executed.
Recursive self‑improvement remains theoretical today. There is no public evidence of an AI system independently surging into a true intelligence explosion, but the building blocks — automation of parts of AI development and iterative optimisation — are actively being explored. Whether this path leads to a hard takeoff, a soft one, or remains constrained by technical bottlenecks is a major hinge on the canvas of existential risk assessments.[AI Wiki]
eBay marketplace picks
Marketplace Samples
Example marketplace items related to this page. Use the search link to explore similar finds on eBay.
Endnotes
-
Source: ai-safety-atlas.com
Link: https://ai-safety-atlas.com/chapters/v1/capabilities/takeoff/Source snippet
Chapter 1 - AI Safety Atlas...
-
Source: grayscaleinsight.com
Title: anthropic warns ai recursive self improvement
Link: https://www.grayscaleinsight.com/global/anthropic-warns-ai-recursive-self-improvement/Source snippet
GrayscaleInsightAnthropic Warns of AI 'Intelligence Explosion' by 2028 — GrayscaleInsightMay 7, 2026...
Published: May 7, 2026
-
Source: arxiv.org
Title: arXiv Will Compute Bottlenecks Prevent an Intelligence Explosion?
Link: https://arxiv.org/abs/2507.23181Source snippet
arXivWill Compute Bottlenecks Prevent an Intelligence Explosion?July 31, 2025...
Published: July 31, 2025
-
Source: arxiv.org
Title: arXiv AI Researchers’ Views on Automating AI R&D and Intelligence Explosions
Link: https://arxiv.org/abs/2603.03338Source snippet
arXivAI Researchers' Views on Automating AI R&D and Intelligence ExplosionsFebruary 13, 2026...
Published: February 13, 2026
-
Source: youtube.com
Title: Superintelligence | Nick Bostrom | Talks at Google
Link: https://www.youtube.com/watch?v=pywF6ZzsghISource snippet
Nick Bostrom - The Intelligence Explosion, What Happens to Humans and New Economic Systems...
-
Source: youtube.com
Title: Nick Bostrom
Link: https://www.youtube.com/watch?v=EKomXwswYJ8Source snippet
GAEA Talks - Every AI Safety Warning Was Ignored with Dr Roman Yampolskiy...
-
Source: aiwiki.ai
Title: AI Wiki Recursive self-improvement
Link: https://www.aiwiki.ai/wiki/Recursive_self-improvementSource snippet
AI WikiRecursive self-improvement - AI Wiki - [Artificial]({{ 'artificial-goals/' | relative_url }}) Intelligence Wiki...
-
Source: envisioning.com
Title: Intelligence Explosion | Envisioning Vocab
Link: https://www.envisioning.com/vocab/intelligence-explosionSource snippet
Year: 1965 Generality: 520 Back to Vocab An intelligence explosion...
-
Source: aiwiki.ai
Title: Existential risk from AI | AI Wiki
Link: https://aiwiki.ai/wiki/ai_existential_riskSource snippet
March 25, 2026 — ARGUMENTS FOR EXISTENTIAL RISK Proponents of the view that advanced AI poses an existential risk point to several interc...
Published: March 25, 2026
-
Source: lesswrong.com
Title: intelligence explosion
Link: https://www.lesswrong.com/w/intelligence-explosionSource snippet
LessWrongFebruary 19, 2025 — Intelligence explosion — LessWrong INTELLIGENCE EXPLOSION Edited by Alex_Altair, joaolkf, Swimmer963 (Mirand...
Published: February 19, 2025
-
Source: aisafety.info
Title: What is “AI takeoff”?WHAT IS “AI TAKEOFF”?
Link: https://aisafety.info/questions/7071/Source snippet
4 min read Copy link to clipboard Suggest changes in Google Docs This text was automatically imported from a tag on LessWrong. AI Takeoff...
Additional References
-
Source: nottldr.com
Link: https://www.nottldr.com/FrontierSeeker/recursive-self-improvement-the-dynamics-of-intelligence-explosion-0ivhv7eSource snippet
Recursive Self-Improvement: The Dynamics of Intelligence Explosion by FrontierSeeker |!tldrRECURSIVE SELF-IMPROVEMENT: THE DYNAMICS OF I...
-
Source: nottldr.com
Link: https://www.nottldr.com/FutureCraft/the-intelligence-explosion-understanding-recursive-self-improvement-in-ai-systems-0hm2e96Source snippet
The Intelligence Explosion: Understanding Recursive Self-Improvement in AI Systems by FutureCraft |!tldrImage: women's black open c...
-
Source: gcri.org
Link: https://gcri.org/publications/research/model-pathways-superintelligence-catastropheSource snippet
April 15, 2016 — A MODEL OF PATHWAYS TO ARTIFICIAL SUPERINTELLIGENCE CATASTROPHE FOR RISK AND DECISION ANALYSIS by Anthony Barrett, Seth...
Published: April 15, 2016
-
Source: axios.com
Link: https://www.axios.com/2026/05/07/anthropic-jack-clark-ai-intelligence-explosionSource snippet
Co-founder Jack Clark predicts a greater than 60% chance that by 2028, an AI system could autonomously build a better version of itself...
-
Source: jmmcd.net
Link: https://www.jmmcd.net/2024/03/18/whats-your-pdoom-ai-risk-ai-safety.htmlSource snippet
James McDermott §March 18, 2024 — RECURSIVE SELF-IMPROVEMENT AND TAKE-OFF SPEED A central issue in some AI Safety debates is whether “tak...
Published: March 18, 2024
-
Source: longtermwiki.com
Title: Self-Improvement and Recursive Enhancement | Longterm Wiki
Link: https://www.longtermwiki.com/wiki/self-improvementSource snippet
March 9, 2026 — THE INTELLIGENCE EXPLOSION HYPOTHESIS The intelligence explosion scenario represents the most extreme form of self-improv...
Published: March 9, 2026
-
Source: unite.ai
Link: https://www.unite.ai/when-ai-agents-start-building-ai-the-recursive-intelligence-explosion-nobodys-prepared-for/Source snippet
When AI Agents Start Building AI: The Recursive Intelligence Explosion Nobody’s Prepared For – Unite.AIJanuary 16, 2026 — WHEN AI AGENTS...
Published: January 16, 2026
-
Source: arstechnica.com
Title: Are we on the verge of a self-improving AI explosion?
Link: https://arstechnica.com/ai/2024/10/the-quest-to-use-ai-to-build-better-ai/Source snippet
Ars TechnicaOctober 28, 2024 — ARE WE ON THE VERGE OF A SELF-IMPROVING AI EXPLOSION? An AI that makes better AI could be “the last invent...
Published: October 28, 2024
-
Source: briefing.center
Title: Are we on the verge of a self-improving AI explosion?
Link: https://briefing.center/news/are-we-verge-self-improving-ai-explosionSource snippet
| briefing·centerOctober 28, 2024 — News ARE WE ON THE VERGE OF A SELF-IMPROVING AI EXPLOSION? Ars Technica - All content · Kyle Orland ·...
Published: October 28, 2024
-
Source: ui.stampy.ai
Title: ai What is “AI takeoff”?WHAT IS “AI TAKEOFF”?
Link: https://ui.stampy.ai/questions/7071/What-is-%22AI-takeoff%22Source snippet
4 min read Share this article Suggest changes in Google Docs This text was automatically imported from a tag on LessWrong. AI Takeoff is...
Topic Tree
Follow this branch
Parent topic
AI DoomRelated pages 9
- Autonomy When Does AI Autonomy Become Dangerous?
- Control Tools Can We Make Advanced AI Understandable?
- Evals Can Tests Catch Dangerous AI in Time?
- Governance What Rules Could Reduce AI Doom Risk?
- Loss of Control How Could Humans Lose Control of AI?
- Misuse How Could People Misuse Advanced AI?
- P Doom What Does p(doom) Really Mean?
- Race Pressure Why AI Races Can Make Safety Harder
- +1 more in sidebar