Within Fast takeoff
Why obvious AI disruption may come too late
Stock-market shocks, job losses, and political disruption may be dramatic, but they could appear only after the key capability jump has happened.
On this page
- Why public signals lag technical capability
- Normal progress versus recursive acceleration
- What forecasters could miss before a hard takeoff
Page outline Jump by section
Introduction
One of the central claims in hard-takeoff or FOOM (fast intelligence explosion) arguments is that the most obvious signs of disruption may not be the most useful warnings. Stock-market turmoil, mass unemployment, political instability, or dramatic social change would certainly attract public attention. The concern is that these events might occur only after the crucial transition has already happened: the point at which AI systems become capable enough to accelerate their own development, evade effective oversight, or gain decisive strategic advantages. [Machine Intelligence Research Institute]intelligence.orgMachine Intelligence Research InstituteThe Hanson-Yudkowsky AI-Foom Debate eBookNovember 6, 2024 — In late 2008, economist Robin Hanson a…
This creates an uncomfortable possibility for AI doom scenarios. A world waiting for visible social disruption before treating advanced AI as an existential risk could be looking at lagging indicators rather than leading ones. By the time ordinary institutions recognise that something extraordinary is happening, the underlying capability jump may already be well underway or complete. [Machine Intelligence Research Institute]intelligence.orgMachine Intelligence Research InstituteThe Hanson-Yudkowsky AI-Foom Debate eBookNovember 6, 2024 — In late 2008, economist Robin Hanson a…
Why public signals lag technical capability
Most social and economic indicators measure effects rather than causes. Unemployment rises after firms change their hiring behaviour. Political systems react after economic incentives shift. Financial markets often respond only once participants recognise that a technological change is commercially important.
Hard-takeoff advocates argue that a decisive AI capability transition could occur upstream of all these visible effects. The first systems capable of accelerating AI research might exist inside a small number of laboratories, data centres, government programmes, or corporate environments long before the wider public experiences major disruption. [AI Alignment Forum]alignmentforum.orgtakeoff speeds have a huge effect on what it means to work 1Whether AI is a…
In this view, the sequence is not:
- Society experiences chaos.
- Researchers realise AI has become dangerous.
Instead, the sequence could be:
- AI becomes extremely effective at research, engineering, planning, or persuasion.
- Organisations using it gain large advantages.
- Capabilities continue to compound.
Amazon book picks
Further Reading
Books and field guides related to Why obvious AI disruption may come too late. Use these as the next step if you want deeper reading beyond the article.
Human Compatible
Addresses the challenges of responding to increasingly capable AI systems.
Superintelligence
Examines why visible societal effects may lag underlying capability advances.
The Coming Wave
Directly addresses how society may react too slowly to rapidly advancing technologies.
Life 3.0
Explores scenarios where transformative AI arrives before institutions adapt.
eBay marketplace picks
Marketplace Samples
Example marketplace items related to this page. Use the search link to explore similar finds on eBay.
- Only later do economic and political consequences become obvious. [Machine Intelligence Research Institute]intelligence.orgMachine Intelligence Research InstituteThe Hanson-Yudkowsky AI-Foom Debate eBookNovember 6, 2024 — In late 2008, economist Robin Hanson a…
A useful comparison is financial crises. Many crises appear sudden to the public, but the underlying vulnerabilities often accumulated for years. By the time markets visibly collapse, the decisive events have already occurred. FOOM proponents argue that advanced AI could create an even larger gap between underlying reality and public perception. [Machine Intelligence Research Institute]intelligence.orgMachine Intelligence Research InstituteThe Hanson-Yudkowsky AI-Foom Debate eBookNovember 6, 2024 — In late 2008, economist Robin Hanson a…
Normal progress versus recursive acceleration
A major source of disagreement in AI-risk debates concerns whether future AI development will resemble previous technological revolutions.
In a gradual-transition model, society receives repeated warnings. AI systems steadily automate more jobs, businesses adapt over time, regulators respond, and the public becomes familiar with increasingly capable systems. By the time transformative AI arrives, institutions have already had years to learn from earlier deployments. [AI Alignment Forum]alignmentforum.orgtakeoff speeds have a huge effect on what it means to work 1Whether AI is a…
Hard-takeoff arguments reject the assumption that future capability growth must resemble past technological diffusion. They focus on a specific mechanism: AI systems helping to build better AI systems.
If AI becomes a major contributor to AI research itself, then capability improvements could arrive faster than social adaptation mechanisms. Researchers sometimes describe this as recursive self-improvement, though modern discussions often focus less on dramatic self-rewriting and more on AI accelerating the entire research and development cycle. [Machine Intelligence Research Institute]intelligence.orgMachine Intelligence Research InstituteThe Hanson-Yudkowsky AI-Foom Debate eBookNovember 6, 2024 — In late 2008, economist Robin Hanson a…
Under this model, the critical period is not when AI replaces large numbers of workers. The critical period is when AI starts substantially shortening the time required for major advances in algorithms, training methods, engineering, or scientific discovery. Economic disruption may appear only after those capability gains have already accumulated. [Machine Intelligence Research Institute]intelligence.orgMachine Intelligence Research InstituteThe Hanson-Yudkowsky AI-Foom Debate eBookNovember 6, 2024 — In late 2008, economist Robin Hanson a…
This is why some FOOM discussions place more weight on indicators such as AI-assisted AI research, autonomous scientific work, rapid capability jumps between model generations, and shrinking intervals between breakthroughs than on conventional economic statistics. [AI Alignment Forum]alignmentforum.orgtakeoff speeds have a huge effect on what it means to work 1Whether AI is a…
What forecasters could miss before a hard takeoff
A common assumption in public discussion is that transformative AI would announce itself through spectacular social effects. Hard-takeoff advocates argue that several factors could make this assumption unreliable.
Capability is easier to hide than social disruption
Advanced capabilities can emerge inside relatively small technical communities. A model that dramatically improves cybersecurity, scientific research, software engineering, or strategic planning may initially be used by only a handful of organisations.
The wider public might see only incremental product improvements while the most important capabilities remain confined to specialised settings. By the time those capabilities diffuse into the broader economy, they may already have advanced significantly further. [AI Alignment Forum]alignmentforum.orgtakeoff speeds have a huge effect on what it means to work 1Whether AI is a…
Physical reality can lag behind digital capability
Even if an AI system becomes dramatically more capable, factories, supply chains, infrastructure projects, and labour markets still take time to change.
This means the physical world can appear relatively normal while digital systems experience much faster improvement. A capability explosion in software, research, planning, or cyber operations could precede visible economic transformation by months or years. Critics of hard-takeoff scenarios often emphasise these physical bottlenecks, but FOOM advocates respond that existential risk may depend more on loss of control than on immediate physical transformation. [Shtetl-Optimized]scottaaronson.blogShtetl-Optimized Reform AI AlignmentShtetl-OptimizedReform AI AlignmentNovember 20, 2022 — 20 Nov 2022 — We Reform AI-riskers believe that, here just like in high school, th… [Marginal REVOLUTION]marginalrevolution.comome a larger share of the economy over time.Read more…
Institutions may misinterpret early signals
Human organisations are accustomed to gradual technological change. If capability gains arrive faster than expected, decision-makers may repeatedly classify warning signs as temporary anomalies.
A sudden jump in research productivity might be attributed to better tools. Rapid scientific breakthroughs might be treated as ordinary progress. Economic advantages gained by AI-heavy organisations might initially appear as normal competitive success. In a fast-takeoff scenario, these interpretations could delay recognition of what is actually happening. [Forethought]forethought.orgForethoughtPreparing for the Intelligence Explosion - Forethoughtby W MacAskill · Cited by 22 — The challenge of harnessing AI's ability…
Why this matters for AI doom arguments
The claim that societal disruption may be a late alarm is not merely a forecasting observation. It changes how AI doom arguments evaluate risk.
If major public disruption occurs before existential danger emerges, then societies retain opportunities to react. Governments can regulate, companies can slow deployment, researchers can improve safety methods, and international coordination remains possible.
If, however, the most visible disruptions arrive after the decisive capability transition, then waiting for obvious evidence becomes much less attractive as a risk-management strategy. In that world, warning systems need to focus on technical indicators closer to the source of capability growth rather than on downstream social consequences. [Machine Intelligence Research Institute]intelligence.orgMachine Intelligence Research InstituteThe Hanson-Yudkowsky AI-Foom Debate eBookNovember 6, 2024 — In late 2008, economist Robin Hanson a… [AI Alignment]WikipediaAI alignment
This logic helps explain why many AI safety researchers place unusual emphasis on evaluations, interpretability research, monitoring frontier systems, tracking AI-assisted AI development, and understanding emerging autonomous capabilities. The goal is to detect important changes before they appear in unemployment statistics, election outcomes, GDP figures, or financial markets. [Forethought]forethought.orgForethoughtPreparing for the Intelligence Explosion - Forethoughtby W MacAskill · Cited by 22 — The challenge of harnessing AI's ability…
The strongest objection: maybe disruption comes first
The most important criticism of the late-alarm view is that transformative technologies have historically produced visible economic effects before creating existential dangers.
Sceptics argue that a genuinely transformative AI system would need extensive deployment, infrastructure, capital investment, and organisational integration. These processes would likely generate years of observable disruption, providing ample warning. They point to adoption bottlenecks, regulatory friction, physical constraints, and the slow pace of institutional change as reasons to expect a more gradual transition. [Marginal REVOLUTION]marginalrevolution.comome a larger share of the economy over time.Read more…
From this perspective, claims that society could look normal until immediately before a decisive AI transition underestimate how difficult it is to convert intelligence into real-world power.
The disagreement remains unresolved because it depends on uncertain empirical questions: how much AI can accelerate AI research, how quickly capabilities compound, how important physical bottlenecks become, and whether strategic advantages can accumulate faster than institutions can respond. [Machine Intelligence Research Institute]intelligence.orgMachine Intelligence Research InstituteThe Hanson-Yudkowsky AI-Foom Debate eBookNovember 6, 2024 — In late 2008, economist Robin Hanson a… [AI Alignment]WikipediaAI alignment
The practical takeaway
For people concerned about AI doom, the key lesson is that dramatic social disruption is not necessarily an early warning signal. Mass unemployment, market upheaval, or political instability could be highly visible indicators, yet still arrive after the most important capability transition has already occurred.
The late-alarm argument does not prove that a FOOM will happen. Nor does it show that a hard takeoff is more likely than a gradual transition. What it does suggest is that forecasting based solely on public economic and social signals may be insufficient. If a fast takeoff is possible, the most informative warnings may appear first in technical capability trends, AI-assisted research progress, and the shrinking gap between one frontier generation and the next, rather than in the headlines that eventually dominate public debate. AI Alignment Forum [Machine Intelligence Research Institute]intelligence.orgMachine Intelligence Research InstituteThe Hanson-Yudkowsky AI-Foom Debate eBookNovember 6, 2024 — In late 2008, economist Robin Hanson a…
Endnotes
-
Source: intelligence.org
Link: https://intelligence.org/ai-foom-debate/Source snippet
Machine Intelligence Research InstituteThe Hanson-Yudkowsky AI-Foom Debate eBookNovember 6, 2024 — In late 2008, economist Robin Hanson a...
Published: November 6, 2024
-
Source: marginalrevolution.com
Link: https://marginalrevolution.com/marginalrevolution/2025/02/why-i-think-ai-take-off-is-relatively-slow.htmlSource snippet
ome a larger share of the economy over time.Read more...
-
Source: forethought.org
Link: https://www.forethought.org/research/preparing-for-the-intelligence-explosionSource snippet
ForethoughtPreparing for the Intelligence Explosion - Forethoughtby W MacAskill · Cited by 22 — The challenge of harnessing AI's ability...
-
Source: alignmentforum.org
Title: takeoff speeds have a huge effect on what it means to work 1
Link: https://www.alignmentforum.org/posts/hRohhttbtpY3SHmmD/takeoff-speeds-have-a-huge-effect-on-what-it-means-to-work-1Source snippet
Whether AI is a...
-
Source: alignmentforum.org
Link: https://www.alignmentforum.org/posts/zkF9PNSyDKusoyLkP/investigating-ai-takeover-scenariosSource snippet
AI Alignment ForumInvestigating AI Takeover Scenarios17 Sept 2021 — Here, we discuss variable social, economic and technological characte...
-
Source: scottaaronson.blog
Title: Shtetl-Optimized Reform AI Alignment
Link: https://scottaaronson.blog/?p=6821Source snippet
Shtetl-OptimizedReform AI AlignmentNovember 20, 2022 — 20 Nov 2022 — We Reform AI-riskers believe that, here just like in high school, th...
Published: November 20, 2022
-
Source: Wikipedia
Title: AI alignment
Link: https://en.wikipedia.org/wiki/AI_alignment
Additional References
-
Source: researchgate.net
Link: https://www.researchgate.net/publication/388459864_Gradual_Disempowerment_Systemic_Existential_Risks_from_Incremental_AI_DevelopmentSource snippet
Systemic Existential Risks from Incremental AI Development10 Jan 2025 — This paper examines the systemic risks posed by incremental advan...
-
Source: davidmanheim.medium.com
Link: https://davidmanheim.medium.com/a-tentative-typology-of-ai-foom-scenarios-54ff20c906c3Source snippet
Tentative Typology of AI-Foom Scenarios | by David ManheimA foom-like explosion can quickly make a once-small system more powerful than t...
-
Source: facebook.com
Link: https://www.facebook.com/groups/aisafetyopen/posts/1159381928098165/Source snippet
Advanced AI alignment steps to reduce dangerIn it, I break down the basic case into 5 points: 1) Advanced AI is possible 2) Advanced AI m...
-
Source: ea-crux-project.vercel.app
Link: https://ea-crux-project.vercel.app/ai-transition-model/gradual/Source snippet
Gradual AI TakeoverCausal factors driving gradual loss of human control. Based on Christiano's two-part failure model: proxy optimization...
-
Source: reddit.com
Link: https://www.reddit.com/r/singularity/comments/1rwoy0/intelligence_explosion_aigofoom/Source snippet
Intelligence Explosion: "AI-go-FOOM": r/singularityI found this text in 'Intelligence Explosion Microeconomics', written by Eliezer Yudk...
-
Source: facebook.com
Link: https://www.facebook.com/cnn/posts/tech-ceo-and-co-founder-of-othersideai-matt-shumer-said-in-a-now-viral-article-p/1284670926858889/Source snippet
Tech CEO and co-founder of OthersideAI Matt Shumer said...As Trump signs executive order on [artificial]({{ 'artificial-goals/' | relative_url }}) intelligence, tech giants warn of...
-
Source: podimo.com
Link: https://podimo.com/es/shows/the-nonlinear-library-alignment-forum-top-posts/episode/6e332718-a938-4c85-91d8-e5fe0e933b09Source snippet
What failure looks like by Paul ChristianoThe stereotyped image of AI catastrophe is a powerful, malicious AI system that takes its creat...
-
Source: blog.biocomm.ai
Title: foom warning ai intelligence explosion nearly here in our environment
Link: https://blog.biocomm.ai/2024/11/23/foom-warning-ai-intelligence-explosion-nearly-here-in-our-environment/Source snippet
biocomm.aiFOOM! WARNING. AI INTELLIGENCE EXPLOSION...24 Nov 2024 — “There are no hard problems, only problems that are hard to a certain...
-
Source: responsible.com
Link: https://responsible.com/web-news/what-failure-looks-like-ai-alignment-forum/Source snippet
The stereotyped image of AI catastrophe is a powerful, malicious AI...Read more...
-
Source: x.com
Link: https://x.com/DavidDuvenaud/status/1885009790436352122Source snippet
g, not with sudden violent takeover, but through a gradual loss of...Read more...
Topic Tree