Within Catastrophic Misuse
When Cyberattacks Become an Extinction Risk
Cyber operations are unlikely to cause extinction alone, but they could steal dangerous research, disable defences or amplify another attack.
On this page
- How AI could scale and automate intrusion
- Why cyber risk becomes existential through other threats
- Whether AI gives attackers or defenders the advantage
Page outline Jump by section
Introduction
AI-enabled cyberattacks are not generally viewed as a direct path to human extinction. Even the most sophisticated cyber campaigns have historically caused disruption, financial losses and, in some cases, physical damage, rather than civilisation-ending consequences. Within debates about AI doom and existential risk, cyber capabilities become important for a different reason: they can act as force multipliers for other catastrophic threats.
The central concern is that increasingly capable AI systems could make cyber operations faster, cheaper and more scalable, allowing attackers to steal dangerous research, disable critical defences, compromise military or public-health systems, or coordinate complex campaigns that would previously have required large teams of highly skilled specialists. On this view, cyberattacks are less a standalone extinction mechanism than an enabling technology that could make biological, military or infrastructure-based catastrophes more feasible. At the same time, many cybersecurity experts argue that AI may strengthen defence as much as offence, making the long-term balance highly uncertain.[aisi.gov.uk]aisi.gov.ukAIS I Research Agenda | The AI Security InstituteAIS I Research Agenda | The AI Security Institute
How AI could scale and automate intrusion
Modern cyber operations involve far more than writing malicious code. Attackers typically spend most of their time identifying targets, gathering information, searching for vulnerabilities, adapting tools, maintaining access and coordinating across multiple systems. These are exactly the kinds of knowledge-intensive tasks where large AI models have improved rapidly.
Rather than imagining AI inventing completely new forms of hacking, many researchers expect it to compress the time and expertise needed for existing attacks. AI systems can already assist with software analysis, vulnerability discovery, code generation and scripting, while increasingly capable agent-based systems can perform longer sequences of technical tasks with limited supervision. The UK’s AI Security Institute (AISI) reports that frontier models have improved rapidly across cyber evaluations, with the strongest systems progressing from rarely completing apprentice-level cybersecurity tasks in 2023 to succeeding on roughly half such tasks only a few years later. Some models have also begun demonstrating limited success on expert-level evaluation tasks.[aisi.gov.uk]aisi.gov.ukFrontier AI Trends Report by The AI Security Institute (AISIFrontier AI Trends Report by The AI Security Institute (AISI
From an existential-risk perspective, the most significant changes are not individual exploits but scale:
- AI could search for vulnerabilities continuously across enormous numbers of systems.
- Multiple intrusion campaigns could run simultaneously instead of requiring separate human teams.
- Attack planning, documentation, coding and adaptation could become increasingly automated.
- Smaller organisations or individuals could potentially perform operations previously requiring much larger groups.
This does not mean current models autonomously conduct sophisticated real-world campaigns. Evaluations continue to find substantial gaps between benchmark performance and successful operations in realistic environments, particularly when attacks require long-term planning, adapting to unexpected failures or navigating complex enterprise networks. Nevertheless, researchers increasingly evaluate AI in realistic cyber ranges because laboratory-style benchmarks may underestimate future capabilities.[arXiv]arxiv.orgAgentCyberRange: Benchmarking Frontier AI Systems in Realistic Cyber RangesJune 12, 2026…
Why cyber risk becomes existential through other threats
Cyberattacks rarely threaten humanity on their own. Instead, they become relevant to AI doom arguments when they support another catastrophic pathway.
Stealing dangerous scientific research
One frequently discussed concern is the theft of sensitive research from biotechnology laboratories, pharmaceutical companies or government facilities.
Cyber intrusion could expose pathogen research, laboratory methods, genomic databases or defensive countermeasure research that would otherwise remain difficult to obtain. AI assistance might reduce the expertise required to locate valuable information, navigate unfamiliar systems or analyse large quantities of stolen technical material.
Importantly, stolen information alone would not produce an existential biological threat. Creating a globally catastrophic pathogen would still require substantial laboratory capability, production, delivery and many additional steps. Cyber intrusion is therefore viewed as reducing one barrier within a much longer chain rather than eliminating the entire challenge.
Disabling defensive systems
Another concern is that cyber operations might weaken society’s ability to respond to another attack.
Examples discussed in policy literature include compromising:
- hospital networks during a biological emergency;
- public-health surveillance systems;
- emergency communications;
- logistics supporting vaccine or medicine distribution;
- industrial control systems involved in critical infrastructure.
The cyberattack itself would not necessarily cause catastrophic casualties. Instead, it could delay detection, slow response or reduce resilience during another crisis.
Supporting military escalation
Cyber operations have long been recognised as potential contributors to military crises.
Advanced AI could accelerate intelligence gathering, automate network penetration or increase the speed of cyber campaigns targeting military communications and decision-support systems. Some AI-risk researchers argue that this could increase the risk of accidental escalation during periods of geopolitical tension, particularly if attacks create uncertainty about an opponent’s intentions or capabilities.
However, moving from cyber disruption to nuclear war or human extinction involves many additional assumptions. Most analyses therefore present cyber operations as one contributor to escalation risk rather than an independent existential threat.[GOV.UK]GOV.UKFrontier AI: capabilities and risks – discussion paperFrontier AI: capabilities and risks – discussion paper
Why speed and scale matter more than novel attacks
Many discussions of AI cyber risk focus less on entirely new capabilities than on changing economics.
Historically, sophisticated cyber operations have been constrained by scarce expert labour. Skilled vulnerability researchers, malware developers and intrusion specialists are expensive and difficult to replace. AI could reduce those bottlenecks by automating routine technical work while allowing experienced operators to supervise many more simultaneous campaigns.
Several consequences follow:
- vulnerabilities may be discovered more quickly;
- the delay between discovering and exploiting security flaws may shrink;
- attacks can be customised for individual organisations instead of relying on generic tools;
- successful techniques may be replicated rapidly across thousands of targets.
From an existential-risk perspective, this matters because catastrophic misuse often requires many supporting operations rather than a single spectacular hack. Compressing months of technical work into days could make coordinated misuse easier, even if no individual cyber capability appears revolutionary. Researchers have therefore begun framing evaluations around entire attack chains rather than isolated technical tasks.[arxiv.org]arxiv.orgarXiv A Framework for Evaluating Emerging Cyberattack Capabilities of AIarXiv A Framework for Evaluating Emerging Cyberattack Capabilities of AI
Whether AI gives attackers or defenders the advantage
One of the largest unresolved questions is whether AI ultimately benefits attackers more than defenders.
Arguments for an offensive advantage emphasise that attackers need only find one successful path into a system, while defenders must secure everything. AI could automate vulnerability discovery, malware adaptation and reconnaissance across enormous numbers of potential targets.
Arguments for a defensive advantage point to equally powerful uses:
- automated code review;
- vulnerability detection before software release;
- faster incident response;
- improved malware analysis;
- continuous network monitoring;
- automated security patching;
- defensive AI agents working around the clock.
The UK’s National Cyber Security Centre argues that frontier AI is likely to strengthen both offensive and defensive cyber capabilities. Rather than changing the basic principles of cybersecurity, it increases the consequences of weak security practices. Organisations that already struggle with patching, authentication and system hardening may find those weaknesses exploited much more rapidly.[National Cyber Security Centre]ncsc.gov.ukOpen source on ncsc.gov.uk.
AISI similarly treats the offence–defence balance as an empirical research question rather than a settled conclusion. Its cyber programme explicitly studies whether defensive interventions can preserve or increase defender advantage as models become more capable.[aisi.gov.uk]aisi.gov.ukAIS I Research Agenda | The AI Security InstituteAIS I Research Agenda | The AI Security Institute
What evidence exists today?
Current evidence supports several relatively modest conclusions while leaving larger existential claims uncertain.
There is strong evidence that frontier AI models are becoming substantially better at cybersecurity-related tasks, including code understanding, vulnerability identification and other technical activities measured in controlled evaluations.[aisi.gov.uk]aisi.gov.ukFrontier AI Trends Report by The AI Security Institute (AISIFrontier AI Trends Report by The AI Security Institute (AISI
There is growing evidence that AI systems can automate increasingly long sequences of technical work in realistic cyber environments, although success rates remain far below those of expert human penetration testers on complex networks.[arXiv]arxiv.orgAgentCyberRange: Benchmarking Frontier AI Systems in Realistic Cyber RangesJune 12, 2026…
There is much weaker evidence that current AI can independently conduct sophisticated, end-to-end cyber campaigns against hardened real-world targets without extensive human supervision. Existing public evaluations continue to show significant limitations in reliability, long-term planning and adaptation to unexpected obstacles.[arXiv]arxiv.orgAgentCyberRange: Benchmarking Frontier AI Systems in Realistic Cyber RangesJune 12, 2026…
Finally, there is little evidence that cyber capabilities alone constitute an existential threat. Most serious AI-risk analyses instead treat them as enabling technologies whose importance depends on whether they can significantly strengthen more dangerous pathways such as biological misuse, military escalation or attacks on critical infrastructure.[GOV.UK]GOV.UKFrontier AI: capabilities and risks – discussion paperFrontier AI: capabilities and risks – discussion paper
What warning signs would change the risk assessment?
Researchers concerned with AI doom generally watch for developments that would indicate cyber capabilities are becoming strategically significant rather than merely commercially useful.
Potential warning signs include:
- AI systems consistently outperforming experienced human professionals across realistic cyber exercises.
- Reliable autonomous execution of complete multi-stage intrusion campaigns.
- Demonstrated ability to discover previously unknown vulnerabilities at large scale.
- Successful long-duration operations requiring little human intervention.
- Evidence that attackers can repeatedly overcome safety controls and misuse restrictions.
- Real-world incidents showing AI materially reducing the cost or expertise required for major cyber operations.
These indicators would not themselves imply an existential threat. Instead, they would suggest that cyber capabilities are becoming powerful enough to meaningfully amplify other catastrophic misuse scenarios.
How researchers propose reducing the risk
Because cyber risk functions mainly as an enabling mechanism, mitigation efforts focus on preventing AI from dramatically shifting the offence–defence balance.
Current proposals include more realistic cyber capability evaluations before deployment, external testing by security researchers, stronger safeguards around dangerous capabilities, staged model releases, monitoring for emerging cyber competence, improved protection of sensitive research infrastructure and closer coordination between AI developers and national cybersecurity agencies. Researchers also argue for developing defensive AI systems capable of matching or exceeding offensive automation, so that improvements in attack capability are accompanied by comparable improvements in detection and response.[aisi.gov.uk]aisi.gov.ukAIS I Research Agenda | The AI Security InstituteAIS I Research Agenda | The AI Security Institute
Within the broader discussion of catastrophic AI misuse, this reflects an important distinction. AI-enabled cyberattacks are not generally considered a likely route to extinction by themselves. Their significance lies in their potential to remove bottlenecks that currently limit more dangerous forms of misuse, making other catastrophic scenarios easier to organise, coordinate or conceal while simultaneously testing whether defensive institutions can adapt quickly enough to keep pace.
Amazon book picks
Further Reading
Books and field guides related to When Cyberattacks Become an Extinction Risk. Use these as the next step if you want deeper reading beyond the article.
This is how They Tell Me the World Ends
WINNER OF THE FT & McKINSEY BUSINESS BOOK OF THE YEAR AWARD 2021The instant New York Times bestsellerA Financial Times and The Times Book...
The Perfect Weapon
From Russia’s tampering with the US election to the WannaCry hack that temporarily crippled Britain’s NHS, cyber has become the weapon of...
Sandworm
"With the nuance of a reporter and the pace of a thriller writer, Andy Greenberg gives us a glimpse of the cyberwars of the future while...
The Fifth Domain
An urgent new warning from two bestselling security experts--and a gripping inside look at how governments, firms, and ordinary citizens...
eBay marketplace picks
Marketplace Samples
Live-tested eBay searches with available results related to this page.
Selected fromcybersecurity poster oneBay.co.uk.
Endnotes
1.
Source: aisi.gov.uk
Title: AIS I Research Agenda | The AI Security Institute
Link:https://www.aisi.gov.uk/research-agenda
2.
Source: ncsc.gov.uk
Link:https://www.ncsc.gov.uk/frontier-ai
3.
Source: GOV.UK
Title: Frontier AI: capabilities and risks – discussion paper
Link:https://www.gov.uk/government/publications/frontier-ai-capabilities-and-risks-discussion-paper/frontier-ai-capabilities-and-risks-discussion-paper
4.
Source: aisi.gov.uk
Title: Frontier AI Trends Report by The AI Security Institute (AISI)
Link:https://www.aisi.gov.uk/frontier-ai-trends-report
5.
Source: arxiv.org
Link:https://arxiv.org/abs/2606.14295
Source snippet
AgentCyberRange: Benchmarking Frontier AI Systems in Realistic Cyber RangesJune 12, 2026...
Published: June 12, 2026
6.
Source: arxiv.org
Title: arXiv A Framework for Evaluating Emerging Cyberattack Capabilities of AI
Link:https://arxiv.org/abs/2503.11917
7.
Source: ncsc.gov.uk
Link:https://www.ncsc.gov.uk/blogs/why-cyber-defenders-need-to-be-ready-for-frontier-ai
Source snippet
March 30, 2026 — WHY CYBER DEFENDERS NEED TO BE READY FOR FRONTIER AI Understanding the threats and staying ahead of the adversary Paul J...
Published: March 30, 2026
8.
Source: GOV.UK
Title: The risks posed by fut
Link:https://www.gov.uk/government/publications/frontier-ai-capabilities-and-risks-discussion-paper/future-risks-of-frontier-ai-annex-a
Source snippet
risks of frontier AI (Annex A) - GOV.UKApril 28, 2025 — This paper is focussed on risks from Frontier AI (artificial intelligence), but e...
Published: April 28, 2025
9.
Source: aisi.gov.uk
Link:https://www.aisi.gov.uk/blog/5-key-findings-from-our-first-frontier-ai-trends-report
Source snippet
Frontier AI Trends Report draws on 2 years' worth of evaluations to provide accessible insights into the trajectory of AI develo...
Additional References
10.
Source: deepmind.google
Link:https://deepmind.google/blog/evaluating-potential-cybersecurity-threats-of-advanced-ai/
Source snippet
Building secure AGI: Evaluating emerging cyber security capabilities of advanced AI — Google DeepMindApril 2, 2025 — April 2, 2025 Respon...
Published: April 2, 2025
11.
Source: irregular.com
Title: Frontier Cyber: Bringing Offensive Cyber Evaluations to Real Systems
Link:https://www.irregular.com/research/frontiercyber
Source snippet
FrontierCyber: Bringing Offensive Cyber Evaluations to Real Systems - IrregularJune 22, 2026 — FRONTIERCYBER: BRINGING OFFENSIVE CYBER EV...
Published: June 22, 2026
12.
Source: frontiersecurityinstitute.org
Link:https://www.frontiersecurityinstitute.org/
Source snippet
FRONTIER AI LABS AND THE NATIONAL SECURITY ENTERPRISE — DELIVERING THE EVALUATIONS, BRIEFINGS...
13.
Source: rdi.berkeley.edu
Title: frontier ai impact on cybersecurity
Link:https://rdi.berkeley.edu/frontier-ai-impact-on-cybersecurity/
Source snippet
AI's Impact on the Cybersecurity LandscapeFRONTIER AI'S IMPACT ON THE CYBERSECURITY LANDSCAPE: CURRENT STATUS AND FUTURE DIRECTIONS Yujin...
14.
Source: youtube.com
Link:https://www.youtube.com/watch?v=9f2vG30HUvs
Source snippet
Agentic AI in Cybersecurity: An Analytical Briefing...
15.
Source: youtube.com
Title: Did We Not Notice AGI Already Happened? | Joshua Achiam
Link:https://www.youtube.com/watch?v=LCrRmz8dA-E
Source snippet
Anthropic's Mythos, the UK AI Security Institute, and the Dawn of Autonomous Cyber Infiltration...
16.
Source: youtube.com
Title: AI’s Rising Risks: Hacking, Virology, Loss of Control — With Dan Hendrycks
Link:https://www.youtube.com/watch?v=WcOlCtgreyQ
Source snippet
Did We Not Notice AGI Already Happened? | Joshua Achiam...
17.
Source: youtube.com
Title: Agentic AI in Cybersecurity: An Analytical Briefing
Link:https://www.youtube.com/watch?v=JqLT8H5Q3F4
Source snippet
GAEA Talks - Every AI Safety Warning Was Ignored with Dr Roman Yampolskiy...
18.
Source: frontiermodelforum.org
Link:https://www.frontiermodelforum.org/technical-reports/managing-advanced-cyber-risks-in-frontier-ai-frameworks/
Source snippet
Managing Advanced Cyber Risks in Frontier AI Frameworks - Frontier Model ForumFebruary 13, 2026 — TECHNICAL REPORT MANAGING ADVANCED CYBE...
Published: February 13, 2026
19.
Source: youtube.com
Title: GAEA Talks
Link:https://www.youtube.com/watch?v=jQsEF_HFbAE