Branchoria AI Doom AI Doom

Clear, balanced answers on advanced AI, loss of control and existential risk.

Within AI Doom

How Could People Misuse Advanced AI?

Some existential AI risk comes from humans using advanced models to magnify biological, cyber or military threats.

On this page

  • Biological and chemical threat assistance
  • Cyberattack automation
  • Military escalation and command risks
Preview for How Could People Misuse Advanced AI?

Introduction

One of the most contested strands in the debate about AI doom and existential risk is not only whether AI could behave in harmful ways on its own, but how humans might misuse powerful AI systems to cause catastrophic damage on a scale that threatens civilisation or even human survival. This page focuses squarely on that question: not the classic alignment problem of an AI pursuing unintended goals, but the very real possibility that people use increasingly capable AI to magnify biological threats, automate cyberattack and sabotage, or accelerate military escalation — with consequences that, in the extreme, could be catastrophic. While current AI systems are far from superintelligent, researchers and policymakers agree that misuse risks deserve serious attention now, because these pathways already intersect with domains — biosecurity, cybersecurity, geopolitics — where errors or deliberate harm can have cascading systemic effects.[GOV.UK]

Overview image for Misuse

Biological Threat Assistance: AI-Amplified Biosecurity Risks

AI’s ability to process, summarise and generate biological information at scale has opened new opportunities in research and healthcare — but those same capabilities could lower barriers to designing, optimising or deploying biological threats.

  • Dual-use research concerns. Scientific work on so-called dual-use capabilities identifies how AI could be misused to augment biological threat creation by assisting in the design of harmful agents, even when the original goal was legitimate research. The concern is that increasingly capable models could accelerate tasks such as protein design or genetic editing in ways that outpace current oversight frameworks.[PLOS]journals.plos.orgPLOSDual-use capabilities of concern of biological AI models | PLOS Computational BiologyMay 8, 2025…Published: May 8, 2025
  • Criminal dissemination and enhanced pathogens. Analyses of emerging bio-threat landscapes flag the risk that AI tools might help adversaries — state or non‑state — to design genetically modified organisms (GMOs) with enhanced lethality, resistance to treatment, or broader host ranges. Such manipulation could trigger widespread outbreaks that overwhelm healthcare systems and disrupt social stability if safeguards and surveillance fail.[Frontiers]frontiersin.orgFrontiersFrontiers | Artificial intelligence challenges in the face of biological threats: emerging catastrophic risks for public healthM…
  • No strong current evidence but plausible future risk. Government reports on frontier AI recognise that while present-day models do not demonstrably enable novice users to create biological weapons, future systems with deeper domain knowledge might significantly lower the expertise required to attempt such activities. This doesn’t mean biological catastrophe is imminent — but it does mean the potential trajectory of capability demands governance and research into mitigation now.[GOV.UK]

In summary, the biological misuse pathway illustrates how human-directed use of AI, particularly if combined with advances in biotechnology, could enable actors to pursue high‑consequence biological missions that were previously out of reach — a core concern for those who see AI misuse as a catastrophic risk vector.

Misuse illustration 1

Cyberattack Automation: Scaling Offensive Capabilities

Cybersecurity has long been a battleground of attackers and defenders, and advanced AI threatens to significantly shift the balance by automating and amplifying offensive operations.

  • AI uplift in cyber offence. Technical research into cybersecurity risk modelling shows that AI can systematically increase the efficacy, frequency and reach of cyberattacks by automating tasks that currently require skilled humans, from discovering vulnerabilities to crafting exploit code or bypassing defences. This systematic uplift accelerates the pace of attacks and broadens the pool of potential attackers.[arXiv]arxiv.orgarXiv Toward Quantitative Modeling of Cybersecurity Risks Due to AI MisusearXivToward Quantitative Modeling of Cybersecurity Risks Due to AI MisuseDecember 9, 2025…Published: December 9, 2025
  • Ease of weaponising AI for zero‑day exploits. There are emerging reports — including indicated use of AI‑assisted discovery of zero‑day vulnerabilities — that illustrate how AI is no longer merely a defensive tool but a dual‑use one: malicious actors could leverage it to find novel exploits or bypass security controls at scale, making existing legal and institutional frameworks for cybercrime less effective.[Reddit]reddit.comThe cybersecurity risks of AI-created exploits: Google Zero Day ExploitsRedditThe cybersecurity risks of AI-created exploits: Google Zero Day Exploits.May 13, 2026…Published: May 13, 2026
  • Broad concern in policy circles. National security and scientific assessments emphasise that general‑purpose AI could uplift the cyber expertise of a broad range of actors, including those lacking deep technical training, by automating complex operations like social engineering or code generation. While the same technologies also bolster cyber defence, the asymmetric advantage could favour attackers absent robust safeguards.[GOV.UK]

Taken together, this paints a picture where AI‑assisted cyber offence could magnify the destructive impact of digital attacks — from crippling infrastructure to undermining critical services — especially if adversaries weaponise AI faster than defences can adapt.

Studies reveal risks of relying on AI systems for military combat | Generation AI | full episode

Channel: Arizona’s Family (3TV / CBS 5) · Views: 254 · Uploaded: April 2026 · Length: 26 minutes

Open on YouTube

Military Escalation and Command Risks

AI is increasingly integrated into defence planning, from decision support to autonomous systems. This raises distinct misuse concerns in the context of war, deterrence, and human control over force.

  • AI in military decision‑making. Academic research on AI’s role in ‘resort‑to‑force’ decisions highlights the risk that incorporating AI into strategic choices — such as whether to escalate a conflict — could mediate or distort human judgement under pressure if operators over‑rely on algorithmic outputs or if adversaries exploit AI‑influenced systems.[cambridge]cambridge.orgCambridge University Press & AssessmentIntegrators at war: Mediating in AI-assisted resort-to-force decisions | Cambridge Forum on AI: La… University Press & Assessment
  • Global catastrophic risk through escalation. Longer‑form analyses of military AI argue that rapid adoption of AI for command, control, and autonomous weapons platforms could contribute to global catastrophic risk by lowering thresholds for conflict, shortening decision times, and complicating arms‑race dynamics, especially among nuclear‑armed states. Autonomous or semi‑autonomous systems may interact with strategic warning systems in unpredictable ways that enhance the chance of inadvertent escalation.[SSRN]papers.ssrn.comSSRNMilitary Artificial Intelligence as Contributor to Global Catastrophic Risk by Matthijs M. Maas, Kayla Matteucci, Di Cooke:: SSRNMay…
  • Governance and human oversight challenges. Even when AI systems are not themselves autonomous agents with independent agency, their integration into military operations raises questions about control and accountability — how decisions are made, who bears responsibility, and whether safeguards can operate under crisis conditions.

In the military sphere, misuse is less about rogue actors in the traditional sense and more about geopolitical dynamics and organisational incentives that could push states to deploy powerful AI systems in ways that increase the likelihood of large‑scale conflict or catastrophic mistakes.

Misuse illustration 2

Weighing Misuse in the Broader AI Doom Debate

Across biosecurity, cybersecurity and military contexts, several themes recur in the discourse on catastrophic misuse:

  • Human agency and dual use. Unlike the classic ‘loss of control’ alignment argument, misuse risks centre on humans using powerful AI to do harm — intentionally or negligently. Advanced AI systems amplify human capabilities, so the same tools that improve research or defence could also be repurposed for destructive ends.[ResearchTrend.AI]researchtrend.aiThe AI Risk Spectrum: From Dangerous Capabilities to Existential Threats | ResearchTrend.AIAugust 19, 2025…Published: August 19, 2025
  • Barriers and uncertainties. Experts broadly agree that present AI does not yet enable easy creation of biological weapons or fully autonomous offensive cyber capabilities, but they also emphasise uncertainty about how quickly capabilities will grow and where governance can keep pace.[GOV.UK]
  • Policy and governance gaps. Independent evaluations of AI industry preparedness underscore that many powerful AI developers lack comprehensive safeguards to prevent misuse — a gap that contributes to anxiety in both safety research and policy communities about the ability to manage these risks as capabilities increase.[Axios]axios.comAI firms flunk existential risk planning, new report findsDespite public statements by many AI company leaders acknowledging the importance of mitigating such risks, none of the firms were found…

Critics of misuse‑focused doom narratives argue that catastrophic human misuse is no more likely than the misuse of other powerful technologies — and that robust governance, norms, and international cooperation could prevent extreme outcomes — but proponents counter that the scale, speed and accessibility of AI‑enabled capabilities make this a distinct and urgent challenge.

AI Gets Military Secrets - What Could Go Wrong? | Warning Shots #40

Channel: The AI Risk Network | AI Safety · Views: 7.1K · Uploaded: May 2026 · Length: 28 minutes

Open on YouTube

Mike McCormick: AI Acceleration vs Risks, Funding Global Resilience, AGI scenarios, U.S. vs China

Channel: Delphi Digital · Views: 306 · Uploaded: October 2025 · Length: 1 hour 42 minutes

Open on YouTube

Implications and What Matters for the Future

Understanding catastrophic misuse is crucial because it represents a bridge between near‑term observable trends and the speculative but serious concerns about existential harm:

  • Warning signs to watch. Indicators such as widespread AI‑powered cyber incidents, deliberate use of AI tools by malicious actors, and rapid military integration with limited oversight could signal escalating misuse risks.
  • Risk mitigation paths. Experts propose targeted capability restrictions, improved governance frameworks, tighter export controls on high‑impact AI, and stronger international norms — particularly in biosecurity and military AI policy.
  • Uncertainty and plural perspectives. While misuse pathways are grounded in plausible mechanisms, there is active debate about how plausible catastrophic misuse is relative to other AI disaster scenarios, and about which interventions will be most effective.

By centring on the human dimensions of misuse — decisions, incentives, and institutional controls — this view of AI doom complements alignment concerns and underscores that existential threats from AI could arise not only from errant machine goals but also from how powerful tools are wielded in contexts where the stakes are highest.

Misuse illustration 3

Amazon book picks

Further Reading

Books and field guides related to How Could People Misuse Advanced AI?. Use these as the next step if you want deeper reading beyond the article.

Browse more on Amazon: Human Compatible Superintelligence The Coming Wave

As an Amazon Associate I earn from qualifying purchases.

eBay marketplace picks

Marketplace Samples

Example marketplace items related to this page. Use the search link to explore similar finds on eBay.

Using USA
Browse more on eBay.com

Example items shown for inspiration; availability and pricing can change. Branchoria may earn a commission if you purchase through outbound eBay links.

Browse more on eBay.co.uk

Example items shown for inspiration; availability and pricing can change. Branchoria may earn a commission if you purchase through outbound eBay links.

Endnotes

  1. Source: GOV.UK
    Title: International scientific report on the safety of advanced AI: interim report
    Link: https://www.gov.uk/government/publications/international-scientific-report-on-the-safety-of-advanced-ai/international-scientific-report-on-the-safety-of-advanced-ai-interim-report
    Source snippet

    22, 2025...

  2. Source: GOV.UK
    Title: Frontier AI: capabilities and risks – discussion paper
    Link: https://www.gov.uk/government/publications/frontier-ai-capabilities-and-risks-discussion-paper/frontier-ai-capabilities-and-risks-discussion-paper

  3. Source: journals.plos.org
    Link: https://journals.plos.org/ploscompbiol/article?id=10.1371%2Fjournal.pcbi.1012975
    Source snippet

    PLOSDual-use capabilities of concern of biological AI models | PLOS Computational BiologyMay 8, 2025...

    Published: May 8, 2025

  4. Source: arxiv.org
    Title: arXiv Toward Quantitative Modeling of Cybersecurity Risks Due to AI Misuse
    Link: https://arxiv.org/abs/2512.08864
    Source snippet

    arXivToward Quantitative Modeling of Cybersecurity Risks Due to AI MisuseDecember 9, 2025...

    Published: December 9, 2025

  5. Source: reddit.com
    Title: The cybersecurity risks of AI-created exploits: Google Zero Day Exploits
    Link: https://www.reddit.com/r/Technology_Law/comments/1tcae7l/the_cybersecurity_risks_of_aicreated_exploits/
    Source snippet

    RedditThe cybersecurity risks of AI-created exploits: Google Zero Day Exploits.May 13, 2026...

    Published: May 13, 2026

  6. Source: cambridge.org
    Link: https://www.cambridge.org/core/journals/cambridge-forum-on-ai-law-and-governance/article/integrators-at-war-mediating-in-aiassisted-resorttoforce-decisions/D176832403A3BE49069FD0E86038D989
    Source snippet

    Cambridge University Press & AssessmentIntegrators at war: Mediating in AI-assisted resort-to-force decisions | Cambridge Forum on AI: La...

  7. Source: papers.ssrn.com
    Link: https://papers.ssrn.com/sol3/Delivery.cfm/SSRN_ID4136610_code2918001.pdf?abstractid=4115010
    Source snippet

    SSRNMilitary [Artificial]({{ 'artificial-goals/' | relative_url }}) Intelligence as Contributor to Global Catastrophic Risk by Matthijs M. Maas, Kayla Matteucci, Di Cooke:: SSRNMay...

  8. Source: researchtrend.ai
    Link: https://researchtrend.ai/papers/2508.13700
    Source snippet

    The AI Risk Spectrum: From Dangerous Capabilities to Existential Threats | ResearchTrend.AIAugust 19, 2025...

    Published: August 19, 2025

  9. Source: axios.com
    Title: AI firms flunk existential risk planning, new report finds
    Link: https://www.axios.com/2025/12/03/ai-risks-agi-anthropic-google-openai
    Source snippet

    Despite public statements by many AI company leaders acknowledging the importance of mitigating such risks, none of the firms were found...

  10. Source: GOV.UK
    Link: https://www.gov.uk/government/publications/frontier-ai-capabilities-and-risks-discussion-paper/future-risks-of-frontier-ai-annex-a
    Source snippet

    Executive summary 2. Context 3. Current Frontier AI capabilities 4. Future Frontier AI capabilities 5. Other critical uncert...

  11. Source: cambridge.org
    Link: https://www.cambridge.org/core/journals/european-journal-of-international-security/article/inadvertent-escalation-in-the-age-of-intelligence-machines-a-new-model-for-nuclear-risk-in-the-digital-age/D1F1FC47D12FA4DCB12D1648412B696B
    Source snippet

    tional Security | Cambridge CoreOctober 15, 2021 — INADVERTENT ESCALATION IN THE AGE OF INTELLIGENCE MACHINES: A NEW MODEL FOR NUCLEAR RI...

    Published: October 15, 2021

  12. Source: frontiersin.org
    Link: https://www.frontiersin.org/articles/10.3389/frai.2024.1382356/full
    Source snippet

    FrontiersFrontiers | Artificial intelligence challenges in the face of biological threats: emerging catastrophic risks for public healthM...

  13. Source: papers.cool
    Link: https://papers.cool/arxiv/2508.13700
    Source snippet

    Immersive Paper DiscoveryAugust 19, 2025 — 2508.13700 Total: 1 #1 THE AI RISK SPECTRUM: FROM DANGEROUS CAPABILITIES TO EXISTENTIAL THREAT...

    Published: August 19, 2025

  14. Source: journals.ub.uni-koeln.de
    Link: https://journals.ub.uni-koeln.de/index.php/phai/article/view/2801
    Source snippet

    Survival Stories: a Taxonomic Analysis of AI Existential Risk | Philosophy of AIJune 20, 2025 — AI SURVIVAL STORIES: A TAXONOMIC ANALYSIS...

    Published: June 20, 2025

  15. Source: ai-safety-atlas.com
    Link: https://ai-safety-atlas.com/chapters/v1/risks/misuse-risks/
    Source snippet

    Misuse Risks - Chapter 2 - AI Safety AtlasEven when systems nominally keep humans in control, combat conditions can make this control mor...

Additional References

  1. Source: tandfonline.com
    Link: https://www.tandfonline.com/doi/abs/10.1080/23738871.2019.1701693
    Source snippet

    r 9, 2019 — 3,168 Views 34 CrossRef citations to date 0 Altmetric Articles THE AI-CYBER NEXUS: IMPLICATIONS FOR MILITARY ESCALATION, DETE...

  2. Source: cris.technion.ac.il
    Link: https://cris.technion.ac.il/en/publications/understanding-the-theoretical-limits-of-ai-enabled-pathogen-desig
    Source snippet

    the Theoretical Limits of AI-Enabled Pathogen Design: Insights from a Delphi Study - Technion - Israel Institute of TechnologyUNDERSTANDI...

  3. Source: abdn.elsevierpure.com
    Title: the ai cyber nexus implications for military escalation deterrenc
    Link: https://abdn.elsevierpure.com/en/publications/the-ai-cyber-nexus-implications-for-military-escalation-deterrenc
    Source snippet

    AI-cyber nexus: implications for military escalation, deterrence, & strategic stability - The University of Aberdeen Research PortalDecem...

  4. Source: globalsecurity.org
    Link: https://www.globalsecurity.org/military/systems/ai-risks.htm
    Source snippet

    ters strategic stability by compressing decision timelines, potentially undermining nuclear de...

  5. Source: forethought.org
    Title: The biggest risks come from the le
    Link: https://www.forethought.org/research/ai-enabled-coups-how-a-small-group-could-use-ai-to-seize-power
    Source snippet

    AI-Enabled Coups: How a Small Group Could Use AI to Seize PowerApril 15, 2025 — 1 INTRODUCTION This report assesses the risk that a small...

    Published: April 15, 2025

  6. Source: longtermrisk.org
    Title: Discussions of the existential r
    Link: https://longtermrisk.org/overview-of-transformative-ai-misuse-risks-what-could-go-wrong-beyond-misalignment/
    Source snippet

    Overview of Transformative AI Misuse Risks: What Could Go Wrong Beyond Misalignment – Center on Long-Term RiskDecember 9, 2024 — OVERVIEW...

    Published: December 9, 2024

  7. Source: youtube.com
    Link: https://www.youtube.com/watch?v=jAQKdpwXFzs
    Source snippet

    2 AI Gets Military Secrets - What Could Go Wrong? | Warning Shots #40...

  8. Source: youtube.com
    Link: https://www.youtube.com/watch?v=kgHwrOhwe04
    Source snippet

    5 Existential risk: Is artificial intelligence a threat that will end humanity...

  9. Source: safer-ai.org
    Title: a short overview of ai related biorisks
    Link: https://www.safer-ai.org/a-short-overview-of-ai-related-biorisks
    Source snippet

    A short overview of AI-related biorisks – SaferAIApril 21, 2024 — Image A SHORT OVERVIEW OF AI-RELATED BIORISKS PUBLICATION DATE April 21...

    Published: April 21, 2024

  10. Source: youtube.com
    Title: How the Mythos AI Breached Military-Grade Defenses in 18 Seconds
    Link: https://www.youtube.com/watch?v=gjpxQeN4iM8
    Source snippet

    4 Studies reveal risks of relying on AI systems for military combat | Generation AI...

Topic Tree

Follow this branch

Parent topic

AI Doom

Related pages 9

More on this topic 4