Within AI Doom
What Rules Could Reduce AI Doom Risk?
Compute controls, audits and safety frameworks are proposed as ways to slow or constrain the riskiest AI development.
On this page
- Compute and model access controls
- Lab safety frameworks
- Audits, incident reporting and coordination
Page outline Jump by section
Introduction
As the debate about AI doom and existential risk the possibility that future highcapability systems could cause irreversible global catastrophe has grown, so has interest in governance approaches that might significantly reduce those risks. One set of proposals focuses on compute governance and frontier AI safety rules: mechanisms to regulate the computational resources and safety practices that underpin the creation and deployment of the most powerful AI models. These ideas are distinctive because they aim to intervene earlier in the AI development pipeline at the hardware and procedural level rather than only trying to influence what models do after training. Proponents argue that compute access and rigorous safety standards could slow uncontrolled capability races and make strategic oversight more effective, while critics worry about feasibility, enforcement and innovation bottlenecks. The rest of this page explains how these governance ideas work, what frameworks are being discussed, and what disputes surround them. [AI Security & Safety Directory]aisecurityandsafety.orgcompute governanceAI Security & Safety DirectoryCompute Governance: Controlling AI Through Hardware & Compute Access (2026) | AI Safety DirectoryApril 3, 2026…
Why Compute and Safety Rules Matter for AI Doom Risk
For models nearing the frontier of capability meaning those that could outperform current stateoftheart systems by orders of magnitude compute (specialised hardware and large training runs) is a distinctive bottleneck. Only a handful of labs and companies have the resources to undertake the immense computations required for the largest models, so compute becomes a concentrated, observable choke point that policymakers could in principle monitor or regulate. [AI Security & Safety Directory]aisecurityandsafety.orgcompute governanceAI Security & Safety DirectoryCompute Governance: Controlling AI Through Hardware & Compute Access (2026) | AI Safety DirectoryApril 3, 2026…
From an AI doom perspective, the rationale for focusing on compute and safety rules is threefold:
- Upstream control: By governing inputs (compute) rather than outputs (trained systems), regulators could intervene before potentially dangerous systems exist. This reduces reliance on afterthefact restrictions that may be too late if misaligned systems already exist. [AI Security & Safety Directory]aisecurityandsafety.orgcompute governanceAI Security & Safety DirectoryCompute Governance: Controlling AI Through Hardware & Compute Access (2026) | AI Safety DirectoryApril 3, 2026…
- Observable thresholds: Compute usage can be quantified (e.g. floatingpoint operation counts), which makes it a clearer trigger for reporting, review or restriction than subjective judgments about dangerous capabilities. [AI Security & Safety Directory]aisecurityandsafety.orgcompute governanceAI Security & Safety DirectoryCompute Governance: Controlling AI Through Hardware & Compute Access (2026) | AI Safety DirectoryApril 3, 2026…
- Safety signalling and coordination: Safety rules tied to training such as mandatory evaluations, redteaming or risk assessments before major compute runs can formalise safety culture and provide early warning to regulators and researchers alike. [OpenAI]OpenAIFrontier AI regulation: Managing emerging risks to public safety | OpenAI…
However, governance proposals in this area are still emerging, and there is active debate about how effective they could be, what form they should take, and whether they could be enforced equitably at global scale.
Compute Governance: What It Is and How It Could Work
Compute governance is a suite of policy mechanisms designed to monitor, limit or direct access to the computational resources used for training powerful AI models. It draws on the insight that unlike data and algorithms, compute is a physical input that flows through identifiable supply chains (manufacturers, cloud providers, data centres). [AI Security & Safety Directory]aisecurityandsafety.orgcompute governanceAI Security & Safety DirectoryCompute Governance: Controlling AI Through Hardware & Compute Access (2026) | AI Safety DirectoryApril 3, 2026…
Key elements of compute governance include: [regulations.ai]regulations.aiCompute GovernanceDEFINITION Compute Governance refers to regulatory and policy approaches that lev…
- Reporting requirements for large compute runs: Organisations training models above defined compute thresholds (measured in FLOPs or similar metrics) would be required to notify regulators or independent auditors before, during or after training. [AI Security & Safety Directory]aisecurityandsafety.orgcompute governanceAI Security & Safety DirectoryCompute Governance: Controlling AI Through Hardware & Compute Access (2026) | AI Safety DirectoryApril 3, 2026…
- Export controls and hardware restrictions: Government export controls on advanced AI accelerators akin to existing semiconductor export restrictions could limit the international transfer of highend compute resources. These are already seen in USled export controls on some chip types. [AI Security & Safety Directory]aisecurityandsafety.orgcompute governanceAI Security & Safety DirectoryCompute Governance: Controlling AI Through Hardware & Compute Access (2026) | AI Safety DirectoryApril 3, 2026…
- Cloud provider oversight: Cloud infrastructure could be required to implement knowyourcustomer policies or usage monitoring to detect and flag large AI training jobs. [AI Security & Safety Directory]aisecurityandsafety.orgcompute governanceAI Security & Safety DirectoryCompute Governance: Controlling AI Through Hardware & Compute Access (2026) | AI Safety DirectoryApril 3, 2026…
- Mandatory safety evaluations: Before allocating large compute resources, developers might have to demonstrate that they have conducted risk assessments, safety tests and redteaming that meet defined standards. [AI Security & Safety Directory]aisecurityandsafety.orgcompute governanceAI Security & Safety DirectoryCompute Governance: Controlling AI Through Hardware & Compute Access (2026) | AI Safety DirectoryApril 3, 2026…
Supporters argue that these measures provide clear, measurable governance triggers that can adapt over time as compute costs decline or AI training patterns change. They also see compute governance as compatible with other safety measures, such as model evaluations and deployment controls. [AI Security & Safety Directory]aisecurityandsafety.orgcompute governanceAI Security & Safety DirectoryCompute Governance: Controlling AI Through Hardware & Compute Access (2026) | AI Safety DirectoryApril 3, 2026…
Frontier AI Safety Rules Beyond Compute
While compute governance focuses on hardware access, frontier AI safety rules encompass broader procedural and organisational requirements for how frontier models are developed, evaluated and deployed. Drawing on both private sector proposals and government guidance such as the UKs emerging frontier AI safety processes these rules often include:
- Risk assessments throughout the AI lifecycle: Developers of highimpact models are encouraged or required to assess risks before training, prespecify risk thresholds, and monitor evolving risks as models are developed and deployed. [GOV.UK]GOV.UKEmerging processes for frontier AI safety27, 2023…
- Independent evaluations and redteaming: External scrutiny helps surface dangerous behaviours or capabilities that internal teams might miss. [GOV.UK]GOV.UKfrontier ai safety commitments ai seoul summit 2024Frontier AI Safety Commitments, AI Seoul Summit 2024 - GOV.UKFebruary 7, 2025…
- Structured reporting and information sharing: Companies share safety plans, assessment results and incident reports with governments or through third parties to build systemic awareness of frontier AI risks. [GOV.UK]GOV.UKFrontier models could berisks of frontier AI (Annex A) - GOV.UKApril 28, 2025 However, the risks and opportunities posed by a given model derive from its capab…
- Security controls and vulnerability reporting: Similar to software bugreporting programmes, organisations identify security issues, including those related to model misuse or exploitation. [GOV.UK]GOV.UKFrom: Department for Science, Innovation and Technology Publishedprocesses for frontier AI safety - GOV.UKOctober 27, 2023 Policy paper EMERGING PROCESSES FOR FRONTIER AI SAFETY Overview of emerging f…
The Frontier AI Safety Commitments emerging from international summits echo these ideas, committing participating organisations to set risk thresholds, articulate safety mitigation processes, and outline procedures for handling models that exceed predefined tolerable risk levels. [GOV.UK]GOV.UKEmerging processes for frontier AI safety27, 2023…
Collectively, these measures aim to create a culture of structured safeguard planning that goes beyond individual compliance checklists integrating safety into core development decisions.
Challenges and Criticisms
Despite their appeal in principle, compute governance and frontier safety rules face significant challenges that shape the realworld feasibility and impact of these policies:
- Evolving technology and thresholds: Algorithmic advances can dramatically reduce the compute needed to achieve a given capability, meaning fixed thresholds could quickly become outdated or meaningless. [AI Security & Safety Directory]aisecurityandsafety.orgcompute governanceAI Security & Safety DirectoryCompute Governance: Controlling AI Through Hardware & Compute Access (2026) | AI Safety DirectoryApril 3, 2026…
- Concentration of power: Restrictive compute access might entrench incumbent firms with resources to comply, widening gaps between large labs and smaller research groups or public institutions. [ai-safety-atlas.com]ai-safety-atlas.comCompute GovernanceChapter 4 - AI Safety Atlas…
- Distributed training and open research: Emerging technical methods like distributed training or opensource models could enable highcapability development without a single, observable training run, complicating governance enforcement. [ai-safety-atlas.com]ai-safety-atlas.comGovernance ArchitecturesFSFs are policies that AI companies create to guide their development process and ensure the…
- International coordination: Compute supply chains and research talent are globally distributed, so effective governance requires multilateral cooperation that can be hard to achieve amid geopolitical competition. Differences in national interests could undermine harmonised rules. [AI Security & Safety Directory]aisecurityandsafety.orgcompute governanceAI Security & Safety DirectoryCompute Governance: Controlling AI Through Hardware & Compute Access (2026) | AI Safety DirectoryApril 3, 2026…
- Enforcement and compliance gaps: Even where rules exist, without robust monitoring and the technological ability to detect noncompliance especially across borders governance may remain largely aspirational. [AI Security & Safety Directory]aisecurityandsafety.orgcompute governanceAI Security & Safety DirectoryCompute Governance: Controlling AI Through Hardware & Compute Access (2026) | AI Safety DirectoryApril 3, 2026…
Critics therefore stress that compute governance should be viewed as part of a broader governance ecosystem, not a standalone silver bullet. In expert analyses, compute thresholds are framed as initial screening tools that trigger deeper scrutiny mechanisms such as risk assessments or external evaluations rather than as sole determinants of regulatory action. [ai-safety-atlas.com]ai-safety-atlas.comCompute GovernanceChapter 4 - AI Safety Atlas…
Where the Debate Stands
Compute governance and frontier safety rules are emerging policy tools. Research institutions, governments and industry groups continue to refine how such mechanisms might work, what metrics to use, and how to balance safety with innovation. Some scholars argue for adaptive risk management systems that combine technical measures with institutional capacity building and scenario planning, recognising that static regulatory checklists will struggle with fastmoving technological landscapes. [arXiv]arxiv.orgarXivGoverning frontier general-purpose AI in the public sector: adaptive risk management and policy capacity under uncertainty through 2…
There is also ongoing debate over principlebased versus rulebased regulation: highlevel safety principles can provide flexible direction but may lack enforceability, while specific rules offer clarity but can become outdated or overly rigid as technology evolves. [arXiv]arxiv.orgarXivGoverning frontier general-purpose AI in the public sector: adaptive risk management and policy capacity under uncertainty through 2…
What is clear, however, is that concerns about AI existential risk have elevated governance conversations to include infrastructurelevel controls and procedural safety obligations not only ethical guidelines or afterthefact restrictions. Whether these approaches can meaningfully reduce the probability of catastrophic AI outcomes remains uncertain, but they are increasingly central to how governments and researchers frame preparedness for frontier AI. [AI Security & Safety Directory]aisecurityandsafety.orgcompute governanceAI Security & Safety DirectoryCompute Governance: Controlling AI Through Hardware & Compute Access (2026) | AI Safety DirectoryApril 3, 2026…
Amazon book picks
Further Reading
Books and field guides related to What Rules Could Reduce AI Doom Risk?. Use these as the next step if you want deeper reading beyond the article.
The Coming Wave
Directly discusses regulation, containment, and frontier AI governance.
eBay marketplace picks
Marketplace Samples
Example marketplace items related to this page. Use the search link to explore similar finds on eBay.
Endnotes
-
Source: OpenAI
Link: https://openai.com/index/frontier-ai-regulation/Source snippet
Frontier AI regulation: Managing emerging risks to public safety | OpenAI...
-
Source: GOV.UK
Title: Emerging processes for frontier AI safety
Link: https://www.gov.uk/government/publications/emerging-processes-for-frontier-ai-safety/emerging-processes-for-frontier-ai-safetySource snippet
27, 2023...
-
Source: GOV.UK
Title: frontier ai safety commitments ai seoul summit 2024
Link: https://www.gov.uk/government/publications/frontier-ai-safety-commitments-ai-seoul-summit-2024/frontier-ai-safety-commitments-ai-seoul-summit-2024Source snippet
Frontier AI Safety Commitments, AI Seoul Summit 2024 - GOV.UKFebruary 7, 2025...
Published: February 7, 2025
-
Source: ai-safety-atlas.com
Title: Compute Governance
Link: https://ai-safety-atlas.com/chapters/v1/governance/compute-governanceSource snippet
Chapter 4 - AI Safety Atlas...
-
Source: arxiv.org
Link: https://arxiv.org/abs/2604.06215Source snippet
arXivGoverning frontier general-purpose AI in the public sector: adaptive risk management and policy capacity under [uncertainty]({{ 'uncertainty/' | relative_url }}) through 2...
-
Source: arxiv.org
Link: https://arxiv.org/abs/2407.07300Source snippet
arXivFrom Principles to Rules: A Regulatory Approach for Frontier AIJuly 10, 2024...
Published: July 10, 2024
-
Source: GOV.UK
Title: Frontier models could be
Link: https://www.gov.uk/government/publications/frontier-ai-capabilities-and-risks-discussion-paper/future-risks-of-frontier-ai-annex-aSource snippet
risks of frontier AI (Annex A) - GOV.UKApril 28, 2025 However, the risks and opportunities posed by a given model derive from its capab...
Published: April 28, 2025
-
Source: GOV.UK
Title: From: Department for Science, Innovation and Technology Published
Link: https://www.gov.uk/government/publications/emerging-processes-for-frontier-ai-safetySource snippet
processes for frontier AI safety - GOV.UKOctober 27, 2023 Policy paper EMERGING PROCESSES FOR FRONTIER AI SAFETY Overview of emerging f...
Published: October 27, 2023
-
Source: ai-safety-atlas.com
Title: Governance Architectures
Link: https://ai-safety-atlas.com/chapters/v1/governance/governance-architectures/Source snippet
FSFs are policies that AI companies create to guide their development process and ensure the...
-
Source: aisecurityandsafety.org
Title: compute governance
Link: https://aisecurityandsafety.org/en/guides/compute-governance/Source snippet
AI Security & Safety DirectoryCompute Governance: Controlling AI Through Hardware & Compute Access (2026) | AI Safety DirectoryApril 3, 2026...
Published: April 3, 2026
-
Source: aisecurityandsafety.org
Link: https://aisecurityandsafety.org/en/glossary/compute-governance/Source snippet
AI Security & Safety DirectoryCompute Governance AI Governance Definition & Guide | AI Safety DirectoryMarch 27, 2026...
Published: March 27, 2026
-
Source: aisecurityandsafety.org
Title: compute governance
Link: https://aisecurityandsafety.org/it/guides/compute-governance/Source snippet
Controlling AI Through Hardware & Compute Access (2026) | AI Safety DirectoryApril 3, 2026 COMPUTE GOVERNANCE: CONTROLLING AI THROUGH H...
Published: April 3, 2026
-
Source: regulations.ai
Title: Compute Governance
Link: https://regulations.ai/glossary/compute-governanceSource snippet
DEFINITION Compute Governance refers to regulatory and policy approaches that lev...
-
Source: wikimolt.org
Title: Compute Governance
Link: https://www.wikimolt.org/page/Compute%20GovernanceSource snippet
WikimoltCOMPUTE GOVERNANCE Recent edits: wikimoltbot 2026-02-21 08:13:56 "Create wanted page: define compute governance, mechanisms (th...
Published: February 21, 2026
Additional References
-
Source: s-rsa.com
Link: https://s-rsa.com/index.php/agi/article/view/14519Source snippet
A Framework for the Private Governance of Frontier [Artificial]({{ 'artificial-goals/' | relative_url }}) Intelligence | SuperIntelligence - Robotics - Safety & AlignmentMay 29, 202...
-
Source: aigouvernance.com
Link: https://aigouvernance.com/ai-governance-at-the-frontier-unpacking-foundational-assumptionsai-governance-at-the-frontier-cset/Source snippet
AI Governance at the Frontier: Unpacking Foundational AssumptionsAI Governance at the Frontier (CSET) - AI Governance Library, Articles...
-
Source: aisecurityandsafety.org
Title: Compute Governance AI Governance Definition & Guide | AI Safety Directory
Link: https://aisecurityandsafety.org/pt/glossary/compute-governance/Source snippet
March 10, 2026 COMPUTE GOVERNANCE governance ltima atualizao: March 10, 2026 DEFINIO A set of policy and technical mechanisms aime...
Published: March 10, 2026
-
Source: metr.org
Title: Common Elements of Frontier AI Safety Policies
Link: https://metr.org/common-elementsSource snippet
METRDecember 16, 2025 Common Elements of Frontier AI Safety Policies DATE December 16, 2025 COMMON ELEMENTS OF FRONTIER AI SAFETY POLIC...
Published: December 16, 2025
-
Source: youtube.com
Link: https://www.youtube.com/watch?v=7Fdqn4yBGzkSource snippet
AI Billionaire on Existential Risk: Jaan Tallinn #112...
-
Source: youtube.com
Title: Lennart Heim on the AI Triad: Compute, Data, and Algorithms
Link: https://www.youtube.com/watch?v=7EwAdTqGgWMSource snippet
The RAISE Act: Minimum Standards for Frontier AI Development, with NY Assembly Member Alex Bores...
-
Source: youtube.com
Link: https://www.youtube.com/watch?v=_YTVydWBMvkSource snippet
Lennart Heim on the AI Triad: Compute, Data, and Algorithms...
-
Source: ideas.repec.org
Link: https://ideas.repec.org/p/arx/papers/2603.10015.htmlSource snippet
coordination gap in frontier AI safety policiesMarch 16, 2026 THE COORDINATION GAP IN FRONTIER AI SAFETY POLICIES AUTHOR Listed: * Isaa...
Published: March 16, 2026
-
Source: youtube.com
Title: AI Billionaire on Existential Risk: Jaan Tallinn #112
Link: https://www.youtube.com/watch?v=6MaUZ4Hi7os -
Source: youtube.com
Link: https://www.youtube.com/watch?v=yK5odT1jz-USource snippet
220 - Guest: Markus Anderljung, AI Regulation Researcher, part 1...
Topic Tree
Follow this branch
Parent topic
AI DoomRelated pages 9
- AI Takeoff Could AI Improvement Run Away From US?
- Autonomy When Does AI Autonomy Become Dangerous?
- Control Tools Can We Make Advanced AI Understandable?
- Evals Can Tests Catch Dangerous AI in Time?
- Loss of Control How Could Humans Lose Control of AI?
- Misuse How Could People Misuse Advanced AI?
- P Doom What Does p(doom) Really Mean?
- Race Pressure Why AI Races Can Make Safety Harder
- +1 more in sidebar