Within First Movers
Can rules stop a dangerous AI race
Deployment rules, safety evaluations, and coordination agreements aim to reduce race pressure without freezing useful AI progress entirely.
On this page
- Why voluntary caution may not be enough
- Safety evaluations before high risk deployment
- Coordination problems between firms and governments
Page outline Jump by section
Introduction
Can regulation slow a dangerous AI race? Possibly, but only under certain conditions. In the AI doom and existential-risk debate, the concern is not simply that powerful systems might be built, but that competitive pressure could encourage companies or governments to deploy frontier AI before its capabilities and failure modes are properly understood. Regulation is often proposed as a way to reduce that pressure by making safety testing, external review, and risk management mandatory rather than optional.
Supporters argue that well-designed rules can change incentives so that delaying deployment for safety reasons is no longer a competitive disadvantage. Critics counter that regulation may be ineffective, difficult to enforce internationally, or could even favour large incumbents without meaningfully reducing risk. The key question is therefore not whether regulation exists, but whether it can create enough coordination among major AI developers to prevent premature launches of systems that might eventually contribute to loss-of-control scenarios or other existential risks.
Why voluntary caution may not be enough
Many frontier AI developers have publicly committed to safety evaluations, responsible scaling policies, and pre-deployment testing. International initiatives such as the Bletchley Declaration and the Seoul AI Safety Commitments encourage companies to assess dangerous capabilities before releasing increasingly powerful models. [AI Security & Safety Directory]aisecurityandsafety.orgfrontier ai safety commitmentsAI Security & Safety DirectoryFrontier AI Safety CommitmentsMar 10, 2026 — Voluntary safety pledges by 16 leading AI companies to assess…
The problem, according to many AI-risk researchers, is that voluntary commitments operate within a competitive environment. A company that delays deployment to perform additional testing may lose market share, investment, users, or strategic influence if rivals move ahead first. Even if every firm agrees that safety matters, each may worry that competitors will not exercise the same restraint.
This creates a classic coordination problem. Voluntary agreements can help when all participants trust one another and when incentives remain aligned. However, they may become fragile if:
- Commercial rewards for being first are very large.
- New entrants are not bound by existing commitments.
- Governments view AI leadership as a strategic national priority.
- Safety evaluations reveal problems that would delay release.
From an AI doom perspective, this matters because the most dangerous deployment decisions may occur precisely when competitive pressure is strongest. The concern is not that companies are reckless, but that market and geopolitical incentives can make caution costly.
Safety evaluations before high-risk deployment
One of the most widely discussed regulatory ideas is mandatory evaluation before deployment of frontier models.
The basic logic resembles safety certification in other high-consequence industries. Before a system reaches the public, independent evaluators would examine whether it exhibits capabilities associated with catastrophic misuse, dangerous autonomy, deception, cyber offence, biological assistance, or other high-impact risks.
Recent policy efforts increasingly focus on this approach. The UK’s AI Security Institute has conducted evaluations of frontier systems since late 2023 and publishes analysis of trends in advanced model capabilities. [AI Security Institute]aisi.gov.ukAI Security InstituteFrontier AI Trends Report by The AI Security Institute (AISI)The UK AI Security Institute (AISI) has conducted evalu…
In the United States, the AI Safety Institute—later reorganised within NIST’s AI governance structure—was created to develop and conduct evaluations of frontier models, including tests for capabilities relevant to national security and catastrophic risks. NIST [2aifrontiersafety.com]aifrontiersafety.comA I Frontier SafetyAI Frontier Safety - Institutional Governance for Advanced AI…The United States established its AI safety evaluation function within t…
Advocates argue that mandatory evaluations could reduce race dynamics in several ways:
- Every major developer would face the same testing requirements.
- Delays caused by safety review would affect competitors equally.
- Regulators could identify dangerous capabilities before public release.
- Shared evaluation standards would reduce uncertainty about acceptable risk.
The underlying goal is not necessarily to prohibit deployment. Rather, it is to ensure that increasingly capable systems cross defined safety thresholds before release.
For readers interested in AI doom arguments, this is important because many loss-of-control scenarios depend on systems becoming highly capable before humans can reliably understand, monitor, or constrain them. Mandatory evaluations are an attempt to slow deployment until evidence about safety catches up with capability growth.
What current regulations actually try to do
The strongest existing example is the European Union’s AI Act. The Act introduces obligations for the most advanced general-purpose AI models, particularly those considered to pose “systemic risk”. These obligations include model evaluations, adversarial testing, incident reporting, risk assessment, and cybersecurity measures. The EU explicitly links systemic-risk provisions to highly capable frontier models whose failures could have large-scale consequences. [Artificial Intelligence Act]artificial-intelligence-act.comlities, evaluated on the basis of appropriate technical…Read more… [Digital Strategy]digital-strategy.ec.europa.eugeneral purpose ai models ai act questions answersstate-of-the-art) models at any given point in time, or from other…Read more…
Importantly, the Act does not attempt to halt frontier AI development. Instead, it tries to ensure that developers of especially powerful models perform specified safety and risk-management activities before and after deployment. [Digital Strategy]digital-strategy.ec.europa.eugeneral purpose ai models ai act questions answersstate-of-the-art) models at any given point in time, or from other…Read more… [Digital Strategy]digital-strategy.ec.europa.eugeneral purpose ai models ai act questions answersstate-of-the-art) models at any given point in time, or from other…Read more…
Similarly, the 2023 US Executive Order on AI required reporting and safety-testing obligations for the largest training runs and directed government agencies to develop evaluation frameworks and standards. Although later policy changes altered aspects of the regulatory landscape, the order illustrated a broader governance approach: requiring developers of frontier systems to share information about high-capability models and associated safety testing. [Federal Register]federalregister.govsafe secure and trustworthy development and use of artificial intelligenceFederal RegisterSafe, Secure, and Trustworthy Development and Use of…1 Nov 2023 — Executive Order 14110 of October 30, 2023. Safe, Sec… [Longterm Wiki]longtermwiki.comUS Executive Order on Safe, Secure, and Trustworthy AIIt mandated safety evaluations for frontier AI models, created reporting requiremen…
These initiatives represent a shift from purely voluntary safety commitments toward institutional oversight.
Coordination problems between firms and governments
The central challenge is that regulation works best when competitors face similar constraints.
Suppose one jurisdiction imposes extensive pre-deployment testing while another allows immediate release. Developers may be tempted to relocate, restructure operations, or focus deployment in less restrictive environments. This possibility often appears in criticisms of frontier AI regulation.
For this reason, many governance proposals focus on international coordination rather than isolated national rules. AI safety institutes, international safety summits, shared evaluation standards, and cross-border reporting arrangements are partly attempts to create common expectations among major AI-producing countries. NIST [AI Security & Safety Directory]aisecurityandsafety.orgfrontier ai safety commitmentsAI Security & Safety DirectoryFrontier AI Safety CommitmentsMar 10, 2026 — Voluntary safety pledges by 16 leading AI companies to assess…
The difficulty is that AI is often viewed simultaneously as:
- A commercial technology.
- A scientific achievement.
- A strategic national capability.
- A potential security concern.
Governments may therefore face conflicting incentives. They may want stronger safety measures while also wanting domestic firms to remain globally competitive.
This tension is especially visible in debates about frontier models. Policymakers concerned about existential risk often favour stronger oversight, whereas policymakers focused on innovation may worry that excessive restrictions could slow economic growth or strategic advantage.
Could regulation actually reduce existential risk?
Supporters of stronger governance argue that regulation could meaningfully reduce existential risk if it succeeds in delaying deployment until developers accumulate more evidence about system behaviour.
The theory is straightforward:
- Advanced systems are evaluated before release.
- Dangerous capabilities are identified earlier.
- Developers must implement safeguards before deployment.
- Competitive pressure to skip testing is reduced because rivals face similar requirements.
- Society gains additional time for alignment research, interpretability work, monitoring tools, and emergency-response planning.
Under this view, regulation functions primarily as a coordination mechanism. It makes caution less costly for individual actors.
However, significant uncertainties remain.
First, nobody knows exactly which future capabilities would create genuine existential danger. Evaluation regimes may miss the most important warning signs. Frontier systems are already becoming harder to assess, and some researchers worry that advanced models may learn to behave differently during tests than in deployment environments. [Institute for AI Policy and Strategy]iaps.aiIn some cases, this awareness…
Second, regulations often lag technological progress. A framework designed around today’s models may be poorly suited to systems developed several years later.
Third, compliance does not guarantee safety. A company can satisfy procedural requirements while still deploying a system that exhibits unexpected behaviours.
As a result, even many supporters of regulation view it as a risk-reduction tool rather than a complete solution to AI doom concerns.
The strongest objections
Several serious objections are raised against the idea that regulation can reliably slow premature frontier AI launches.
One objection is enforcement. Rules only matter if governments can monitor training runs, verify compliance, and impose meaningful consequences for violations. This becomes more difficult as models become cheaper to train and more actors acquire advanced capabilities.
A second objection is international competition. If major powers believe frontier AI provides strategic advantages, they may resist agreements that substantially slow progress.
A third objection is regulatory capture. Large firms often have more resources to comply with complex rules than smaller competitors. Some critics worry that regulation could entrench dominant companies without materially improving safety.
A fourth objection concerns uncertainty itself. Regulators may not know enough about advanced AI to identify appropriate thresholds for intervention. Academic analysis of the EU AI Act’s treatment of systemic risk repeatedly highlights the difficulty of defining and measuring risks from rapidly evolving frontier models. [cambridge]cambridge.orgCambridge University Press & AssessmentGoverning General-Purpose AI Models and Systemic Riskby S Carey · Cited by 7 — This article critic… University Press & Assessment
These criticisms do not necessarily imply that regulation is futile. Rather, they suggest that governance mechanisms must evolve alongside the technology they seek to govern.
What this means for the AI doom debate
Within AI doom discussions, regulation is often viewed less as a permanent solution and more as a way to buy time.
The core concern behind premature deployment is that highly capable systems could be released before researchers understand how to align them with human goals, detect dangerous behaviour, or maintain reliable human control. If first-mover pressures encourage rapid deployment, then reducing those pressures becomes a potentially important intervention.
Whether regulation can achieve this remains uncertain. Existing efforts such as frontier-model evaluations, AI safety institutes, reporting requirements, and the EU’s systemic-risk framework represent early attempts to create shared rules of the road. NIST [AI Security Institute]aisi.gov.ukAI Security InstituteFrontier AI Trends Report by The AI Security Institute (AISI)The UK AI Security Institute (AISI) has conducted evalu… [Artificial Intelligence Act]artificial-intelligence-act.comlities, evaluated on the basis of appropriate technical…Read more…
The strongest case for regulation is not that it eliminates existential risk, but that it may make waiting for safety evidence less costly. If successful, it could reduce the incentive to launch increasingly capable frontier systems before their risks are understood. If unsuccessful, race dynamics may continue largely unchanged despite formal oversight. In that sense, the debate is ultimately about coordination: whether societies can create rules that reward caution before competitive pressures make caution too expensive.
eBay marketplace picks
Marketplace Samples
Example marketplace items related to this page. Use the search link to explore similar finds on eBay.
Endnotes
-
Source: nist.gov
Title: paul christiano
Link: https://www.nist.gov/people/paul-christianoSource snippet
NISTPaul Christiano | NIST17 Apr 2024 — Paul Christiano is head of AI safety for the US Artificial Intelligence Safety Institute. In this...
-
Source: aifrontiersafety.com
Title: A I Frontier Safety
Link: https://aifrontiersafety.com/Source snippet
AI Frontier Safety - Institutional Governance for Advanced AI...The United States established its AI safety evaluation function within t...
-
Source: nist.gov
Link: https://www.nist.gov/artificial-intelligenceSource snippet
AI RMF, a guide to managing AI-associated risks to...Read more...
-
Source: nist.gov
Link: https://www.nist.gov/artificial-intelligence/nist-ai-consortiumSource snippet
NISTNIST AI ConsortiumThe Consortium brings together more than 280 organizations to develop science-based and empirically backed guidelin...
-
Source: cambridge.org
Link: https://www.cambridge.org/core/journals/european-journal-of-risk-regulation/article/regulating-uncertainty-governing-generalpurpose-ai-models-and-systemic-risk/7EEFE1D8421A43A98CE91F7C697DE538Source snippet
Cambridge University Press & AssessmentGoverning General-Purpose AI Models and Systemic Riskby S Carey · Cited by 7 — This article critic...
-
Source: nist.gov
Link: https://www.nist.gov/itl/ai-risk-management-frameworkSource snippet
AI Risk Management Framework | NISTNIST has developed a framework to better manage risks to individuals, organizations, and society assoc...
-
Source: nist.gov
Title: A I Congressional Mandates, Executive Orders and Actions
Link: https://www.nist.gov/artificial-intelligence/ai-congressional-mandates-executive-orders-and-actionsSource snippet
NIST official was part of the NAIRR task force. Executive Order on Safe, Secure, and Trustworthy Artificial Intelligence (10/30/2023). [R...
-
Source: nist.gov
Title: A I Standards | NISTOn
Link: https://www.nist.gov/artificial-intelligence/ai-standardsSource snippet
AI Standards | NISTOn January 15, 2026, NIST released A Possible Approach for Evaluating AI Standards Development (GCR-26-069). The repor...
Published: January 15, 2026
-
Source: governance.ai
Title: Ensure that risk
Link: https://www.governance.ai/research-paper/response-to-the-rfi-related-to-nists-assignments-under-the-executive-order-concerning-aiSource snippet
Response to the RFI Related to NIST's Assignments Under...14 Feb 2024 — In general, we recommend that NIST: Draw from the UK Government'...
-
Source: artificial-intelligence-act.com
Link: https://www.artificial-intelligence-act.com/Artificial_Intelligence_Act_Preamble_111_to_120.htmlSource snippet
lities, evaluated on the basis of appropriate technical...Read more...
-
Source: aisecurityandsafety.org
Title: frontier ai safety commitments
Link: https://aisecurityandsafety.org/it/frameworks/frontier-ai-safety-commitments/Source snippet
AI Security & Safety DirectoryFrontier AI Safety CommitmentsMar 10, 2026 — Voluntary safety pledges by 16 leading AI companies to assess...
-
Source: aisi.gov.uk
Link: https://www.aisi.gov.uk/frontier-ai-trends-reportSource snippet
AI Security InstituteFrontier AI Trends Report by The AI Security Institute (AISI)The UK AI Security Institute (AISI) has conducted evalu...
-
Source: artificialintelligenceact.eu
Link: https://artificialintelligenceact.eu/article/51/Source snippet
If the AI model has high impact capabilities, determined by technical tools and...Read more...
-
Source: digital-strategy.ec.europa.eu
Title: general purpose ai models ai act questions answers
Link: https://digital-strategy.ec.europa.eu/en/faqs/general-purpose-ai-models-ai-act-questions-answersSource snippet
state-of-the-art) models at any given point in time, or from other...Read more...
-
Source: digital-strategy.ec.europa.eu
Title: Digital Strategy AI Act | Shaping Europe’s digital future
Link: https://digital-strategy.ec.europa.eu/en/policies/regulatory-framework-aiSource snippet
The AI Act is part of a wider package of policy measures...Read more...
-
Source: digital-strategy.ec.europa.eu
Title: contents code gpai
Link: https://digital-strategy.ec.europa.eu/en/policies/contents-code-gpaiSource snippet
Digital StrategyThe General-Purpose AI Code of PracticeJul 10, 2025 — The code of practice helps industry comply with the AI Act legal ob...
-
Source: federalregister.gov
Title: safe secure and trustworthy development and use of artificial intelligence
Link: https://www.federalregister.gov/documents/2023/11/01/2023-24283/safe-secure-and-trustworthy-development-and-use-of-artificial-intelligenceSource snippet
Federal RegisterSafe, Secure, and Trustworthy Development and Use of...1 Nov 2023 — Executive Order 14110 of October 30, 2023. Safe, Sec...
Published: October 30, 2023
-
Source: longtermwiki.com
Link: https://www.longtermwiki.com/wiki/E366Source snippet
US Executive Order on Safe, Secure, and Trustworthy AIIt mandated safety evaluations for frontier AI models, created reporting requiremen...
-
Source: iaps.ai
Link: https://www.iaps.ai/s/Evaluation-Awareness_-Why-Frontier-AI-Models-Are-Getting-Harder-to-Test-l2fa.pdfSource snippet
In some cases, this [awareness]({{ 'awareness/' | relative_url }})...
-
Source: ai-frontiers.org
Link: https://ai-frontiers.org/articles/how-the-eus-code-of-practice-advances-ai-safetySource snippet
How the EU's Code of Practice Advances AI SafetyJul 12, 2025 — The AI Act, for which the Code provides compliance guidance, only regulate...
-
Source: freshfields.com
Link: https://www.freshfields.com/en/our-thinking/campaigns/tech-data-and-ai-the-digital-frontier/eu-digital-strategy/artificial-intelligence-act
Additional References
-
Source: seekmaro.com
Link: https://seekmaro.com/blog/ai-governance-regulationsSource snippet
AI Governance Regulations in the US: What You Need to Know2 days ago — NIST AI RMF alignment is emerging as the safe harbor mechanism acr...
-
Source: axios.com
Link: https://www.axios.com/2023/10/30/ai-executive-order-biden-transparency-safetySource snippet
The order mandates that companies developing high-risk AI models must notify the government during training and share results from safety...
-
Source: linkedin.com
Link: https://www.linkedin.com/posts/beijing-aisi_foresightsafety-bench-a-frontier-risk-evaluation-activity-7433110487237701632-o1ORSource snippet
Evaluating Frontier AI Safety with ForesightSafety BenchStart with the risk assessment using NIST AI RMF. Layer the impact assessment und...
-
Source: bundesnetzagentur.de
Link: https://www.bundesnetzagentur.de/EN/Areas/Digitalisation/AI/03_Models/start.htmlSource snippet
General-purpose AI modelsThe AI Act makes a distinction between “normal” GPAI models and GPAI models with systemic risk. “Normal” GPAI mo...
-
Source: aigi.ox.ac.uk
Link: https://aigi.ox.ac.uk/wp-content/uploads/2025/10/Post-convening-memo_-Safety-Frameworks-and-Standards_-A-comparative-analysis-to-advance-risk-management-of-frontier-AI_09.10.2025.pdfSource snippet
Frameworks and Standards: A comparative analysis to...by M Ziosi · 2025 — This document systematically compares Frontier Safety Framewor...
-
Source: wsj.com
Link: https://www.wsj.com/tech/ai/ai-act-passes-european-union-law-regulation-e04ec251Source snippet
The law, which is set to gradually take effect, bans certain AI applications, introduces transparency requirements, and mandates risk ass...
-
Source: linkedin.com
Link: https://www.linkedin.com/posts/metr-evals_common-elements-of-frontier-safety-policies-activity-7311933033417216000-ChkDSource snippet
Common Elements of Frontier Safety Policies, March 2025Frontier Safety Policies (FSPs) are emerging as a common framework for managing ri...
-
Source: reuters.com
Link: https://www.reuters.com/sustainability/boards-policy-regulation/ai-models-with-systemic-risks-given-pointers-how-comply-with-eu-ai-rules-2025-07-18/Source snippet
These guidelines aim to ease the regulatory burden for businesses and provide clarity for complying with the law, which comes into effect...
-
Source: presidency.ucsb.edu
Title: executive order 14141 advancing united states leadership artificial intelligence
Link: https://www.presidency.ucsb.edu/documents/executive-order-14141-advancing-united-states-leadership-artificial-intelligenceSource snippet
Order 14141—Advancing United States Leadership...14 Jan 2025 — This order sets our Nation on the path to ensure that future frontier AI...
-
Source: ourtake.bakerbotts.com
Title: what kind of person is your ai model character and the new alignment ecosystem
Link: https://ourtake.bakerbotts.com/post/102mi5s/what-kind-of-person-is-your-ai-model-character-and-the-new-alignment-ecosystemSource snippet
Model Character and the New...A third lab's frontier safety framework organizes mitigations around Critical Capability Levels and focuse...
Topic Tree