Within State Rivalry
Why Hidden AI Progress Makes Caution Look Dangerous
When governments cannot see rivals' models or test results, they may assume the worst and cut safety time to avoid being surprised.
On this page
- What governments cannot reliably observe
- How uncertainty turns caution into strategic risk
- Which transparency measures could reduce worst case assumptions
Page outline Jump by section
Introduction
A central concern in AI doom debates is not only that countries compete, but that they compete while knowing remarkably little about one another’s most advanced systems. Governments rarely have direct access to rivals’ frontier AI models, internal safety evaluations, or unpublished research. Instead, they must infer capabilities from public demonstrations, intelligence reports, commercial announcements and fragmentary technical evidence. When the consequences of being wrong appear extremely large, decision-makers may assume the worst.
Supporters of this concern argue that secrecy can transform ordinary uncertainty into a race dynamic. If one state cannot tell whether a rival is months or years ahead, slowing development to conduct more safety testing may begin to look strategically dangerous rather than responsible. Critics accept that secrecy complicates policy but question whether AI progress is sufficiently opaque or decisive to justify these fears. The debate therefore centres on how hidden capabilities influence incentives, not on whether secrecy itself is unusual.
What governments cannot reliably observe
Unlike nuclear weapons programmes, frontier AI development leaves relatively few unmistakable external signals. Large computing facilities can sometimes be detected, but they reveal little about what a model can actually do, how reliable it is, or whether it has passed demanding safety evaluations.
Several kinds of information usually remain hidden:
- Internal capability evaluations that never become public.
- Safety failures discovered during development.
- Experimental models that are abandoned before release.
- Classified government AI projects.
- Commercial research protected for competitive reasons.
- The amount and quality of specialised computing resources available.
- New training techniques that dramatically improve performance without requiring proportionally larger hardware.
This matters because modern AI capabilities often emerge through combinations of algorithms, data, engineering and compute rather than any single observable breakthrough. External observers may know that another country is investing heavily in AI without knowing whether it has achieved capabilities that would substantially alter military or economic competition. UK government assessments likewise note that frontier AI development involves major uncertainties, limited visibility and incomplete metrics for measuring dangerous capabilities, making oversight inherently difficult.[GOV.UK]GOV.UK28, 2025…
For AI doom advocates, the problem is not merely missing information but asymmetric information. Developers know much more about their own systems than competitors do. Rivals therefore have incentives to prepare for possibilities rather than confirmed realities.
How uncertainty turns caution into strategic risk
The mechanism resembles the classic security dilemma from international relations. One government’s attempt to improve its own security unintentionally makes others feel less secure because intentions cannot be directly observed.
Applied to frontier AI, the mechanism works in several stages.
First, governments believe advanced AI could eventually provide significant strategic advantages in intelligence analysis, cyber operations, scientific research or military planning.
Second, they cannot reliably determine how capable competitors’ unpublished systems actually are.
Third, because underestimating a rival could prove extremely costly, they increasingly plan around worst-case assumptions instead of average expectations.
Finally, those assumptions create pressure to shorten development timelines, accelerate deployment and reduce delays for extensive external evaluation.
Importantly, none of these steps requires malicious intent. Even leaders who privately favour stronger safety practices may conclude that unilateral caution creates unacceptable strategic vulnerability if they suspect competitors are moving faster.
Researchers studying frontier AI governance have argued that this uncertainty creates incentives for governments to prioritise speed unless mechanisms exist for verifying that competitors are exercising similar restraint. Proposed regulatory approaches such as reporting requirements, capability evaluations and supervisory oversight are partly intended to reduce this information gap.[arXiv]arxiv.orgarXiv Frontier AI Regulation: Managing Emerging Risks to Public SafetyFrontier AI Regulation: Managing Emerging Risks to Public SafetyJuly 6, 2023…
Why worst-case assumptions can become self-reinforcing
The most concerning feature of secrecy is that it can produce feedback loops rather than isolated mistakes.
Suppose Country A privately believes Country B may possess substantially more capable AI than public evidence suggests. Country A accelerates development to avoid surprise. Country B observes this acceleration but cannot determine whether it reflects genuine breakthroughs or excessive caution. It then speeds up as well.
Neither side needs reliable evidence that the other has actually achieved a decisive advantage. The expectation that the other side might be ahead becomes enough to alter behaviour.
Economists sometimes describe similar situations as coordination failures or prisoner’s dilemma problems. Each participant may prefer a world with greater transparency and more extensive safety testing, yet each also fears becoming the only participant to slow down. Recent discussions among frontier AI developers have increasingly framed competitive pressure in these terms, although the analogy remains contested.[axios.com]axios.comAI labs are grappling with a "prisoner's dilemma"—where all recognize the need for a development slowdown due to safety concerns, but no…
Within AI doom arguments, this mechanism matters because alignment research, interpretability work and dangerous-capability evaluations all require time. If geopolitical uncertainty consistently compresses development schedules, opportunities to discover serious problems before deployment may shrink.
Why AI secrecy differs from previous arms races
Historical comparisons with nuclear weapons are common but imperfect.
Nuclear arsenals eventually became partly observable through satellite imagery, missile testing, inspections and treaties. Although uncertainty remained, many important capabilities produced visible physical evidence.
Frontier AI is different in several respects.
- Capabilities are software-based. A model can improve dramatically without producing obvious external indicators.
- Commercial firms play a leading role. Governments may know even less about private laboratories abroad than about foreign military programmes.
- Progress can appear discontinuous. New techniques sometimes produce unexpectedly large improvements that outsiders could not easily predict.
- Many advances remain proprietary. Competitive commercial incentives reinforce national-security secrecy.
These features increase the possibility that governments rely on intelligence estimates with wide confidence intervals rather than direct observation. The result is not necessarily systematic overestimation, but greater uncertainty about where competitors actually stand.
Can transparency reduce worst-case assumptions?
Many AI governance proposals attempt to reduce uncertainty without requiring countries to reveal commercially valuable or militarily sensitive information.
Suggested approaches include:
- Shared reporting requirements for very large training runs.
- Independent third-party evaluations of frontier models.
- Secure mechanisms allowing trusted government or accredited experts to inspect systems without disclosing proprietary details.
- Common evaluation standards for dangerous capabilities.
- Incident reporting when unexpected behaviours are discovered.
- International dialogue on capability thresholds that warrant additional scrutiny.
The challenge is balancing transparency against legitimate security concerns. Revealing too much about powerful models could itself increase misuse risks or expose commercially valuable research. Consequently, many proposals favour selective transparency—providing trusted evaluators with confidential access rather than publishing every technical detail.
Recent work on secure third-party evaluation argues that safety-critical assessment should not be blocked by commercial confidentiality or national-security concerns, while also recognising that access must be carefully controlled. Government reports similarly identify transparency throughout development and deployment as an important governance tool, although they note that its effectiveness declines if only a few jurisdictions participate.[Royal United Services Institute]rusi.orgRoyal United Services InstituteDeveloping a Framework for Secure Third-Party Access to Frontier AI | Royal United Services InstituteMay 1…
How strong is the evidence for this concern?
Evidence for the mechanism is mixed because the most relevant decisions occur inside governments and frontier AI developers, where much information remains confidential.
There is strong evidence for several background facts:
- Frontier AI research is concentrated in organisations that keep substantial capability information private.
- Governments increasingly regard advanced AI as strategically important.
- Existing safety evaluations cannot perfectly predict future dangerous capabilities.
- Policymakers face genuine uncertainty about rivals’ progress.[arxiv.org]arxiv.orgarXiv Evaluating Frontier Models for Dangerous CapabilitiesEvaluating Frontier Models for Dangerous CapabilitiesMarch 20, 2024…
The larger inferential leap concerns behaviour. It is much harder to demonstrate that secrecy will actually cause governments to reduce safety precautions enough to increase existential risk. That claim depends on future political decisions, perceptions of strategic advantage and the pace of AI progress, all of which remain uncertain.
Critics also argue that states possess extensive intelligence capabilities and may have better information than public observers assume. Others contend that countries often exaggerate rivals’ technological progress for domestic political reasons, meaning worst-case racing could arise from political incentives as much as genuine informational gaps.
For AI doom advocates, however, the precise probability is less important than the structure of the problem. If hidden capabilities consistently encourage governments to assume competitors are further ahead than they can verify, then secrecy itself becomes part of the mechanism by which strategic competition can make caution appear increasingly risky.
Amazon book picks
Further Reading
Books and field guides related to Why Hidden AI Progress Makes Caution Look Dangerous. Use these as the next step if you want deeper reading beyond the article.
The Age of A. I.
An A.I. that learned to play chess discovered moves that no human champion would have conceived of. Driverlesscars edge forward at red li...
Superintelligence
This profoundly ambitious and original book picks its way carefully through a vast tract of forbiddingly difficult intellectual terrain.
The Coming Wave
"We are approaching a critical threshold in the history of our species. Everything is about to change. Soon you will live surrounded by A...
The Secret World
'The most comprehensive narrative of intelligence compiled ... unrivalled' Max Hastings, Sunday Times 'Captivating, insightful and master...
eBay marketplace picks
Marketplace Samples
Live-tested eBay searches with available results related to this page.
Selected fromcybersecurity pin oneBay.co.uk.
Endnotes
1.
Source: GOV.UK
Link:https://www.gov.uk/government/publications/frontier-ai-capabilities-and-risks-discussion-paper/future-risks-of-frontier-ai-annex-a
Source snippet
28, 2025...
2.
Source: GOV.UK
Title: Frontier AI: capabilities and risks – discussion paper
Link:https://www.gov.uk/government/publications/frontier-ai-capabilities-and-risks-discussion-paper/frontier-ai-capabilities-and-risks-discussion-paper
3.
Source: arxiv.org
Title: arXiv Frontier AI Regulation: Managing Emerging Risks to Public Safety
Link:https://arxiv.org/abs/2307.03718
Source snippet
Frontier AI Regulation: Managing Emerging Risks to Public SafetyJuly 6, 2023...
Published: July 6, 2023
4.
Source: axios.com
Link:https://www.axios.com/2026/07/30/ai-safety-slowdown-anthropic-openai
Source snippet
AI labs are grappling with a "prisoner's dilemma"—where all recognize the need for a development slowdown due to safety concerns, but no...
5.
Source: arxiv.org
Title: arXiv Evaluating Frontier Models for Dangerous Capabilities
Link:https://arxiv.org/abs/2403.13793
Source snippet
Evaluating Frontier Models for Dangerous CapabilitiesMarch 20, 2024...
Published: March 20, 2024
6.
Source: aisi.gov.uk
Title: Frontier AI Trends Report by The AI Security Institute (AISI)
Link:https://www.aisi.gov.uk/frontier-ai-trends-report
7.
Source: evals.alignment.org
Title: 2026 05 19 frontier risk report
Link:https://evals.alignment.org/blog/2026-05-19-frontier-risk-report/
Source snippet
Risk Report (February to March 2026) - METRMay 19, 2026 — Frontier Risk Report (February to March 2026) DATE May 19, 2026 [Input: Join ou...
Published: May 19, 2026
8.
Source: GOV.UK
Title: international ai safety report 2025
Link:https://www.gov.uk/government/publications/international-ai-safety-report-2025/international-ai-safety-report-2025
9.
Source: GOV.UK
Title: www.gov.uk Emerging processes for frontier AI safety
Link:https://www.gov.uk/government/publications/emerging-processes-for-frontier-ai-safety/emerging-processes-for-frontier-ai-safety
10.
Source: evals.alignment.org
Link:https://evals.alignment.org/
11.
Source: rusi.org
Link:https://www.rusi.org/explore-our-research/publications/research-papers/developing-framework-secure-third-party-access-frontier-ai
Source snippet
Royal United Services InstituteDeveloping a Framework for Secure Third-Party Access to Frontier AI | Royal United Services InstituteMay 1...
Additional References
12.
Source: fas.org
Title: Rather than predicting a single future for AI, this report aims to h
Link:https://fas.org/publication/converging-risks/
Source snippet
Converging Risks: AI and the Future of Global SecurityMay 20, 2026 — It emerges from the interactions between increasingly capable tools...
Published: May 20, 2026
13.
Source: youtube.com
Title: Robert Trager on International AI Governance and Cybersecurity at AI Companies
Link:https://www.youtube.com/watch?v=1-3x9os9rr4
Source snippet
How Humanity Survives The US-China AI Arms Race...
14.
Source: youtube.com
Title: AI Extinction Risk: Superintelligence, AI Arms Race & Safety Controls
Link:https://www.youtube.com/watch?v=hAfPF-iCaWU
Source snippet
Jaan Tallinn on Existential Risk and [AI Race]({{ 'ai-race/' | relative_url }}) Dynamics...
15.
Source: youtube.com
Title: Jaan Tallinn on Existential Risk and AI Race Dynamics
Link:https://www.youtube.com/watch?v=6MaUZ4Hi7os
Source snippet
Escaping an Anti-Human Future: A Conversation with Tristan Harris...
16.
Source: youtube.com
Title: How Humanity Survives The US-China AI Arms Race
Link:https://www.youtube.com/watch?v=EGBfdqxavp0
Source snippet
AI Extinction Risk: Superintelligence, AI Arms Race & Safety Controls...
17.
Source: deepmind.google
Title: Evaluating Frontier Models for Dangerous Capabilities — Google Deep Mind
Link:https://deepmind.google/research/publications/78150/
18.
Source: itu.int
Title: the annual ai governance report 2025 steering the future of ai
Link:https://www.itu.int/epublications/en/publication/the-annual-ai-governance-report-2025-steering-the-future-of-ai
19.
Source: youtube.com
Title: Escaping an Anti-Human Future: A Conversation with Tristan Harris
Link:https://www.youtube.com/watch?v=90irsXaKxZA
20.
Source: ojs.aaai.org
Link:https://ojs.aaai.org/index.php/AAAI/article/view/35022