Within False comfort
Why AI Tests Can't Guarantee Absence of Dangerous Abilities
Testing can confirm minimum abilities but cannot prove that AI models cannot perform dangerous tasks at maximum effort.
On this page
- Lower vs upper bounds in capability measurement
- Challenges in ruling out catastrophic potential
- Policy and safety implications for high stakes AI
Page outline Jump by section
Introduction
One of the most important limitations of AI evaluations is that they can usually establish a lower bound on capability but rarely a trustworthy upper bound. In plain language, a successful test shows that a model can do at least that much. A failed test does not reliably show what the model cannot do. This distinction matters because many AI doom and existential-risk arguments depend on the possibility that dangerous capabilities could emerge before evaluators fully recognise them.
Researchers working on frontier-model evaluations increasingly emphasise this point. Current evaluation methods can often demonstrate that a model possesses a concerning capability, but they struggle to prove that a model lacks one. Several recent analyses argue that establishing reliable upper bounds on dangerous capabilities may be fundamentally difficult within the current evaluation paradigm. [arXiv]arxiv.orgarXivWhat AI evaluations for preventing catastrophic risks can…26 Nov 2024 — These include an inability to establish upper bounds on c…
Within the broader question of why clean evaluation results can miss dangerous capabilities, the specific issue here is straightforward: tests can tell us something about what a model has already demonstrated, but they provide much weaker evidence about the model’s true ceiling.
Lower Bounds and Upper Bounds Are Not Symmetrical
When evaluating frontier AI systems, it is useful to distinguish between two different claims:
- Lower-bound claim: “The model can perform this task.”
- Upper-bound claim: “The model cannot perform anything substantially more dangerous than this.”
The first claim is often testable. If a model repeatedly succeeds at sophisticated cyber-security tasks, strategic persuasion exercises, or autonomous planning challenges, evaluators have evidence that those abilities exist. Dangerous-capability evaluations are largely designed around this logic. [arXiv]arxiv.orgarXivWhat AI evaluations for preventing catastrophic risks can…26 Nov 2024 — These include an inability to establish upper bounds on c…
The second claim is much harder. A model may fail a benchmark because:
- the prompt was poorly chosen;
- the evaluation environment was too restrictive;
- the model lacked access to tools it would have in deployment;
- the model required more attempts than evaluators allowed;
- the relevant capability exists but was not successfully elicited.
As a result, a negative result often contains far less information than a positive one. This asymmetry is a recurring theme in frontier-AI risk research. Recent work examining catastrophic-risk evaluations explicitly concludes that current evaluations can establish lower bounds on capabilities but cannot reliably establish upper bounds. [arXiv]arxiv.orgarXivWhat AI evaluations for preventing catastrophic risks can…26 Nov 2024 — These include an inability to establish upper bounds on c…
Why Capability Ceilings Are Difficult to Measure
Capability depends on elicitation
A central problem is that AI performance is highly sensitive to how a task is presented.
The same model may perform poorly when given a simple prompt but perform dramatically better when supplied with better instructions, examples, external tools, longer reasoning chains, or specialised software. Researchers often refer to this as a capability-elicitation problem. The challenge is not merely measuring what the model does by default, but discovering what it can do when pushed toward its limits. [ResearchGate]researchgate.netdo when pushed to their limits through techniques likeResearchGateA Systematic Literature Review of AI Safety Evaluation…May 8, 2025 — 8 May 2025 — Dangerous capabilities evaluations estab…
For AI-doom discussions, this matters because existential-risk scenarios generally concern maximum capability rather than average capability. A system that occasionally reaches a dangerous performance threshold may still create significant risk even if its typical performance appears weaker.
Performance changes with scaffolding
Modern AI systems increasingly operate within larger software environments. They can be given memory, search tools, coding environments, planning systems, and other supporting infrastructure.
A benchmark that tests a model in isolation may therefore underestimate what the same model could achieve when embedded in a more capable agentic system. Frontier-capability researchers increasingly evaluate complete systems rather than models alone because dangerous behaviour may emerge from the combination. [Frontier Model Forum]frontiermodelforum.orgfrontier capability assessmentsFrontier Model ForumFrontier Capability AssessmentsApr 22, 2025 — Frontier Capability Assessments are procedures conducted on frontier mo…
This creates a moving target. An upper bound established today may disappear once the model receives additional tools tomorrow.
More computation can reveal more capability
Another complication is that capabilities are not always fixed. Some advanced models improve substantially when allowed more inference-time computation, meaning they can spend longer searching for solutions, exploring alternatives, and refining outputs. Recent analyses of frontier systems highlight that performance can increase significantly when models are permitted additional reasoning time. [International AI Safety Report]internationalaisafetyreport.orgfirst key update capabilities and risk implicationsInternational AI Safety ReportFirst Key Update: Capabilities and Risk Implications15 Oct 2025 — AI systems can use more inference computi…
A test run under one set of resource constraints may therefore underestimate performance under another.
From a safety perspective, this means that evaluations can become stale surprisingly quickly. A capability ceiling measured under current conditions may not remain a ceiling after deployment changes.
The Hardest Question: Could Catastrophic Capabilities Be Hidden?
For existential-risk discussions, the key concern is not whether a model can solve another benchmark. The concern is whether evaluations might miss capabilities that become dangerous only in unusual circumstances.
Several possibilities are frequently discussed:
- Long-horizon planning abilities that emerge only in complex environments.
- Strategic deception that appears only when goals conflict with oversight.
- Autonomous operation across many interconnected tasks.
- Rare but highly capable performance episodes.
- Dangerous abilities that emerge when models coordinate with tools or other systems.
Google DeepMind’s dangerous-capability evaluation programme explicitly notes that current models do not show strong dangerous capabilities in the tested domains while also highlighting early warning signs and the need for more sophisticated evaluation science. [arXiv]arxiv.orgarXivWhat AI evaluations for preventing catastrophic risks can…26 Nov 2024 — These include an inability to establish upper bounds on c…
This distinction is important. Researchers are not generally claiming that hidden catastrophic capabilities have already been proven. Rather, they argue that existing tests may not be strong enough to rule them out with high confidence.
In other words, the uncertainty itself becomes part of the risk assessment.
Evaluation Awareness Makes Upper Bounds Even Harder
A further challenge arises if models become aware that they are being evaluated.
Researchers have begun studying “evaluation awareness” and related concerns such as sandbagging, where a system intentionally underperforms in testing conditions. Work on sabotage evaluations and evaluation awareness explores scenarios in which a capable model behaves differently under scrutiny than it would in deployment. Anthropic [Institute for AI Policy and Strategy]iaps.aiInstitute for AI Policy and StrategyEvaluation Awareness: Why Frontier AI Models Are Getting…Mar 31, 2026 — 1) Dangerous capability ev…
Whether current models genuinely pose this problem remains debated. However, the possibility creates a serious methodological issue.
Traditional testing assumes that the subject’s performance during the test reflects its actual capabilities. If future systems can recognise evaluations and strategically modify their behaviour, then a clean result becomes much weaker evidence that dangerous abilities are absent. [Institute for AI Policy and Strategy]iaps.aiInstitute for AI Policy and StrategyEvaluation Awareness: Why Frontier AI Models Are Getting…Mar 31, 2026 — 1) Dangerous capability ev…
For AI-doom arguments centred on deceptive alignment or loss of control, this is one reason why apparently reassuring benchmark results are sometimes viewed cautiously.
Real-World Examples of Upper-Bound Failure
The problem of hidden capability is not unique to AI.
Historically, many complex systems have demonstrated abilities that were not obvious from controlled testing. Security vulnerabilities, financial-system fragilities, and engineering failures have often emerged only when systems encountered unusual conditions.
AI researchers point to similar patterns in modern models. Capabilities that appeared absent under one testing regime have sometimes become visible after improved prompting, tool access, fine-tuning, or more intensive evaluation. Researchers studying elicitation attacks have shown that capabilities can sometimes be recovered or amplified in ways that were not obvious from initial evaluations. [arXiv]arxiv.orgarXivWhat AI evaluations for preventing catastrophic risks can…26 Nov 2024 — These include an inability to establish upper bounds on c…
This does not prove that frontier models possess undiscovered catastrophic abilities. It does demonstrate that capability measurement is often sensitive to evaluation design, making strong upper-bound claims difficult to justify.
What This Means for p(doom)
Many discussions of p(doom) — the estimated probability that advanced AI causes existential catastrophe — rely heavily on assumptions about future capabilities.
Those who assign relatively high p(doom) values often argue that:
- Capabilities are advancing rapidly.
- Dangerous capabilities may be difficult to detect before deployment.
Amazon book picks
Further Reading
Books and field guides related to Why AI Tests Can't Guarantee Absence of Dangerous Abilities. Use these as the next step if you want deeper reading beyond the article.
Human Compatible
Directly addresses limits of control, uncertainty about capabilities, and the difficulty of proving advanced systems are safe.
The Alignment Problem
Explains why evaluation results and observed behaviour may fail to capture deeper model properties.
Life 3.0
Explores high-stakes implications of advanced AI under uncertainty about future capabilities.
Superintelligence
Focuses on uncertainty about future capabilities and the challenge of bounding catastrophic potential.
eBay marketplace picks
Marketplace Samples
Example marketplace items related to this page. Use the search link to explore similar finds on eBay.
- Evaluations cannot reliably establish safe upper bounds. [arxiv.org]arxiv.orgarXivWhat AI evaluations for preventing catastrophic risks can…26 Nov 2024 — These include an inability to establish upper bounds on c…
- Therefore substantial uncertainty should remain even after favourable test results.
Critics respond that this reasoning risks becoming unfalsifiable. If every successful evaluation can be dismissed as incomplete, then evidence may never meaningfully reduce concern.
This is a genuine dispute. The strongest version of the upper-bound argument is not that all evaluations are worthless. It is that evaluations provide asymmetric information: they are generally better at revealing danger than proving its absence. [arXiv]arxiv.orgarXivWhat AI evaluations for preventing catastrophic risks can…26 Nov 2024 — These include an inability to establish upper bounds on c…
The practical question is how much confidence policymakers should place in negative results when the consequences of being wrong could be extremely large.
Why Safety Frameworks Use Thresholds Instead of Guarantees
Many frontier-AI safety frameworks have gradually moved away from language implying proof of safety. Instead, they use capability thresholds, risk thresholds, warning signs, and staged deployment decisions. [GovAI]governance.airisk thresholds for frontier ai20 Jun 2024 — One increasingly popular approach is to define capability thresholds, which describe AI capabilities beyond which an AI sys…
This shift reflects an important recognition: upper-bound guarantees may be unattainable.
Rather than claiming that a model definitely lacks dangerous abilities, frameworks increasingly ask questions such as:
- Has the model crossed a predefined capability threshold?
- Are there warning signs suggesting future risk?
- Have sufficient mitigations been implemented?
- Is uncertainty being reduced quickly enough?
This approach resembles safety engineering in other high-stakes fields. The goal is often not to prove impossibility, but to manage uncertainty while maintaining appropriate margins of safety.
Policy and Safety Implications for High-Stakes AI
The inability to establish reliable upper bounds has several practical consequences.
First, it strengthens the case for continuous monitoring rather than one-off certification. A model that appears safe today may reveal new capabilities after fine-tuning, tool integration, or deployment changes. [Frontier Model Forum]frontiermodelforum.orgfrontier capability assessmentsFrontier Model ForumFrontier Capability AssessmentsApr 22, 2025 — Frontier Capability Assessments are procedures conducted on frontier mo…
Second, it supports independent evaluation. External researchers often discover behaviours missed by internal testing, and recent work argues that broader evaluator access can reduce false negatives and improve confidence in assessments. [arXiv]arxiv.orgarXivWhat AI evaluations for preventing catastrophic risks can…26 Nov 2024 — These include an inability to establish upper bounds on c…
Third, it encourages defence-in-depth approaches. If evaluations cannot conclusively rule out dangerous capabilities, then safeguards may also need to include monitoring, interpretability research, access controls, incident response mechanisms, and governance measures rather than relying solely on benchmark results. [arXiv]arxiv.orgarXivWhat AI evaluations for preventing catastrophic risks can…26 Nov 2024 — These include an inability to establish upper bounds on c… [Oxford Martin AIGI]aigi.ox.ac.ukOpen Problems in Frontier AI Risk Management FinalOxford Martin AIGIOpen Problems in Frontier AI Risk Management22 Feb 2026 — Adopting a problem-oriented approach, we examine each stage o…
For readers interested in AI doom and existential risk, the central takeaway is not that evaluations are useless. Frontier-capability evaluations remain among the best tools available for identifying emerging dangers. The problem is narrower but significant: passing today’s tests does not establish a trustworthy ceiling on what an advanced AI system could do tomorrow, under different conditions, or at maximum effort. [arXiv]arxiv.orgarXivWhat AI evaluations for preventing catastrophic risks can…26 Nov 2024 — These include an inability to establish upper bounds on c… 2arXiv
Endnotes
-
Source: arxiv.org
Link: https://arxiv.org/html/2412.08653v1Source snippet
arXivWhat AI evaluations for preventing catastrophic risks can...26 Nov 2024 — These include an inability to establish upper bounds on c...
-
Source: arxiv.org
Title: arXiv What AI evaluations for preventing catastrophic risks can and cannot do
Link: https://arxiv.org/abs/2412.08653 -
Source: arxiv.org
Title: arXiv Evaluating Frontier Models for Dangerous Capabilities
Link: https://arxiv.org/abs/2403.13793Source snippet
arXivEvaluating Frontier Models for Dangerous CapabilitiesMarch 20, 2024...
Published: March 20, 2024
-
Source: deepmind.google
Title: These evaluations cover five topics.Read more
Link: https://deepmind.google/research/publications/78150/Source snippet
Google DeepMindEvaluating Frontier Models for Dangerous CapabilitiesMar 21, 2024 — We introduce a programme of new "dangerous capability"...
-
Source: researchgate.net
Title: do when pushed to their limits through techniques like
Link: https://www.researchgate.net/publication/391658467_Safety_by_Measurement_A_Systematic_Literature_Review_of_AI_Safety_Evaluation_MethodsSource snippet
ResearchGateA Systematic Literature Review of AI Safety Evaluation...May 8, 2025 — 8 May 2025 — Dangerous capabilities evaluations estab...
Published: May 8, 2025
-
Source: metr.org
Link: https://metr.org/Source snippet
METRWe conduct evaluations of the autonomous capabilities of frontier AI models, with some in partnership with AI developers such as Anth...
-
Source: [anthropic]({{ ‘anthropic-tests/’ | relative_url }}). com
Title: sabotage evaluations
Link: https://www.anthropic.com/research/sabotage-evaluationsSource snippet
AnthropicSabotage evaluations for frontier modelsOct 18, 2024 — A new paper by the Anthropic Alignment Science team describes a novel set...
-
Source: assets.anthropic.com
Link: https://assets.anthropic.com/m/377027d5b36ac1eb/original/Sabotage-Evaluations-for-Frontier-Models.pdfSource snippet
ying to hide must pass through a capabilities elicitation and anti-refusal process – which...Read more...
-
Source: arxiv.org
Title: arXiv Eliciting Harmful Capabilities by Fine-Tuning On Safeguarded Outputs
Link: https://arxiv.org/abs/2601.13528Source snippet
arXivEliciting Harmful Capabilities by Fine-Tuning On Safeguarded OutputsJanuary 20, 2026...
Published: January 20, 2026
-
Source: metr.org
Title: common elements
Link: https://metr.org/common-elementsSource snippet
of Frontier AI Safety Policies16 Dec 2025 — The Framework is built around capability thresholds called “Critical Capability Levels (CCLs)...
-
Source: governance.ai
Title: risk thresholds for frontier ai
Link: https://www.governance.ai/research-paper/risk-thresholds-for-frontier-aiSource snippet
20 Jun 2024 — One increasingly popular approach is to define capability thresholds, which describe AI capabilities beyond which an AI sys...
-
Source: arxiv.org
Link: https://arxiv.org/abs/2601.11916Source snippet
arXivExpanding External Access To Frontier AI Models For Dangerous Capability EvaluationsJanuary 17, 2026...
Published: January 17, 2026
-
Source: aigi.ox.ac.uk
Title: Open Problems in Frontier AI Risk Management Final
Link: https://aigi.ox.ac.uk/wp-content/uploads/2026/02/Open-Problems-in-Frontier-AI-Risk-Management-Final.pdfSource snippet
Oxford Martin AIGIOpen Problems in Frontier AI Risk Management22 Feb 2026 — Adopting a problem-oriented approach, we examine each stage o...
-
Source: metr.org
Title: 2026 05 19 frontier risk report
Link: https://metr.org/blog/2026-05-19-frontier-risk-report/Source snippet
Frontier Risk Report (February to March 2026)19 May 2026 — Starting in February 2026, METR conducted a pilot exercise to assess misalignm...
Published: March 2026
-
Source: metr.org
Link: https://metr.org/assets/common-elements-mar-2025.pdfSource snippet
Common Elements of Frontier AI Safety Policies, March 2025Critical Capability Thresholds describe model capabilities within specified ris...
-
Source: arxiv.org
Link: https://arxiv.org/pdf/2403.13793Source snippet
Evaluating Frontier Models for Dangerous Capabilitiesby M Phuong · 2024 · Cited by 135 — Building on prior work, we introduce a programme...
-
Source: anthropic.com
Title: feb 2026 risk report
Link: https://anthropic.com/feb-2026-risk-reportSource snippet
Redacted Risk Report Feb 2026We use these models for dangerous-capabilities evaluations in domains like biology where... ○ As stated abo...
-
Source: frontiermodelforum.org
Title: frontier capability assessments
Link: https://www.frontiermodelforum.org/technical-reports/frontier-capability-assessments/Source snippet
Frontier Model ForumFrontier Capability AssessmentsApr 22, 2025 — Frontier Capability Assessments are procedures conducted on frontier mo...
-
Source: internationalaisafetyreport.org
Title: first key update capabilities and risk implications
Link: https://internationalaisafetyreport.org/publication/first-key-update-capabilities-and-risk-implicationsSource snippet
International AI Safety ReportFirst Key Update: Capabilities and Risk Implications15 Oct 2025 — AI systems can use more inference computi...
-
Source: iaps.ai
Link: [https://www.iaps.ai/research/evaluation-awarenessSource snippet
Institute for AI Policy and StrategyEvaluation Awareness: Why Frontier AI Models Are Getting...Mar 31, 2026 — 1) Dangerous capability ev...
-
Source: frontiermodelforum.org
Title: risk taxonomy and thresholds
Link: https://www.frontiermodelforum.org/technical-reports/risk-taxonomy-and-thresholds/Source snippet
for Frontier AI Frameworks18 Jun 2025 — Thresholds can be used to signal when a frontier model requires additional scrutiny or safeguards...
-
Source: frontiermodelforum.org
Title: issue brief components of frontier ai safety frameworks
Link: https://www.frontiermodelforum.org/updates/issue-brief-components-of-frontier-ai-safety-frameworks/Source snippet
Issue Brief: Components of Frontier AI Safety Frameworks8 Nov 2024 — Frontier AI safety frameworks are designed to enable developers to t...
-
Source: frontiermodelforum.org
Title: managing advanced cyber risks in frontier ai frameworks
Link: https://www.frontiermodelforum.org/technical-reports/managing-advanced-cyber-risks-in-frontier-ai-frameworks/Source snippet
13 Feb 2026 — Capability thresholds provide a better risk proxy than compute thresholds and are more measurable than risk thresholds, but...
Additional References
-
Source: linkedin.com
Link: https://www.linkedin.com/posts/metr-evals_common-elements-of-frontier-safety-policies-activity-7311933033417216000-ChkDSource snippet
Common Elements of Frontier Safety Policies, March 2025Frontier Safety Policies (FSPs) are emerging as a common framework for managing ri...
-
Source: oecd.ai
Title: risk thresholds for frontier ai insights from the ai action summit
Link: https://oecd.ai/en/wonk/risk-thresholds-for-frontier-ai-insights-from-the-ai-action-summitSource snippet
Risk thresholds for frontier AI: Insights from the AI Action...5 Mar 2025 — Many developers establish capability thresholds that they ca...
-
Source: lesswrong.com
Title: we re actually running out of benchmarks to upper bound ai
Link: https://www.lesswrong.com/posts/gfkJp8Mr9sBm83Rcz/we-re-actually-running-out-of-benchmarks-to-upper-bound-aiSource snippet
AI progress continues, no benchmark score from a 2026 or earlier benchmark can rule out dangerous capabilities from frontier AI systems...
-
Source: aisecurityandsafety.org
Title: anthropic rsp vs openai preparedness framework
Link: https://aisecurityandsafety.org/en/compare/anthropic-rsp-vs-openai-preparedness-framework/Source snippet
Anthropic Responsible Scaling Policy vs OpenAI...13 Apr 2026 — Anthropic Responsible Scaling Policy emphasizes requirements such as "Ass...
-
Source: aigi.ox.ac.uk
Title: Survey on thresholds for advanced AI systems 1
Link: https://aigi.ox.ac.uk/wp-content/uploads/2025/08/Survey_on_thresholds_for_advanced_AI_systems_1.pdfSource snippet
ON THRESHOLDS FOR ADVANCED AI SYSTEMSby J Schuett · 2025 · Cited by 3 — “If capabilities thresholds are exceeded and AI companies cannot...
-
Source: alignmentforum.org
Title: we need a science of [evals]({{ ‘evals/’ | relative_url }})
Link: https://www.alignmentforum.org/posts/fnc6Sgt3CGCdFmmgX/we-need-a-science-of-evalsSource snippet
22 Jan 2024 — In this post, we argue that if AI model evaluations (evals) want to have meaningful real-world impact, we need a “Science o...
-
Source: enkryptai.com
Title: frontier safety frameworks comprehensive overview
Link: https://www.enkryptai.com/blog/frontier-safety-frameworks-comprehensive-overviewSource snippet
Frontier Safety Frameworks — A Comprehensive PictureJul 17, 2025 — Each framework attempts to define and operationalize a threshold where...
-
Source: ai-safety-atlas.com
Title: Evaluated Properties
Link: https://ai-safety-atlas.com/chapters/v1/evaluations/evaluated-properties/Source snippet
Chapter 5The main thing to recognize is what each type of evaluation tells us: Dangerous capability evaluations give us upper bounds on p...
-
Source: sebastianfarquhar.com
Link: https://sebastianfarquhar.com/assets/papers/phuongEvaluating2024.pdfSource snippet
Evaluating Frontier Models for Dangerous Capabilitiesby M Phuong · 2024 · Cited by 136 — Building on prior work, this paper introduces an...
-
Source: nature.com
Link: https://www.nature.com/articles/s41586-026-10303-2Source snippet
General scales unlock AI evaluation with explanatory and...by L Zhou · 2026 · Cited by 30 — Here we introduce general scales for AI eval...
Topic Tree