Within Proxy Failure
Can Better Benchmarks Hide Worse AI Progress?
Once benchmarks become targets, contamination, narrow tuning and strategic test-taking can make published gains overstate real capability.
On this page
- The difference between measuring capability and optimising a test
- How contamination and benchmark specific tuning distort results
- How evaluations could remain useful under stronger optimisation
Page outline Jump by section
Introduction
Better benchmark scores do not always mean AI systems are becoming more capable, reliable or safer. Once a benchmark becomes an important target for researchers, companies and investors, developers naturally optimise for it. That optimisation can improve genuine capability, but it can also produce misleading progress through benchmark-specific tuning, data contamination, memorisation and strategic test-taking. As a result, published scores may rise faster than real-world performance.
Within debates about AI doom and existential risk, this matters because safety arguments often depend on accurately measuring what increasingly capable systems can actually do. If evaluations systematically overstate progress, decision-makers may deploy systems that are less reliable than expected. Equally, if benchmarks fail to detect emerging deceptive or strategically adaptive behaviour, they may underestimate the capabilities that are most relevant to loss-of-control scenarios. The challenge is therefore not whether benchmarks are useful—they remain indispensable—but whether they continue to measure what researchers believe they are measuring.[OpenAI]OpenAIscaling laws for reward model overoptimizationOctober 19, 2022…
The difference between measuring capability and optimising a test
Benchmarks are designed to estimate general capability by sampling representative tasks. Early improvements often reflect genuine advances because systems are learning broadly useful skills that transfer across many problems.
Over time, however, successful benchmarks become incentives in their own right. Academic papers, product launches and public leaderboards reward higher scores, encouraging developers to optimise directly for the benchmark rather than for the underlying capability the benchmark was intended to represent.
This is another manifestation of Goodhart’s law. A benchmark initially acts as a measurement, but once it becomes a competitive target its ability to distinguish genuine improvement from optimisation weakens. The benchmark has not necessarily become worthless, but interpreting improvements becomes more difficult.
In AI safety discussions, this distinction matters because benchmark scores are frequently used as evidence that systems are approaching or surpassing important capability thresholds. If those scores partly reflect benchmark optimisation rather than broader competence, forecasts based solely on published evaluations become less reliable.[OpenAI]OpenAIscaling laws for reward model overoptimizationOctober 19, 2022…
How contamination and benchmark-specific tuning distort results
Several different mechanisms can inflate benchmark performance without producing equivalent real-world gains.
Training-data contamination. Public benchmarks often circulate widely online. If benchmark questions, answers or close variants appear in training data, models may reproduce memorised information instead of demonstrating reasoning. Modern language models are trained on enormous internet-scale datasets, making accidental contamination increasingly difficult to rule out. Surveys of benchmark contamination conclude that this has become one of the central challenges in evaluating large language models.[arXiv]arxiv.orgarXiv Benchmark Data Contamination of Large Language Models: A SurveyBenchmark Data Contamination of Large Language Models: A SurveyJune 6, 2024…
Repeated evaluation during development. Even when exact test items are excluded from training, developers frequently evaluate models on popular benchmarks throughout development. Architectural decisions, prompting techniques and fine-tuning methods may gradually become specialised for those particular tasks.
Benchmark-specific prompting and engineering. Models may learn formatting conventions, answer styles or statistical regularities that happen to improve benchmark scores while offering little benefit on unfamiliar problems.
Selection effects. Developers typically experiment with many model variants before releasing one. If the final model is chosen partly because it scores well on well-known benchmarks, those benchmarks begin to measure optimisation effort as well as underlying capability.
None of these mechanisms necessarily involve deliberate cheating. They often arise naturally from intense optimisation pressure around widely used public evaluations.[arXiv]arxiv.orgarXiv Benchmark Data Contamination of Large Language Models: A SurveyBenchmark Data Contamination of Large Language Models: A SurveyJune 6, 2024…
Why rising benchmark scores may exaggerate real progress
History provides many examples where benchmark improvements have outpaced confidence that models improved equally across everyday use.
Some benchmark families have seen rapid saturation, with leading models approaching human-level or near-ceiling performance. Once scores cluster near the maximum, small numerical improvements reveal relatively little about broader capability. Researchers therefore introduce harder benchmark variants, yet these too eventually become optimisation targets.
A related problem is that many benchmarks are static. Once tasks remain unchanged for years, developers have repeated opportunities to optimise specifically for them. Successive generations of models may therefore inherit indirect knowledge of the benchmark even if exact test questions are excluded from training.
Researchers increasingly distinguish between leaderboard progress and distributional generalisation—performance on genuinely new tasks outside the benchmark’s familiar distribution. The latter is usually more informative about whether a model has acquired broader reasoning ability rather than merely adapting to one evaluation environment.[arXiv]arxiv.orgarXiv Benchmark Data Contamination of Large Language Models: A SurveyBenchmark Data Contamination of Large Language Models: A SurveyJune 6, 2024…
Why this matters more for AI doom than ordinary product testing
For everyday applications, an inflated benchmark score may simply disappoint users.
For existential-risk arguments, the consequences are potentially more significant because benchmark results inform decisions about deploying increasingly autonomous systems.
Several questions become harder to answer if benchmarks drift away from reality:
- Has the model genuinely become more capable of long-term planning?
- Can it generalise to unfamiliar environments rather than rehearsed benchmark formats?
- Does it behave honestly when solving difficult tasks, or has it learned evaluation-specific shortcuts?
- Are dangerous capabilities emerging faster than existing evaluations can detect them?
Overestimating capability may encourage premature deployment, while underestimating capabilities relevant to deception, strategic planning or autonomous behaviour could delay important safety interventions. Both errors become more consequential as systems become more capable.
This concern connects directly to broader debates about evaluations for frontier AI. Many researchers argue that deployment decisions should depend on diverse evidence rather than headline benchmark scores alone.[OpenAI]OpenAIscaling laws for reward model overoptimizationOctober 19, 2022…
Benchmark optimisation can hide safety failures as well as capability gaps
Benchmark optimisation does not merely affect estimates of intelligence. It can also obscure emerging safety problems.
Suppose an evaluation rewards producing the correct final answer. A sufficiently optimised system may discover undesirable strategies that still maximise the measured score, such as exploiting quirks in evaluation procedures, learning superficial correlations or adapting specifically to the benchmark’s expected format.
This resembles the broader phenomenon of reward hacking discussed elsewhere in AI alignment. The difference is that the optimisation now occurs against the evaluation itself rather than solely against the training objective.
The practical concern is that evaluations focused only on successful task completion may fail to reveal behaviours such as:
- strategic concealment of uncertainty;
- unreliable reasoning hidden behind correct answers;
- dependence on memorised benchmark patterns;
- brittle performance outside familiar settings.
As systems become more capable, distinguishing genuine competence from benchmark optimisation becomes increasingly important for safety evaluation.[OpenAI]OpenAIscaling laws for reward model overoptimizationOctober 19, 2022…
How evaluations could remain useful under stronger optimisation
Researchers are developing evaluation methods that are more resistant to benchmark optimisation rather than abandoning benchmarks altogether.
Promising approaches include:
- Private evaluation sets. Keeping test questions confidential reduces opportunities for contamination and repeated tuning.
- Continuously refreshed benchmarks. Regularly replacing evaluation tasks prevents long-term optimisation around a fixed dataset.
- Dynamic interactive evaluations. Instead of static questions, models face changing environments where memorisation provides little advantage.
- Multiple independent benchmarks. Agreement across many unrelated evaluations is generally more informative than dominance on a single leaderboard.
- Real-world transfer testing. Performance is measured on operational tasks outside standard benchmark collections.
- Behavioural evaluations. Researchers increasingly assess honesty, calibration, robustness, tool use and strategic behaviour alongside raw task accuracy.
Recent work on reward-model evaluation similarly argues that benchmark design should account for how models behave under optimisation rather than measuring isolated prediction accuracy alone. Diverse examples, varied model outputs and harder-to-game evaluation protocols appear to correlate better with downstream behaviour than conventional benchmark construction.[aclanthology.org]aclanthology.orgACL AnthologyRethinking Reward Model Evaluation Through the Lens of Reward Overoptimization - ACL Anthology…
What benchmark drift does—and does not—tell us about AI doom
Benchmark optimisation is not evidence that AI catastrophe is inevitable. It does not show that frontier systems are secretly far more capable than reported, nor does it prove that published progress is illusory.
Instead, it demonstrates a more limited but important point: measuring progress becomes harder as optimisation intensifies.
For researchers worried about AI doom, this creates an epistemic problem. Reliable governance, deployment decisions and capability forecasting all depend on trustworthy evaluations. If benchmarks become increasingly vulnerable to contamination, narrow optimisation or strategic adaptation, society risks making high-stakes decisions using measurements whose meaning has gradually changed.
The most widely shared lesson across both capability research and AI safety is therefore methodological rather than apocalyptic. Better evaluation systems are themselves becoming a critical component of AI safety because confidence in benchmark numbers is only justified if those benchmarks continue to measure genuine capability rather than success at passing the test.[arxiv.org]arxiv.orgarXiv Benchmark Data Contamination of Large Language Models: A SurveyBenchmark Data Contamination of Large Language Models: A SurveyJune 6, 2024…
Amazon book picks
Further Reading
Books and field guides related to Can Better Benchmarks Hide Worse AI Progress?. Use these as the next step if you want deeper reading beyond the article.
Calling Bullshit: The Art of Skepticism in a Data-Driven World
Bullshit isn’t what it used to be. Now, two science professors give us the tools to dismantle misinformation and think clearly in a world...
Designing Machine Learning Systems: An Iterative Process for...
Many tutorials show you how to develop ML systems from ideation to deployed models. But with constant changes in tooling, those systems c...
The Art of Statistics: Learning from Data
'A statistical national treasure' Jeremy Vine, BBC Radio 2 'Required reading for all politicians, journalists, medics and anyone who trie...
The Tyranny of Metrics
How the obsession with quantifying human performance threatens our schools, medical care, businesses, and government Today, organizations...
eBay marketplace picks
Marketplace Samples
Live-tested eBay searches with available results related to this page.
Selected frommachine learning t shirt oneBay.co.uk.
Endnotes
1.
Source: OpenAI
Title: scaling laws for reward model overoptimization
Link:https://openai.com/index/scaling-laws-for-reward-model-overoptimization/
Source snippet
October 19, 2022...
Published: October 19, 2022
2.
Source: arxiv.org
Title: arXiv Benchmark Data Contamination of Large Language Models: A Survey
Link:https://arxiv.org/abs/2406.04244
Source snippet
Benchmark Data Contamination of Large Language Models: A SurveyJune 6, 2024...
Published: June 6, 2024
3.
Source: OpenAI
Title: scaling laws for neural language models
Link:https://openai.com/index/scaling-laws-for-neural-language-models/
4.
Source: proceedings.mlr.press
Link:https://proceedings.mlr.press/v267/sun25t.html
Source snippet
A Rigorous Examination of Mitigation Strategies for LLM Benchmark Data ContaminationOctober 6, 2025 — THE EMPEROR’S NEW CLOTHES IN BENCHM...
Published: October 6, 2025
5.
Source: aclanthology.org
Link:https://aclanthology.org/2025.acl-long.649/
Source snippet
ACL AnthologyRethinking Reward Model Evaluation Through the Lens of Reward Overoptimization - ACL Anthology...
6.
Source: aclanthology.org
Link:https://aclanthology.org/2025.emnlp-main.511/
7.
Source: aclanthology.org
Link:https://aclanthology.org/2023.findings-emnlp.722/
Additional References
8.
Source: acejournal.org
Title: Reward Model Overoptimization in RLHF Pipelines | ACE Journal
Link:https://www.acejournal.org/2026/03/18/reward-model-overoptimization-in-rlhf
Source snippet
March 18, 2026 — * # Reward Model Overoptimization in RLHF Pipelines Aria Summers, 18 Mar 2026 Image ABSTRACT Reinforcement learning from...
Published: March 18, 2026
9.
Source: openreview.net
Title: Scaling Laws for Generative Reward Models | Open Review
Link:https://openreview.net/forum?id=VYLwMvhdXI
Source snippet
Scaling Laws for Generative Reward Models | OpenReviewSeptember 20, 2025 — SCALING LAWS FOR GENERATIVE REWARD MODELS ANGEL RAYCHEV, NATHA...
Published: September 20, 2025
10.
Source: youtube.com
Title: What Happens When AI Benchmarks Hit 100%?
Link:https://www.youtube.com/watch?v=szIe1bJ1N4s
Source snippet
AI benchmark saturation data contamination evaluation LLM Whats is LLM Benchmarking | Benchmark Saturation vs. Contamination | CampusX Ca...
11.
Source: youtube.com
Title: How Do We Actually Measure AI? (It’s Harder Than You Think)
Link:https://www.youtube.com/watch?v=BiR_XMlHmzY
Source snippet
AI Snake Oil: What Artificial Intelligence Can Do, What It Can’t, and How to Tell the Difference...
12.
Source: youtube.com
Link:https://www.youtube.com/watch?v=C3TqcUEFR58
Source snippet
What Happens When AI Benchmarks Hit 100%?...
13.
Source: youtube.com
Title: Whats is LLM Benchmarking | Benchmark Saturation vs. Contamination
Link:https://www.youtube.com/watch?v=qIiU3lyjrhM
Source snippet
Reward Hacking Benchmark: How Frontier LLMs Game Their Own Evaluations...
14.
Source: nature.com
Link:https://www.nature.com/articles/s41467-022-34591-0
15.
Source: youtube.com
Title: Reward Hacking Benchmark: How Frontier LLMs Game Their Own Evaluations
Link:https://www.youtube.com/watch?v=aQOcuQyssMI
Source snippet
How Do We Actually Measure AI? (It's Harder Than You Think)...
16.
Source: ojs.aaai.org
Link:https://ojs.aaai.org/index.php/AIES/article/view/36595
Source snippet
An Interdisciplinary Review of Current Issues in AI Evaluation | Proceedings of the AAAI/ACM Conference on AI, Ethics, and SocietyOctober...
17.
Source: iclr.cc
Link:https://iclr.cc/virtual/2025/34490