Within Current Benchmarks
What Long Horizon Benchmarks Reveal About AI Reliability
Benchmarks like METR, LongCLI-Bench, and RetailBench show AI reliability drops sharply as task chains lengthen.
On this page
- Summary of METR and time horizon metrics
- Failure patterns in software and data analysis benchmarks
- Trends over recent years and measured horizon improvements
Page outline Jump by section
Introduction
Current large‑language‑model (LLM) agents often perform impressively on short problems — answer a question, fix a tiny bug, or write a paragraph. But when researchers ask whether these systems can sustain multi‑step work over hours or dozens of interdependent actions, a very different picture emerges. Independent long‑horizon benchmarks — datasets and evaluation suites designed to measure how reliably an AI agent can complete extended tasks — consistently show that performance falls off sharply as task chains lengthen, error modes compound, and context maintenance becomes critical. These results matter for debates about AI risk because many loss‑of‑control and dangerous autonomy scenarios assume an AI would need to carry out sustained work without frequent human oversight. Long‑horizon benchmarks like task‑completion time horizons, LongCLI‑Bench, and LongDS‑Bench provide real, quantifiable evidence of where contemporary systems break, and how dramatically performance degrades beyond short, isolated tasks. [METR]metr.orgMETRTask-Completion Time Horizons of Frontier AI ModelsThe task-completion time horizon is the task duration (measured by human expert co…
What Time‑Horizon Metrics Like METR Reveal — and What They Don’t
One influential effort to quantify long‑task ability is the task‑completion time horizon metric developed by the research outfit METR (Model Evaluation and Threat Research). Instead of simple single‑turn accuracy, this metric estimates the duration of a task (as measured by how long a human expert would take) that an agent can complete with a given level of reliability, for example with roughly 50 % chance of success under automated evaluation. METR’s latest published numbers show a progression from seconds‑long tasks in early models to tasks measured in hours for recent frontier systems. [METR]metr.orgWe show that this metric has been consistently exponentially iMeasuring AI Ability to Complete Long Tasks - METRMarch 19, 2025 — Measuring AI Ability to Complete Long Tasks We propose measuring AI pe…
The headline interpretation many safety and capability watchers place on this is that agents are extending their effective autonomous horizon — roughly doubling how long a problem they can reliably complete over time. But two important caveats emerge from both METR’s own documentation and independent analyses:
- Internal uncertainty is high. METR’s confidence intervals around time‑horizon estimates are often very wide — for example, a reported ~12‑hour horizon for a leading system might be statistically compatible with much lower or higher values depending on task sampling — and the group itself cautions that measurements above 16 hours are unreliable with the current task suite. [METR]metr.org2026 01 22 time horizon limitationsClarifying limitations of time horizon22 Jan 2026 — METR Logo. METR researches, develops, and evaluates frontier AI systems to measure ho…
- Benchmark construction shapes interpretation. Critics and technical reviewers point out that the available METR tasks are self‑contained software problems with clear success criteria, not necessarily representative of broader long‑horizon reasoning in changing environments. Fitting logistic curves across a relatively small set of tasks (e.g., ~14 tasks in some ranges) can magnify measurement noise and make headline figures fragile. [Medium]medium.comAre AI time-horizons (still) doubling every 7 months?MediumAre AI time-horizons (still) doubling every 7 months?March 11, 2026 — A critical review of METR's 'Task-Completion Time Horizons of…
In other words, while time‑horizon metrics sketch a direction — that models are extending how long a problem they can sometimes sustain — they do not show robust, general, or reliably high performance on genuinely long, interdependent tasks. They instead highlight how rapidly success probability declines as task complexity and duration increase beyond the single‑turn regime. [METR]metr.orgMETRTask-Completion Time Horizons of Frontier AI ModelsThe task-completion time horizon is the task duration (measured by human expert co…
Explicit Long‑Horizon Benchmarks Show Sharp Reliability Drops
Where time‑horizon curves provide a broad view of trends, specialised long‑horizon benchmarks give direct evidence of failure patterns on tasks that require extended planning, state maintenance, and error recovery.
LongCLI‑Bench: Complex, Multi‑Step Engineering Tasks
The LongCLI‑Bench benchmark was created to test AI agents on real‑world, extended software engineering workflows through command‑line interfaces. Rather than single edits or tiny patches, LongCLI tasks involve sequences of operations — from building a project from scratch, to adding features, to fixing bugs and performing refactorings — each requiring planning, multiple tool invocations, and context tracking. [arXiv]arxiv.orgarXiv Long DS-Bench: On the Failure of Long-Horizon Agentic Data AnalysisarXiv Long DS-Bench: On the Failure of Long-Horizon Agentic Data Analysis
When researchers ran a suite of state‑of‑the‑art agents against LongCLI‑Bench, the overall pass rates were very low: even the best‑performing commercial combinations rarely exceeded ~16.7 % on the full task suite, and most attempts stalled before completing even a third of the required steps. [app.argminai.com]app.argminai.comLongCLI-Bench: a 20-task CLI benchmark showing state-of-the-art agents pass <20% on long-horizon engineering tasks | Argmin AIFebruary 15… The benchmark also records step‑level scores that reveal where failures happen: many agents fail not because of a single off‑by‑one error, but because of basic planning or coordination breakdowns early in the workflow that cascade into complete task abandonment. [HyperAI]hyper.aiHyperAILongCLI-Bench: A Preliminary Benchmark and Study for Long-horizon Agentic Programming in Command-Line Interfaces | Papers | HyperAI…
These results underline two points that simple pass/fail metrics obscure:
- Agents often cannot initiate and sustain the correct sequence of actions even when each individual action seems straightforward; and
- When agents do proceed, many regressions or unintended side‑effects occur because they fail to balance new requirements with preserving existing functionality. Liner
LongDS‑Bench: Evolving Context and State in Data Analysis
Another benchmark, LongDS‑Bench, focuses on long‑horizon data‑analysis tasks drawn from real Kaggle notebooks and similar workflows. These tasks require an agent to maintain, update, restore, and compose evolving analytical state across more than two thousand interactive turns — far beyond static or short prompting scenarios. [arXiv]arxiv.orgarXiv Long DS-Bench: On the Failure of Long-Horizon Agentic Data AnalysisarXiv Long DS-Bench: On the Failure of Long-Horizon Agentic Data Analysis
On this benchmark, even the strongest models achieve only ~48 % average accuracy, and performance drops roughly 47 percentage points between early and late stages of an episode. Crucially, long‑horizon errors — mistakes tied to failure to track previous state or correct intermediate results — account for more than half of all failures, suggesting that the key bottleneck isn’t just “not enough steps” but an inability to maintain a coherent evolving representation of the task over time. [arXiv]arxiv.orgarXiv Long DS-Bench: On the Failure of Long-Horizon Agentic Data AnalysisarXiv Long DS-Bench: On the Failure of Long-Horizon Agentic Data Analysis
Trends and Consistency Across Benchmarks
Across these benchmarks, a clear and consistent pattern emerges: as the effective horizon — whether measured in turns, actions, or contextual dependencies — grows, agent performance degrades markedly and nonlinearly. Short‑horizon metrics that look at isolated problems or single pass/fail outcomes mask these structural patterns of degradation. Metrics designed to capture reliability over extended sequences consistently show that:
- Success probability declines across steps. Where early actions might be handled competently, later actions accumulate residual mistakes or lose context, leading to systematic drop‑offs in reliability. [arXiv]arxiv.orgarXiv Long DS-Bench: On the Failure of Long-Horizon Agentic Data AnalysisarXiv Long DS-Bench: On the Failure of Long-Horizon Agentic Data Analysis
- Agents struggle with planning vs execution. Failures often trace back to poor initial planning, incorrect assumptions about intermediate states, or lack of recovery mechanisms when paths diverge from what was expected. [app.argminai.com]app.argminai.comLongCLI-Bench: a 20-task CLI benchmark showing state-of-the-art agents pass <20% on long-horizon engineering tasks | Argmin AIFebruary 15…
- Human collaboration currently improves outcomes. In LongCLI‑Bench, incorporating human guidance or structured plans raises success rates compared with purely autonomous runs, suggesting that hybrid workflows remain more effective than standalone agents for complex, extended tasks. [HyperAI]hyper.aiHyperAILongCLI-Bench: A Preliminary Benchmark and Study for Long-horizon Agentic Programming in Command-Line Interfaces | Papers | HyperAI…
These trends hold across engineering, analysis, and heuristic diagnostic tasks: no current benchmark shows robust, high‑reliability performance on tasks that genuinely require extended, interdependent action sequences with stateful reasoning.
Implications for Evaluating AI Risk
For lay readers and people thinking about advanced AI risk, this evidence from long‑horizon benchmarks serves three key clarifications:
eBay marketplace picks
Marketplace Samples
Example marketplace items related to this page. Use the search link to explore similar finds on eBay.
- Capability on short tasks is a poor predictor of long‑horizon reliability. Systems that look strong on single‑turn evaluations can still fail consistently when subtle dependencies stretch over dozens or hundreds of steps. [TianPan]tianpan.co2026 04 10 long horizon evaluation gap agent benchmarksTianPanThe Long-Horizon Evaluation Gap: Why Your Agent…10 Apr 2026 — Single-turn benchmarks give a false sense of security for product…
- State drift, error accumulation, and planning deficits are real mechanisms of failure. These aren’t artefacts of one benchmark; they show up across domains (software, data analysis) and across multiple measurement frameworks. [arXiv]arxiv.orgarXiv Long DS-Bench: On the Failure of Long-Horizon Agentic Data AnalysisarXiv Long DS-Bench: On the Failure of Long-Horizon Agentic Data Analysis
- Current agents are far from being reliably autonomous in extended real‑world settings. Even with state‑of‑the‑art models and careful benchmark design, performance on long‑horizon tasks often hovers at rates well below what humans would consider “dependable,” especially without human guidance. [app.argminai.com]app.argminai.comLongCLI-Bench: a 20-task CLI benchmark showing state-of-the-art agents pass <20% on long-horizon engineering tasks | Argmin AIFebruary 15…
In short, the best available empirical evidence suggests that as we lengthen the horizon — the number of steps, dependencies, and interlinked decisions an agent must make — AI task failure becomes the norm rather than the exception. For researchers and policymakers thinking seriously about high‑impact risks, those failure patterns matter more than headline short‑task performance scores. [TianPan]tianpan.co2026 04 10 long horizon evaluation gap agent benchmarksTianPanThe Long-Horizon Evaluation Gap: Why Your Agent…10 Apr 2026 — Single-turn benchmarks give a false sense of security for product…
Endnotes
-
Source: metr.org
Link: https://metr.org/time-horizons/Source snippet
METRTask-Completion Time Horizons of Frontier AI ModelsThe task-completion time horizon is the task duration (measured by human expert co...
-
Source: arxiv.org
Title: arXiv Long DS-Bench: On the Failure of Long-Horizon Agentic Data Analysis
Link: https://arxiv.org/abs/2605.30434 -
Source: medium.com
Title: Are AI time-horizons (still) doubling every 7 months?
Link: https://medium.com/%40AIchats/are-ai-time-horizons-still-doubling-every-7-months-6262ed2bcc6aSource snippet
MediumAre AI time-horizons (still) doubling every 7 months?March 11, 2026 — A critical review of METR's 'Task-Completion Time Horizons of...
Published: March 11, 2026
-
Source: arxiv.org
Link: https://arxiv.org/abs/2602.14337Source snippet
arXivLongCLI-Bench: A Preliminary Benchmark and Study for Long-horizon Agentic Programming in Command-Line InterfacesFebruary 15, 2026...
Published: February 15, 2026
-
Source: app.argminai.com
Link: https://app.argminai.com/arxiv-dashboard/papers/2602.14337v2Source snippet
LongCLI-Bench: a 20-task CLI benchmark showing state-of-the-art agents pass <20% on long-horizon engineering tasks | Argmin AIFebruary 15...
-
Source: liner.com
Link: https://liner.com/review/longclibench-preliminary-benchmark-and-study-for-longhorizon-agentic-programming-inSource snippet
LinerLongCLI-Bench: A Preliminary Benchmark and Study for Long-horizon Agentic Programming in Command-Line Interfaces [Quick Review]Febru...
-
Source: tianpan.co
Title: 2026 04 10 long horizon evaluation gap agent benchmarks
Link: https://tianpan.co/blog/2026-04-10-long-horizon-evaluation-gap-agent-benchmarksSource snippet
TianPanThe Long-Horizon Evaluation Gap: Why Your Agent...10 Apr 2026 — Single-turn benchmarks give a false sense of security for product...
-
Source: metr.org
Title: We show that this metric has been consistently exponentially i
Link: https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/?_bhlid=eb9ba26f893982d302f59d4adee697067ed90a41Source snippet
Measuring AI Ability to Complete Long Tasks - METRMarch 19, 2025 — Measuring AI Ability to Complete Long Tasks We propose measuring AI pe...
Published: March 19, 2025
-
Source: arxiv.org
Title: The Long-Horizon Task Mirage?
Link: https://arxiv.org/html/2604.11978v1Source snippet
Diagnosing Where and...13 Apr 2026 — Our findings offer an initial methodological step toward systematic, cross-domain analysis of long...
-
Source: metr.org
Title: 2026 01 22 time horizon limitations
Link: https://metr.org/notes/2026-01-22-time-horizon-limitations/Source snippet
Clarifying limitations of time horizon22 Jan 2026 — METR Logo. METR researches, develops, and evaluates frontier AI systems to measure ho...
-
Source: medium.com
Link: https://medium.com/agentic-builders/5-agent-design-patterns-for-long-running-ai-agents-423ff3f73850Source snippet
ion patterns from Google's Agent Runtime that demos never...
-
Source: hyper.ai
Link: https://hyper.ai/en/papers/2602.14337Source snippet
HyperAILongCLI-Bench: A Preliminary Benchmark and Study for Long-horizon Agentic Programming in Command-Line Interfaces | Papers | HyperAI...
-
Source: papers.cool
Link: https://papers.cool/arxiv/2604.16788Source snippet
LongBench: Evaluating Robotic Manipulation Policies on Real-World Long-Horizon Tasks | Cool Papers - Immersive Paper DiscoveryApril 18, 2...
-
Source: papers.cool
Link: https://papers.cool/arxiv/2602.14337Source snippet
LongCLI-Bench: A Preliminary Benchmark and Study for Long-horizon Agentic Programming in Command-Line Interfaces | Cool Papers - Immersiv...
-
Source: hyper.ai
Link: https://hyper.ai/fr/papers/2602.14337Source snippet
LongCLI-Bench: Un benchmark préliminaire et une étude sur la programmation agente à horizon long dans les interfaces en ligne de command...
Additional References
-
Source: researchtrend.ai
Link: https://researchtrend.ai/papers/2603.29231Source snippet
Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents | ResearchTrend.AIMarch 31, 2026 — BEYOND PASS@1: A RELIABILIT...
Published: March 31, 2026
-
Source: xwang2775.github.io
Link: https://xwang2775.github.io/horizon-leaderboard/Source snippet
HORIZON Leaderboard — Long-Horizon Agent EvaluationAn initial diagnostic benchmark for systematically constructing tasks and characterizi...
-
Source: ai-search.io
Link: https://ai-search.io/papers/longcli-bench-a-preliminary-benchmark-and-study-for-long-horizon-agentic-programming-in-command-line-interfacesSource snippet
LongCLI-Bench: A Preliminary Benchmark and Study for Long-horizon Agentic Programming in Command-Line Interfaces - AI for Dummies - Under...
-
Source: gist.science
Link: https://gist.science/paper/2602.14337Source snippet
LongCLI-Bench: A Preliminary Benchmark and Study for Long-horizon Agentic Programming in Command-Line Interfaces | Gist.ScienceFebruary 2...
-
Source: epoch.ai
Link: https://epoch.ai/benchmarks/metr-time-horizonsSource snippet
METR Time HorizonsThis metric represents the estimated time (in minutes or hours) that a human expert would typically take to complete ta...
-
Source: reddit.com
Link: https://www.reddit.com/r/singularity/comments/1qyx3k3/oai_researcher_noam_brown_responds_to_question/Source snippet
OAI researcher Noam Brown responds to question about...OAI researcher Noam Brown responds to question about absurd METR pace saying it w...
-
Source: tldr.takara.ai
Link: https://tldr.takara.ai/p/2603.29231Source snippet
pass@1: A Reliability Science Framework for Long-Horizon LLM Agents | Takara TLDRImage: DS1 spectrogram: Beyond pass@1: A Reliability Sci...
-
Source: tldr.takara.ai
Link: https://tldr.takara.ai/p/2602.14337Source snippet
takara.aiLongCLI-Bench: A Preliminary Benchmark and Study for Long-horizon Agentic Programming in Command-Line Interfaces | Takara TLDRIm...
-
Source: researchgate.net
Link: https://www.researchgate.net/publication/404021178_LongBench_Evaluating_Robotic_Manipulation_Policies_on_Real-World_Long-Horizon_TasksSource snippet
(PDF) LongBench: Evaluating Robotic Manipulation Policies on Real-World Long-Horizon TasksApril 18, 2026 — LONGBENCH: EVALUATING ROBOTIC...
Published: April 18, 2026
-
Source: theregister.com
Title: microsoft researchers find ai models and agents cant handle long running tasks
Link: https://www.theregister.com/ai-ml/2026/05/11/microsoft-researchers-find-ai-models-and-agents-cant-handle-long-running-tasks/5238263Source snippet
Microsoft researchers find AI models and agents can't...11 May 2026 — "Our findings show that current LLMs introduce substantial errors...
Published: May 2026
Topic Tree