Within Algorithmic Gains
How Much Compute Can Better Algorithms Really Save?
Past efficiency gains show that better methods can deliver much more AI capability from the same hardware budget.
On this page
- The 44 fold Alex Net efficiency gain
- How software and hardware improvements multiply
- What historical gains do and do not prove
Page outline Jump by section
Introduction
Better algorithms can sometimes deliver surprisingly large increases in AI capability without any change in hardware. That matters for debates about AI doom because some proposed risk scenarios assume that slowing chip production or limiting access to advanced hardware will also slow frontier AI development. Historical evidence suggests the picture is more complicated. Improvements in software, model design and training methods have repeatedly reduced the amount of computation needed to reach a given level of performance, allowing researchers to extract much more capability from the same hardware budget.[OpenAI]OpenAIai and efficiencyMay 5, 2020…
This does not prove that algorithmic progress will continue indefinitely or that it can replace hardware scaling forever. It does show, however, that hardware constraints and capability growth are not linked one-for-one. When evaluating AI existential risk, this distinction matters because compute governance proposals often assume that limiting hardware limits capability growth. Historical efficiency gains suggest that software improvements can partially offset such limits, although the extent to which they can continue doing so remains uncertain.[OpenAI]OpenAIai and efficiencyMay 5, 2020…
The 44-fold AlexNet efficiency gain
The clearest historical dataset comes from work by Danny Hernandez and Tom Brown, who measured how much computation was required over time to achieve the same image-classification performance that AlexNet reached in 2012 on the ImageNet benchmark.
Rather than asking whether newer models performed better, they held capability constant and measured how much cheaper it became to achieve that capability. This provides a direct measure of algorithmic efficiency rather than raw performance.
Their analysis found that between 2012 and 2019:
- the compute required to reach AlexNet-level accuracy fell by roughly 44 times
- algorithmic efficiency doubled approximately every 16 months
- over the same period, improvements attributable to Moore’s Law alone would have produced only about an 11-fold hardware improvement.[OpenAI]OpenAIai and efficiencyMay 5, 2020…
The result is often misunderstood. It does not mean AI became 44 times more intelligent. Instead, it means researchers discovered methods that achieved the same capability using dramatically less computation. The savings came from many incremental advances rather than one revolutionary breakthrough, including better optimisation methods, improved neural-network architectures, more effective regularisation techniques and more efficient training procedures.[OpenAI]OpenAIai and efficiencyMay 5, 2020…
For AI-risk discussions, this evidence demonstrates that software innovation has historically been capable of producing improvements comparable to, or even exceeding, advances in hardware over multi-year periods.
How software and hardware improvements multiply
One reason efficiency gains receive attention in AI doom discussions is that they compound with hardware improvements rather than competing against them.
Suppose a new generation of chips performs twice as much computation per pound spent. If, at the same time, researchers discover an algorithm requiring only half as much computation for the same task, the combined improvement is approximately fourfold rather than twofold.
Over multiple years, these gains accumulate.
Instead of thinking about capability growth as being driven solely by faster chips, it is often more accurate to think of effective compute as the product of several factors:
- physical hardware performance
- algorithmic efficiency
- training methodology
- data quality
- hardware utilisation and software optimisation
Each factor increases the useful work obtained from a given computational budget.[OpenAI]OpenAIai and efficiencyMay 5, 2020…
This multiplication effect is one reason many forecasting models distinguish between physical compute and effective compute. A country or company may possess exactly the same number of GPUs while nevertheless becoming capable of training substantially stronger models because researchers learn to use those GPUs more efficiently.
Compute-optimal training shows efficiency is not only about faster code
The AlexNet evidence measured reductions in computation for a fixed capability. More recent work illustrates a different kind of efficiency gain: using the same computation budget more effectively.
A well-known example is DeepMind’s Chinchilla research on compute-optimal scaling. Earlier large language models generally increased parameter counts without proportionally increasing training data. Chinchilla instead demonstrated that, for a fixed training-compute budget, a smaller model trained on substantially more data could outperform much larger models.
Notably, the Chinchilla model:
- used roughly the same total training compute as Gopher
- contained 70 billion rather than 280 billion parameters
- trained on around four times as many tokens
- achieved better performance across many downstream benchmarks while also reducing inference costs.[Google DeepMind]deepmind.googleOpen source on deepmind.google.
This was not a reduction in hardware requirements. Instead, it showed that better allocation of the same computational budget could produce superior results.
For AI-risk analysis, this reinforces the broader lesson that capability growth depends not only on acquiring more hardware but also on learning how to spend available compute more efficiently.
What historical efficiency gains do and do not prove
Historical evidence provides a genuine reason to expect software improvements to remain important, but it does not justify assuming unlimited future acceleration.
Several important limitations deserve emphasis.
First, the 44-fold result comes from a particular benchmark—ImageNet image classification—not from every AI capability. OpenAI’s authors explicitly caution against assuming that the same trend applies universally across all machine learning domains. Some areas may experience much faster or much slower efficiency improvements.[OpenAI]OpenAIai and efficiencyMay 5, 2020…
Second, efficiency improvements often become harder after obvious opportunities have already been exploited. Early deep learning benefited from discovering fundamental techniques that had simply not yet been tried. Later advances may require progressively more difficult scientific breakthroughs.
Third, reducing compute for an established task is different from creating an entirely new capability. A dramatic conceptual breakthrough may initially require enormous computation before later algorithmic refinements make it efficient. The OpenAI analysis explicitly distinguishes between creating new capabilities and making existing capabilities cheaper.[OpenAI]OpenAIai and efficiencyMay 5, 2020…
Finally, some improvements primarily reduce training costs while others mainly reduce inference costs. Both matter economically, but they have different implications for forecasting frontier AI development.
Why this evidence matters for AI doom arguments
Within debates about existential risk, algorithmic efficiency affects several common claims.
One claim is that hardware bottlenecks alone may not reliably halt capability progress. If algorithms continue becoming substantially more efficient, existing hardware may support increasingly capable systems for longer than simple chip-count forecasts would suggest.
A second claim concerns compute governance. Policies that restrict advanced chips may still slow AI development significantly, but historical efficiency gains imply that researchers can partially compensate through better algorithms. Hardware controls therefore interact with software progress rather than replacing the need to monitor it.
A third implication concerns recursive improvement. Some AI-doom scenarios propose that increasingly capable AI systems could themselves accelerate research into better algorithms. Historical efficiency gains do not demonstrate that this feedback loop will occur, but they show that algorithmic innovation has already been a major independent driver of capability growth. Whether AI systems themselves could meaningfully accelerate that process remains an open empirical question rather than an established fact.[OpenAI]OpenAIai and efficiencyMay 5, 2020…
Overall, the historical record supports a cautious conclusion. Better algorithms have repeatedly delivered large compute savings, sometimes matching or exceeding improvements from hardware over several years. This strengthens the case that forecasts of advanced AI should consider both hardware and software trends together. At the same time, there is no evidence that historical efficiency improvements can be extrapolated indefinitely or that they guarantee an intelligence explosion. The strongest evidence supports sustained, significant algorithmic progress—not unlimited or exponential progress without end.[openai.com]OpenAIai and efficiencyMay 5, 2020…
Amazon book picks
Further Reading
Books and field guides related to How Much Compute Can Better Algorithms Really Save?. Use these as the next step if you want deeper reading beyond the article.
The Master Algorithm
"Describes the quest to find the Master Algorithm, which will take machine learning to the next level, allowing computers to learn how to...
Artificial Intelligence: A Modern Approach
Rating: 4.5/5 from 10 Google Books ratings
Artificial intelligence: A Modern Approach, 3e,is ideal for one or two-semester, undergraduate or graduate-level courses in Artificial In...
Artificial Intelligence
Rating: 4.5/5 from 10 Google Books ratings
Artificial intelligence: A Modern Approach, 3e,is ideal for one or two-semester, undergraduate or graduate-level courses in Artificial In...
Deep Learning
Rating: 3.5/5 from 6 Google Books ratings
An introduction to a broad range of topics in deep learning, covering mathematical and conceptual background, deep learning techniques us...
eBay marketplace picks
Marketplace Samples
Live-tested eBay searches with available results related to this page.
Selected fromcomputer chip poster oneBay.co.uk.
Endnotes
1.
Source: OpenAI
Title: ai and efficiency
Link:https://openai.com/index/ai-and-efficiency/
Source snippet
May 5, 2020...
Published: May 5, 2020
2.
Source: arxiv.org
Title: arXiv Measuring the Algorithmic Efficiency of Neural Networks
Link:https://arxiv.org/abs/2005.04305
3.
Source: deepmind.google
Link:https://deepmind.google/blog/an-empirical-analysis-of-compute-optimal-large-language-model-training/
4.
Source: OpenAI
Title: language models are few shot learners
Link:https://openai.com/index/language-models-are-few-shot-learners/
5.
Source: OpenAI
Title: scaling laws for neural language models
Link:https://openai.com/index/scaling-laws-for-neural-language-models/
6.
Source: youtube.com
Title: The Stack of Exponentials: Why AI Capability Grows 10x Every Year
Link:https://www.youtube.com/watch?v=0iAJWtmOTMQ
Source snippet
OpenAI AI and efficiency compute algorithmic progress AlexNet The moment we stopped understanding AI [AlexNet] Welch Labs...
7.
Source: doi.org
Link:https://doi.org/10.48550/arXiv.2203.15556
Additional References
8.
Source: proceedings.mlr.press
Link:https://proceedings.mlr.press/v235/saha24a.html
Source snippet
Barna Saha, Christopher Ye Proceedings of the 41st [International]({{ 'shared-testing/' | relative_url }}) Conference on Machine Learning, PMLR 235:43024-43042, 2024. ABSTRACT...
9.
Source: engineering.fyi
Title: A I and efficiency | Open AI Engineering Blog | Engineering.fyi
Link:https://www.engineering.fyi/article/ai-and-efficiency
Source snippet
AI and efficiency | OpenAI Engineering Blog | Engineering.fyiJune 9, 2022 — AI AND EFFICIENCY [Input: Search articles...] HomeOpenAIAI an...
Published: June 9, 2022
10.
Source: mdpi.com
Title: Assessing Efficiency in Artificial Neural Networks
Link:https://www.mdpi.com/2076-3417/13/18/10286
Source snippet
September 14, 2023 — (This article belongs to the Special Issue Advances in Computer Vision and Semantic Segmentation) Download keyboard_...
Published: September 14, 2023
11.
Source: youtube.com
Title: Neil Thompson | How Algorithmic Progress is driving progress in Big Data and AI
Link:https://www.youtube.com/watch?v=PfEYKyfu0O0
Source snippet
The moment we stopped understanding AI [AlexNet]...
12.
Source: youtube.com
Title: AI can’t cross this line and we don’t know why
Link:https://www.youtube.com/watch?v=5eqRuVp65eY
Source snippet
The Stack of Exponentials: Why AI Capability Grows 10x Every Year...
13.
Source: github.com
Link:https://github.com/AkihikoWatanabe/paper_notes/issues/1827
Source snippet
[Paper Note] Training Compute-Optimal Large Language Models, Jordan Hoffmann+, NeurIPS'22, 2022.03 · Issue #1827 · AkihikoWatanabe/paper_...
14.
Source: mlanthology.org
Link:https://mlanthology.org/neurips/2022/hoffmann2022neurips-empirical/
15.
Source: deepnlp.org
Link:https://www.deepnlp.org/content/articles/an-empirical-analysis-of-compute-optimal-large-language-model-training
16.
Source: jmlr.org
Link:https://www.jmlr.org/papers/v24/22-1208.html
17.
Source: proceedings.neurips.cc
Link:https://proceedings.neurips.cc/paper_files/paper/2022/hash/c1e2faff6f588870935f114ebe04a3e5-Abstract.html