Within Algorithmic Gains
Where Do Algorithmic Workarounds Finally Hit a Wall?
Diminishing returns, costly experiments and physical limits may slow algorithmic progress before it becomes self-sustaining.
On this page
- Diminishing returns in model performance
- The cost of validating new ideas
- Memory, energy and manufacturing constraints
Page outline Jump by section
Introduction
Better algorithms can make AI systems dramatically more efficient, allowing them to achieve higher performance with the same amount of computing power. That possibility is one reason some researchers argue that slowing the growth of chip production would not necessarily slow the advance of frontier AI by the same amount. However, there is an equally important counterpoint: software cannot eliminate every hardware bottleneck.
For debates about AI doom and existential risk, this distinction matters because it affects how quickly AI capabilities might continue improving if physical resources become scarce. Algorithms can reduce wasted computation, improve memory use and make training more efficient, but they still depend on real machines that consume electricity, move data through memory, dissipate heat and must ultimately be manufactured. The question is therefore not whether software can compensate for hardware limits—it clearly can in many cases—but where those gains eventually run into physical and economic constraints.
Where do algorithmic workarounds finally hit a wall?
Algorithmic improvements often postpone hardware bottlenecks rather than remove them. A more efficient training method may allow the same GPU cluster to achieve better performance, but the underlying hardware still has fixed limits on memory capacity, memory bandwidth, communication speed and energy consumption.
Computer science has repeatedly shown that better algorithms can transform practical performance. Sorting algorithms, compression techniques and machine learning optimisers all demonstrate that software matters enormously. Modern AI has experienced similar gains. OpenAI estimated that between 2012 and 2019, the computation needed to reach AlexNet-level image recognition performance fell by roughly 44-fold because of algorithmic improvements alone. Those gains multiplied rather than replaced hardware progress.[OpenAI]OpenAIai and efficiencyMay 5, 2020…
The important implication is that hardware and software are complements, not substitutes. Better algorithms extract more value from existing chips, but they do not remove the need for chips altogether.
Diminishing returns become increasingly important
One reason software cannot bypass every hardware limit is that many of the easiest efficiency improvements are discovered first.
Early in the development of a new architecture, researchers may identify obvious inefficiencies. Training procedures become more stable, optimisers improve, data pipelines are refined and memory usage is reduced. These changes can produce striking gains.
Later improvements tend to become harder.
Instead of discovering another simple optimisation, researchers may need to:
- run thousands of expensive experiments;
- search enormous design spaces for small improvements;
- validate ideas across multiple benchmarks;
- confirm that gains transfer to different models rather than exploiting quirks in one dataset.
This pattern is common across engineering. Initial progress is often rapid because many inefficiencies remain. As systems mature, additional improvements generally require disproportionately greater effort.
That does not mean progress stops, but it weakens simple assumptions that algorithmic efficiency will continue improving indefinitely at the same historical rate. Even researchers who expect substantial future software advances generally acknowledge uncertainty about whether past exponential trends will persist.[OpenAI]OpenAIai and efficiencyMay 5, 2020…
Better ideas still have to be discovered experimentally
Discussions of recursive AI improvement sometimes imagine AI systems rapidly inventing increasingly efficient algorithms. Even if such systems become extremely capable, new ideas would still need validation.
Machine learning research is unusually empirical. Many plausible-looking ideas fail when implemented. Others work only for certain model sizes or hardware configurations. Small architectural changes can produce unexpected interactions that are difficult to predict theoretically.
Testing new algorithms therefore requires access to:
- computing hardware;
- benchmark datasets;
- engineering time;
- repeated training runs;
- careful statistical evaluation.
As frontier models become larger, every experiment becomes more expensive. A promising idea that would once have required hours on a workstation may require days or weeks on thousands of accelerators.
This creates an important feedback effect. Better software may accelerate research, but validating that software itself increasingly depends on scarce computational resources.
Memory increasingly becomes the bottleneck
Not every limitation comes from raw arithmetic performance.
Modern AI systems frequently spend much of their time moving data rather than performing calculations. Large language models repeatedly transfer enormous numbers of parameters between memory and processing units. If data cannot arrive quickly enough, arithmetic units sit idle waiting for memory.
Researchers increasingly describe this as the “memory wall”: improvements in processor speed have substantially outpaced improvements in memory bandwidth over many years. Recent analyses argue that memory movement, rather than floating-point computation itself, is becoming the dominant constraint for many transformer inference workloads.[arXiv]arxiv.orgarXiv AI and Memory WallAI and Memory WallMarch 21, 2024…
Software can reduce this problem through techniques such as:
- quantisation;
- improved caching;
- FlashAttention-style algorithms;
- sparse activation;
- better scheduling.
These methods reduce unnecessary data movement, but they cannot eliminate the need to move information entirely. Eventually every model parameter that is actually used must still be stored and retrieved by physical hardware.
Energy cannot be optimised away
Every computation requires energy.
Software improvements often reduce energy per useful result by eliminating unnecessary operations. Nevertheless, each logical operation ultimately corresponds to physical processes inside transistors.
Large AI systems therefore remain constrained by:
- electricity generation;
- power delivery;
- cooling infrastructure;
- thermal management inside chips;
- operating costs over long training runs.
An algorithm that halves computation approximately halves energy requirements for that workload. It does not reduce them to zero.
For AI doom discussions, this distinction matters because some intelligence-explosion scenarios assume capability improvements continue much faster than infrastructure expansion. Hardware constraints suggest that even extremely capable AI systems must still operate within physical energy budgets unless future hardware also improves substantially.
Manufacturing limits remain external to software
Even perfect software cannot manufacture additional GPUs.
Frontier AI currently depends on advanced semiconductor fabrication, specialised packaging technologies, high-bandwidth memory (HBM), networking equipment and large-scale data centres.
Many of these components have supply constraints that software cannot directly remove.
For example:
- advanced lithography machines require years to manufacture;
- high-bandbandwidth memory production remains capacity-limited;
- advanced packaging facilities expand much more slowly than software development;
- new data centres require land, electricity connections and construction.
Algorithms may reduce demand for these resources by improving efficiency, but they cannot instantly increase physical production capacity. When hardware shortages occur, software helps stretch available resources rather than eliminating scarcity itself.[techradar.com]techradar.comIt uniquely supports open standards like PyTorch and Kubernetes, promoting ease of adoption without proprietary lock-in. Existing deploym…
Some limits are set by the laws of physics
Certain bottlenecks arise from fundamental physical constraints rather than engineering choices.
Examples include:
- the speed at which electrical signals propagate;
- heat generated by switching transistors;
- communication delays across large computing clusters;
- thermodynamic limits on energy efficiency;
- finite storage density.
Engineering improvements can approach these limits more closely over time, but cannot surpass them indefinitely.
This does not imply that present hardware is anywhere near absolute physical limits. Considerable room for improvement remains through better chip design, specialised accelerators, optical interconnects, advanced memory technologies and other innovations. The key point is simply that software alone cannot abolish constraints imposed by physics.
What this means for AI doom forecasts
Within AI doom debates, hardware bottlenecks are often discussed because they influence the plausibility and speed of recursive capability growth.
Those who expect rapid AI progress argue that algorithmic innovation has historically delivered gains comparable to, or even exceeding, improvements from hardware alone over meaningful periods. They therefore caution against assuming that slowing chip development would proportionally slow AI capability growth.[OpenAI]OpenAIai and efficiencyMay 5, 2020…
Critics respond that extrapolating past software improvements indefinitely may overlook several slowing forces:
- diminishing returns from optimisation;
- rising experimental costs;
- validation bottlenecks;
- memory bandwidth constraints;[techradar.com]techradar.comIt uniquely supports open standards like PyTorch and Kubernetes, promoting ease of adoption without proprietary lock-in. Existing deploym…
- energy availability;
- semiconductor manufacturing capacity.
These considerations do not show that recursive improvement is impossible. Instead, they suggest that intelligence growth may remain coupled to physical infrastructure more tightly than some simplified recursive self-improvement models imply.
The resulting disagreement is therefore about degree rather than principle. Nearly everyone agrees that software can substantially delay hardware bottlenecks. The central uncertainty is how long those workarounds continue delivering large capability gains before physical limits become the dominant constraint once again.
Amazon book picks
Further Reading
Books and field guides related to Where Do Algorithmic Workarounds Finally Hit a Wall?. Use these as the next step if you want deeper reading beyond the article.
The Master Algorithm
"Describes the quest to find the Master Algorithm, which will take machine learning to the next level, allowing computers to learn how to...
Chip War
***Winner of the 2022 Financial Times Business Book of the Year Award*** ***Selected as one of Barack Obama's Favourite Books of 2023***...
Artificial Intelligence
Rating: 4.5/5 from 10 Google Books ratings
Artificial intelligence: A Modern Approach, 3e,is ideal for one or two-semester, undergraduate or graduate-level courses in Artificial In...
Deep Learning
Rating: 3.5/5 from 6 Google Books ratings
An introduction to a broad range of topics in deep learning, covering mathematical and conceptual background, deep learning techniques us...
eBay marketplace picks
Marketplace Samples
Live-tested eBay searches with available results related to this page.
Selected fromcomputer hardware poster oneBay.co.uk.
Current eBay listing
COMPUTER HARDWARE CHEAT SHEET POSTER detailed educational 20x30 -VY1
Current eBay listing
368417 COMPUTER HARDWARE CHEAT SHEET detailed educational Art Poster
Endnotes
1.
Source: OpenAI
Title: ai and efficiency
Link:https://openai.com/index/ai-and-efficiency/
Source snippet
May 5, 2020...
Published: May 5, 2020
2.
Source: arxiv.org
Title: arXiv Measuring the Algorithmic Efficiency of Neural Networks
Link:https://arxiv.org/abs/2005.04305
3.
Source: OpenAI
Title: Open AIScaling laws for neural language models | Open AI
Link:https://openai.com/index/scaling-laws-for-neural-language-models/
Source snippet
Scaling laws for neural language models | OpenAI...
4.
Source: arxiv.org
Title: arXiv AI and Memory Wall
Link:https://arxiv.org/abs/2403.14123
Source snippet
AI and Memory WallMarch 21, 2024...
Published: March 21, 2024
5.
Source: techradar.com
Link:https://www.techradar.com/pro/those-two-jobs-need-different-physics-rebellions-ceo-says-training-and-inference-need-different-chips
Source snippet
It uniquely supports open standards like PyTorch and Kubernetes, promoting ease of adoption without proprietary lock-in. Existing deploym...
6.
Source: OpenAI
Title: Open AIAI and compute | Open AI
Link:https://openai.com/index/ai-and-compute/
Source snippet
AI and compute | OpenAI...
7.
Source: OpenAI
Title: evaluating chain of thought monitorability
Link:https://openai.com/index/evaluating-chain-of-thought-monitorability/
8.
Source: OpenAI
Title: techniques for training large neural networks
Link:https://openai.com/index/techniques-for-training-large-neural-networks/
9.
Source: OpenAI
Title: how ai training scales
Link:https://openai.com/index/how-ai-training-scales/
10.
Source: OpenAI
Title: ai and compute
Link:https://openai.com/index/ai-and-compute/?trk=public_post_comment-text
Additional References
11.
Source: aibottlenecks.app
Link:https://aibottlenecks.app/essay/von-neumann-bottleneck
Source snippet
May 14, 2026 — Essay · May 14, 2026 · 11 min read THE VON NEUMANN BOTTLENECK: WHY AI'S REAL CONSTRAINT IS MEMORY, NOT COMPUTE Eighty year...
Published: May 14, 2026
12.
Source: sciencedirect.com
Title: Energy use of AI inference, efficiency pathways, and test-time scaling
Link:https://www.sciencedirect.com/science/article/pii/S2542435126001145
Source snippet
"Article Energy use of AI inference, efficiency pathways, and test-time scaling [https://doi.org/10.1016/j.joule.2026..."](https://doi.org/10.1016/j.joule.2026...")...
13.
Source: cloud.google.com
Title: five techniques to reach the efficient frontier of llm inference
Link:https://cloud.google.com/blog/topics/developers-practitioners/five-techniques-to-reach-the-efficient-frontier-of-llm-inference
Source snippet
techniques to reach the efficient frontier of LLM inference | Google Cloud BlogMarch 27, 2026 — Five techniques to reach the efficient fr...
Published: March 27, 2026
14.
Source: youtube.com
Title: Why AI Can’t Scale Forever (Physics Is the Problem)
Link:https://www.youtube.com/watch?v=8YRdTAo0lGQ
Source snippet
The Real Reason Your PC Can't Run AI | Memory Wall Explained...
15.
Source: youtube.com
Title: The Real Reason Your PC Can’t Run AI | Memory Wall Explained
Link:https://www.youtube.com/watch?v=U5P2k0SfPWo
Source snippet
AI Is Running Into a New Limit: Electricity...
16.
Source: nvidia.com
Link:https://www.nvidia.com/en-us/data-center/h100/
17.
Source: microsoft.com
Title: Energy use of AI inference, efficiency pathways, and test-time scaling
Link:https://www.microsoft.com/en-us/research/publication/energy-use-of-ai-inference-efficiency-pathways-and-test-time-scaling/?lang=ko-kr
18.
Source: youtube.com
Title: Memory is the new bottleneck
Link:https://www.youtube.com/watch?v=-nWu-ZIQ7PQ
Source snippet
Why AI Can't Scale Forever (Physics Is the Problem)...
19.
Source: youtube.com
Title: Dylan Patel — The single biggest bottleneck to scaling [AI compute]({{ ‘compute-limits/’ | relative_url }})
Link:https://www.youtube.com/watch?v=mDG_Hx3BSUE
Source snippet
Memory is the new bottleneck...
20.
Source: youtube.com
Title: AI Is Running Into a New Limit: Electricity
Link:https://www.youtube.com/watch?v=raWznWhzYqM