Within Open Weights
Can Open Weight Safeguards Survive Determined Modification?
New safeguards may resist removal, but current evidence does not show that they can reliably survive determined modification.
On this page
- How fine tuning can weaken refusal behaviour
- What tamper resistant research has achieved so far
- Why independent testing remains difficult
Page outline Jump by section
Introduction
The short answer is: not yet, at least not in any proven, general sense. Researchers have made measurable progress towards safeguards that remain effective after an open-weight model is fine-tuned, but current evidence does not show that safety mechanisms can reliably survive a determined attempt to modify the model. This uncertainty matters because, once model weights are publicly released, users can retrain, redistribute and alter them without the original developer’s permission. Within debates about AI doom and existential risk, this raises a practical question: can safety features be embedded deeply enough that they cannot simply be trained away?
Recent research suggests that the answer is more nuanced than either optimists or pessimists sometimes claim. Standard safety fine-tuning is often surprisingly fragile, but new approaches have made safeguards substantially harder—not impossible—to remove. The evidence so far points towards an ongoing technical race rather than a solved problem.
Why ordinary safety training often disappears after fine-tuning
Most current language models acquire their refusal behaviour during post-training rather than during their original pre-training. After learning general language capabilities from enormous text corpora, developers perform additional training to encourage helpful behaviour while refusing certain harmful requests.
The difficulty is that this later stage of training often changes only a relatively small part of the model’s overall behaviour. When another party downloads the model and performs further fine-tuning, those learned refusal patterns may weaken or disappear. Multiple studies have shown that relatively modest additional training can substantially alter safety behaviour without noticeably reducing the model’s general usefulness.[ICLR Proceedings]proceedings.iclr.ccProceedings Tamper-Resistant Safeguards for Open-Weight LLMsICLR ProceedingsTamper-Resistant Safeguards for Open-Weight LLMs…
One striking finding is that this degradation does not necessarily require a malicious actor. Research has found that fine-tuning on ordinary task-specific datasets can unintentionally erode alignment to some degree, while deliberately designed datasets can remove much larger portions of a model’s refusal behaviour. This means the problem extends beyond obvious attempts to “jailbreak” an open-weight model.[arXiv]arxiv.orgOpen source on arxiv.org.
For discussions of open-weight AI and existential risk, this distinction is important. If safeguards disappear through routine customisation, relying solely on post-training safety becomes less attractive as a long-term defence.
What tamper-resistant research has achieved so far
The most prominent attempt to address this problem is research into tamper-resistant safeguards: techniques designed so that safety behaviour survives extensive weight modification.
A widely discussed example is the TAR (Tamper-Resistant Safeguards) approach presented at ICLR 2025. Rather than merely encouraging refusals during standard alignment training, the method attempts to distribute safety-related behaviour throughout the model in ways that remain difficult to overwrite. The researchers report that models retained much more of their safety behaviour after hundreds of fine-tuning steps than conventional safety training, while maintaining similar performance on ordinary tasks.[ICLR Proceedings]proceedings.iclr.ccProceedings Tamper-Resistant Safeguards for Open-Weight LLMsICLR ProceedingsTamper-Resistant Safeguards for Open-Weight LLMs…
This represents genuine technical progress. Previous work frequently showed safeguards disappearing after relatively little additional training, whereas TAR substantially increased the amount of retraining required before safety degraded.
However, the paper does not demonstrate permanent protection. Its evaluations cover particular attack methods, training budgets and benchmark tasks. They do not establish that future attackers—or future models—could not discover more effective ways to remove the safeguards. The authors themselves present the work as opening a research direction rather than solving the underlying problem.[ICLR Proceedings]proceedings.iclr.ccProceedings Tamper-Resistant Safeguards for Open-Weight LLMsICLR ProceedingsTamper-Resistant Safeguards for Open-Weight LLMs…
Could safety be embedded more deeply than post-training?
Some researchers argue that the most robust solution may be to reduce dangerous capabilities before post-training begins.
Recent work from the UK AI Security Institute and collaborators explored filtering selected dual-use knowledge during pre-training itself, rather than relying mainly on later refusal training. Their experiments found substantially greater resistance to adversarial fine-tuning—remaining resistant after far larger amounts of retraining than previous post-training approaches for the tested biological knowledge tasks.[aisi.gov.uk]aisi.gov.ukOpen source on aisi.gov.uk.
Even so, the results come with important limitations.
First, the experiments focus on particular categories of knowledge rather than general model behaviour. Second, the researchers found that filtered models could still make use of dangerous information when it was supplied through external tools or context during inference. In other words, reducing internal knowledge does not automatically eliminate downstream risk. The authors therefore argue for defence in depth rather than treating pre-training interventions as a complete solution.[aisi.gov.uk]aisi.gov.ukOpen source on aisi.gov.uk.
Why independent testing remains difficult
One reason this field remains unsettled is that tamper resistance is difficult to measure objectively.
Unlike conventional machine learning benchmarks, there is no universally accepted test for “successful removal” of safeguards. Researchers must decide:
- how much computing power the attacker receives;
- how many fine-tuning steps are permitted;
- what training data the attacker possesses;
- which harmful capabilities should be measured;
- what counts as preserving useful performance.
Different studies therefore produce results that are not always directly comparable.
This has motivated efforts to build common evaluation frameworks. Recent proposals such as TamperBench aim to standardise attack methods, evaluation metrics and robustness testing across different models and defence strategies, making it easier to compare competing approaches under similar conditions.[arXiv]arxiv.orgTamperBench: Systematically Stress-Testing LLM Safety Under Fine-Tuning and TamperingFebruary 6, 2026…
Without broadly accepted benchmarks, claims that a safeguard is “tamper resistant” should generally be interpreted as “more resistant than previous methods under tested conditions”, not “impossible to remove”.
What this means for AI doom arguments
Within AI doom discussions, tamper resistance is usually treated as a practical engineering question rather than an argument by itself.
Those concerned about existential risk often reason as follows:
- if highly capable open-weight systems become widely available;
- and if their safeguards can easily be removed;
- then developers lose much of their ability to reduce risks after release.
Improving tamper resistance could therefore reduce one important failure mode by making post-release modification more costly or less effective.
Critics, however, note that raising the cost of removing safeguards is different from making removal impossible. Given sufficient incentives, computing resources or future techniques, determined actors may still succeed. They therefore argue that tamper resistance should be viewed as one layer among many—including evaluations, controlled deployment, monitoring, governance and careful decisions about whether extremely capable models should be released openly in the first place.[ICLR Proceedings]proceedings.iclr.ccProceedings Tamper-Resistant Safeguards for Open-Weight LLMsICLR ProceedingsTamper-Resistant Safeguards for Open-Weight LLMs…
What current evidence does—and does not—show
The available evidence supports several reasonably strong conclusions.
There is good evidence that many conventional refusal mechanisms can be weakened through fine-tuning, sometimes surprisingly quickly. There is also credible evidence that newer techniques can make safeguards significantly more robust than earlier approaches.[ICLR Proceedings]proceedings.iclr.ccProceedings Tamper-Resistant Safeguards for Open-Weight LLMsICLR ProceedingsTamper-Resistant Safeguards for Open-Weight LLMs…
At the same time, there is no convincing evidence that current safeguards can reliably survive all determined modification attempts once an attacker controls the model weights. Existing demonstrations remain bounded by particular experimental settings, attack methods and computational budgets.
For that reason, researchers increasingly describe tamper resistance as an active research frontier rather than a solved capability. Progress has been real, but the central question—whether open-weight frontier models can retain meaningful safety guarantees after unrestricted fine-tuning—remains unresolved.
Amazon book picks
Further Reading
Books and field guides related to Can Open Weight Safeguards Survive Determined Modification?. Use these as the next step if you want deeper reading beyond the article.
The Alignment Problem
Finalist for the Los Angeles Times Book Prize A jaw-dropping exploration of everything that goes wrong when we build AI systems and the m...
Human Compatible
A leading artificial intelligence researcher lays out a new approach to AI that will enable us to coexist successfully with increasingly...
Superintelligence
This profoundly ambitious and original book picks its way carefully through a vast tract of forbiddingly difficult intellectual terrain.
Artificial Intelligence: A Modern Approach
Rating: 4.5/5 from 10 Google Books ratings
Artificial intelligence: A Modern Approach, 3e,is ideal for one or two-semester, undergraduate or graduate-level courses in Artificial In...
eBay marketplace picks
Marketplace Samples
Live-tested eBay searches with available results related to this page.
Selected fromcybersecurity pin oneBay.co.uk.
Endnotes
1.
Source: proceedings.iclr.cc
Title: Proceedings Tamper-Resistant Safeguards for Open-Weight LLMs
Link:https://proceedings.iclr.cc/paper_files/paper/2025/hash/fc49a629d33bc2461ed7a715ce44da68-Abstract-Conference.html
Source snippet
ICLR ProceedingsTamper-Resistant Safeguards for Open-Weight LLMs...
2.
Source: arxiv.org
Link:https://arxiv.org/abs/2310.03693
3.
Source: aisi.gov.uk
Link:https://www.aisi.gov.uk/research/deep-ignorance-filtering-pretraining-data-builds-tamper-resistant-safeguards-into-open-weight-llms
4.
Source: arxiv.org
Link:https://arxiv.org/abs/2602.06911
Source snippet
TamperBench: Systematically Stress-Testing LLM Safety Under Fine-Tuning and TamperingFebruary 6, 2026...
Published: February 6, 2026
5.
Source: proceedings.iclr.cc
Link:https://proceedings.iclr.cc/paper_files/paper/2024/hash/83b7da3ed13f06c13ce82235c8eedf35-Abstract-Conference.html
6.
Source: proceedings.iclr.cc
Link:https://proceedings.iclr.cc/paper_files/paper/2025/hash/9d3a4cdf6f70559e8c6fe02170fba568-Abstract-Conference.html
7.
Source: proceedings.iclr.cc
Link:https://proceedings.iclr.cc/paper_files/paper/2025/hash/1022661f3f43406065641f16ce25eafa-Abstract-Conference.html
8.
Source: proceedings.iclr.cc
Link:https://proceedings.iclr.cc/paper_files/paper/2025/hash/f3bfbd65743e60c685a3845bd61ce15f-Abstract-Conference.html
9.
Source: iclr.cc
Link:https://iclr.cc/virtual/2024/23105
10.
Source: proceedings.iclr.cc
Link:https://proceedings.iclr.cc/paper_files/paper/2025/hash/70de9e3948645a1be2de657f14d85c6d-Abstract-Conference.html
11.
Source: proceedings.iclr.cc
Link:https://proceedings.iclr.cc/paper_files/paper/2025/hash/d432fbe4877ee1a6a51632a18e69784f-Abstract-Conference.html
12.
Source: aisi.gov.uk
Link:https://www.aisi.gov.uk/research/open-technical-problems-in-open-weight-ai-model-risk-management
13.
Source: aisi.gov.uk
Link:https://www.aisi.gov.uk/blog/managing-risks-from-increasingly-capable-open-weight-ai-systems
14.
Source: tamper-resistant-safeguards.com
Link:https://www.tamper-resistant-safeguards.com/
Additional References
15.
Source: openreview.net
Link:https://openreview.net/forum?id=lZGgc845Uz
Source snippet
Deep Ignorance: Filtering Pretraining Data Builds Tamper-Resistant Safeguards into Open-Weight LLMs | OpenReviewOctober 15, 2025 — DEEP I...
Published: October 15, 2025
16.
Source: emergentmind.com
Link:https://www.emergentmind.com/papers/2602.06911
Source snippet
TamperBench: Systematically Stress-Testing LLM Safety Under Fine-Tuning and TamperingFebruary 6, 2026 — TAMPERBENCH: SYSTEMATICALLY STRES...
Published: February 6, 2026
17.
Source: anthropic.com
Title: Agentic [Misalignment]({{ ‘misalignment/’ | relative_url }}): How LLMs could be insider threats \ Anthropic
Link:https://www.anthropic.com/research/agentic-misalignment
Source snippet
June 20, 2025 — AGENTIC MISALIGNMENT: HOW LLMS COULD BE INSIDER THREATS Jun 20, 2025 HIGHLIGHTS * We stress-tested 16 leading models from...
Published: June 20, 2025
18.
Source: experts.illinois.edu
Title: tamper resistant safeguards for open weight llms
Link:https://experts.illinois.edu/en/publications/tamper-resistant-safeguards-for-open-weight-llms/
Source snippet
illinois.eduTAMPER-RESISTANT SAFEGUARDS FOR OPEN-WEIGHT LLMS - Illinois ExpertsApril 24, 2025 — TAMPER-RESISTANT SAFEGUARDS FOR OPEN-WEIG...
Published: April 24, 2025
19.
Source: youtube.com
Title: Stephen Casper
Link:https://www.youtube.com/watch?v=ADXO-mT8qIU
Source snippet
Safety Alignment Should be Made More Than Just a Few Tokens Deep (Paper Explained)...
20.
Source: alignment.anthropic.com
Title: alignment faking mitigations
Link:https://alignment.anthropic.com/2025/alignment-faking-mitigations/
Source snippet
Faking MitigationsDecember 16, 2025 — TOWARDS TRAINING-TIME MITIGATIONS FOR ALIGNMENT FAKING IN RL Towards Training-time Mitigations for...
Published: December 16, 2025
21.
Source: youtube.com
Link:https://www.youtube.com/watch?v=-r0XPC7TLzY
Source snippet
NDSS 2025 - Safety Misalignment Against Large Language Models...
22.
Source: youtube.com
Link:https://www.youtube.com/watch?v=5mFb1coDgLY
Source snippet
Two Ways to Break AI Safety — One Is Far More Dangerous...
23.
Source: deepignorance.ai
Link:https://deepignorance.ai/
24.
Source: mlanthology.org
Link:https://mlanthology.org/iclr/2024/qi2024iclr-finetuning/