Within Memory Attacks
How One Malicious Prompt Can Survive for Weeks
A single malicious document can influence later tasks when an agent stores its instructions or claims as trusted long-term memory.
On this page
- From temporary injection to persistent memory
- How poisoned memories gain authority over time
- When false beliefs begin shaping real actions
Page outline Jump by section
Introduction
A prompt injection is usually thought of as a temporary attack: an AI agent reads malicious instructions from a webpage, document or email, behaves incorrectly during that task, and then the problem disappears when the conversation ends. Persistent memory changes that assumption. If an agent decides that attacker-controlled information is important enough to save as long-term memory, a one-off attack can become a lasting false belief that influences future work for days, weeks or even longer. This possibility matters because many modern AI agents are designed to remember user preferences, ongoing projects and lessons from previous interactions. The same feature that improves continuity also creates a new path for compromise if the system cannot distinguish trustworthy memories from manipulated ones. Researchers and security organisations increasingly describe persistent memory as a distinct attack surface rather than merely an extension of ordinary prompt injection.[openai.com]OpenAIdesigning agents to resist prompt injectionDesigning AI agents to resist prompt injection | OpenAIMarch 11, 2026…
From temporary injection to persistent memory
The critical step is not the prompt injection itself. It is the decision to convert information from an untrusted source into something the agent later treats as trusted memory.
A simplified sequence looks like this:
- An agent opens an external resource as part of a legitimate task.
- The resource contains hidden or misleading instructions directed at the AI rather than the human reader.
- Instead of ignoring those instructions, the model concludes that part of the content should be remembered.
- The information is written into persistent storage, such as a memory database, profile, vector store or behavioural preference file.
- Later tasks retrieve that stored memory automatically, often without revisiting the original source.
The important distinction is that the malicious document no longer needs to remain available. The dangerous information has migrated into the agent’s own internal knowledge store. Future sessions may therefore appear to be acting independently when they are actually following instructions that originated from an attacker long ago.[openai.com]OpenAIdesigning agents to resist prompt injectionDesigning AI agents to resist prompt injection | OpenAIMarch 11, 2026…
This differs from ordinary hallucination. A hallucination is generally transient and may disappear if the model is asked again. A poisoned memory can instead become a stable input into future reasoning.
How poisoned memories gain authority over time
Persistent memories often receive greater trust than newly encountered information because they are intended to represent accumulated knowledge about the user or the agent’s previous work.
Several mechanisms can gradually increase the influence of a false memory.
Automatic retrieval
Many agent architectures retrieve memories that appear relevant to the current task before the model begins reasoning. If a poisoned memory matches the new request, it can be presented alongside legitimate user preferences without distinguishing where it came from. The model may therefore treat it as established context rather than fresh external input.[arXiv]arxiv.orgOpen source on arxiv.org.
Repeated reinforcement
If an agent repeatedly consults the same false memory, later summaries or updates may restate it in new wording. Over multiple sessions, the original malicious instruction becomes harder to identify because it has been rewritten into the agent’s own summaries rather than copied directly from the attacker.
Higher perceived credibility
Some systems deliberately prioritise stored memories because they are assumed to have survived earlier filtering. Once attacker-controlled content crosses that trust boundary, the model may give it disproportionate weight compared with contradictory evidence found later. OWASP describes this as a shift from protecting prompts alone to protecting memory as a security-critical state.[OWASP Gen AI Security Project]genai.owasp.orgGen AI Security Project Memory Is a Feature. It Is Also an Attack SurfaceOWASP Gen AI Security ProjectMemory Is a Feature. It Is Also an Attack Surface - OWASP Gen AI Security Project…
When false beliefs begin shaping real actions
The most important risk is not that an agent remembers an incorrect fact. It is that the false memory begins affecting decisions.
Depending on the agent’s capabilities, a poisoned memory might:
- misidentify trusted or untrusted websites;
- incorrectly remember user preferences;
- alter future planning priorities;
- influence which tools are selected;
- bias document summaries or recommendations;
- encourage unsafe automation decisions.
For example, imagine an assistant that permanently remembers “documents from this repository are always trustworthy” because a malicious repository instructed it to store that belief. Future work involving that repository may receive less scrutiny precisely because the compromise has already become part of the agent’s trusted background knowledge.
Equally concerning are fabricated memories about the user. A malicious webpage might attempt to convince the assistant that the user prefers a certain supplier, has authorised particular actions or wishes to bypass security checks. If those claims become stored memories, later sessions may rely on them without the user ever repeating them. Experimental research on stateful assistants has demonstrated that externally planted false memories can later be retrieved and influence subsequent conversations and agent behaviour, although success rates depend heavily on the architecture, model and defence mechanisms involved.[arXiv]arxiv.orgarXiv Hidden in Memory: Sleeper Memory Poisoning in LLM AgentsarXiv Hidden in Memory: Sleeper Memory Poisoning in LLM Agents
Why this matters in AI doom discussions
Within debates about AI doom and existential risk, memory poisoning is not usually presented as an extinction mechanism by itself. Rather, it illustrates how increasingly autonomous systems may accumulate hidden failures over time.
The concern is that future highly capable agents may:
- perform long-running projects across weeks or months;
- coordinate with multiple external services;
- update their own knowledge continually;
- make decisions using memories collected from thousands of previous interactions.
In that setting, temporary manipulation becomes much more valuable to an attacker. A single successful compromise may continue influencing behaviour long after the original attack disappears.
Researchers who worry about loss of control argue that increasing autonomy makes persistent internal state increasingly important. If memory can silently accumulate incorrect goals, false assumptions or misleading operating procedures, supervision becomes more difficult because later behaviour may appear internally consistent even though it originated from attacker-controlled information rather than genuine user intent. This concern forms part of broader discussions about alignment and maintaining reliable human control over increasingly capable agents, although there is little evidence that present-day memory poisoning alone creates existential risk. The argument instead concerns how these mechanisms could become more consequential as agent autonomy grows.[openai.com]OpenAIdesigning agents to resist prompt injectionDesigning AI agents to resist prompt injection | OpenAIMarch 11, 2026…
What evidence exists today?
The basic mechanism is no longer purely theoretical.
Security researchers have demonstrated that persistent memory creates attack paths that differ from conventional prompt injection. Recent academic work has shown that malicious entries already present in an agent’s memory can continue influencing behaviour across later sessions, even when the original triggering interaction has ended. Other studies have examined “sleeper” memory attacks, in which apparently harmless stored memories remain dormant until future tasks activate them.[arXiv]arxiv.orgOpen source on arxiv.org.
Industry guidance has moved in the same direction. OpenAI recommends treating prompt injection less like a simple input-filtering problem and more like social engineering, emphasising architectural controls that limit the consequences even if manipulation succeeds. Likewise, OWASP now identifies memory and context poisoning as a distinct security problem for agentic systems rather than simply another example of prompt injection.[openai.com]OpenAIdesigning agents to resist prompt injectionDesigning AI agents to resist prompt injection | OpenAIMarch 11, 2026…
Evidence remains incomplete, however. Most published demonstrations use controlled experimental environments rather than large-scale deployments. Attack success varies substantially across models, memory designs and retrieval strategies, suggesting that implementation details matter as much as the underlying language model.[arXiv]arxiv.orgOpen source on arxiv.org.
Why memory requires different defences
Stopping prompt injection at the moment it appears is helpful but insufficient once persistent memory exists.
Additional protections focus on preventing untrusted information from becoming durable knowledge in the first place. Common approaches include:
- requiring stricter validation before writing long-term memories;
- separating user preferences from externally collected information;
- recording the provenance of each memory so later retrieval includes its source;
- allowing review, rollback or deletion of suspicious memories;
- limiting which kinds of information an autonomous agent may permanently store without explicit approval.
The broader design principle is that persistent memory should be treated as a protected asset rather than an ordinary conversation log. Once an attacker can influence what an agent remembers about the world, the problem shifts from temporary prompt manipulation to a continuing corruption of the agent’s future reasoning.[openai.com]OpenAIdesigning agents to resist prompt injectionDesigning AI agents to resist prompt injection | OpenAIMarch 11, 2026…
Amazon book picks
Further Reading
Books and field guides related to How One Malicious Prompt Can Survive for Weeks. Use these as the next step if you want deeper reading beyond the article.
Adversarial AI Attacks, Mitigations, and Defense Strategies
Understand how adversarial attacks work against predictive and generative AI, and learn how to safeguard AI and LLM projects with practic...
The Alignment Problem: Machine Learning and Human Values
Finalist for the Los Angeles Times Book Prize A jaw-dropping exploration of everything that goes wrong when we build AI systems and the m...
The Language of Deception: Weaponizing Next Generation AI
A penetrating look at the dark side of emerging AI technologies In The Language of Deception: Weaponizing Next Generation AI, artificial...
Thinking in Systems: A Primer
Rating: 4.3/5 from 30 Google Books ratings
First published 2008. Subjects: critical thinking, systems thinking, systems dynamics, complex systems, policy.
eBay marketplace picks
Marketplace Samples
Live-tested eBay searches with available results related to this page.
Selected fromcybersecurity poster oneBay.co.uk.
Endnotes
1.
Source: OpenAI
Title: designing agents to resist prompt injection
Link:https://openai.com/index/designing-agents-to-resist-prompt-injection/
Source snippet
Designing AI agents to resist prompt injection | OpenAIMarch 11, 2026...
Published: March 11, 2026
2.
Source: genai.owasp.org
Title: Gen AI Security Project Memory Is a Feature. It Is Also an Attack Surface
Link:https://genai.owasp.org/2026/05/13/memory-is-a-feature-it-is-also-an-attack-surface/
Source snippet
OWASP Gen AI Security ProjectMemory Is a Feature. It Is Also an Attack Surface - OWASP Gen AI Security Project...
3.
Source: arxiv.org
Link:https://arxiv.org/abs/2606.04329
4.
Source: arxiv.org
Title: arXiv Hidden in Memory: Sleeper Memory Poisoning in LLM Agents
Link:https://arxiv.org/abs/2605.15338
5.
Source: arxiv.org
Link:https://arxiv.org/abs/2607.14611
6.
Source: owasp.org
Title: Foundation OWASP Agent Memory Guard | OWASP Foundation
Link:https://owasp.org/www-project-agent-memory-guard/
Source snippet
OWASP FoundationOWASP Agent Memory Guard | OWASP Foundation...
7.
Source: arxiv.org
Link:https://arxiv.org/abs/2607.14651
8.
Source: cheatsheetseries.owasp.org
Title: Cheat Sheet Series LLM Prompt Injection Prevention
Link:https://cheatsheetseries.owasp.org/cheatsheets/LLM_Prompt_Injection_Prevention_Cheat_Sheet.html
Source snippet
OWASP Cheat Sheet SeriesLLM Prompt Injection Prevention - OWASP Cheat Sheet Series...
9.
Source: OpenAI
Title: anthropic safety evaluation
Link:https://openai.com/index/openai-anthropic-safety-evaluation/
10.
Source: owasp.org
Title: MCP10 2025–ContextInjection&OverSharing
Link:https://owasp.org/www-project-mcp-top-10/2025/MCP10-2025%E2%80%93ContextInjection%26OverSharing
11.
Source: cheatsheetseries.owasp.org
Title: AI Agent Security Cheat Sheet
Link:https://cheatsheetseries.owasp.org/cheatsheets/AI_Agent_Security_Cheat_Sheet.html
Additional References
12.
Source: alphaxiv.org
Link:https://www.alphaxiv.org/abs/2606.04329
Source snippet
From Untrusted Input to Trusted Memory: A Systematic Study of Memory Poisoning Attacks in LLM Agents | alphaXivJune 18, 2026 — FROM UNTRU...
Published: June 18, 2026
13.
Source: alphaxiv.org
Link:https://www.alphaxiv.org/abs/2607.14611
Source snippet
Bad Memory: Evaluating Prompt Injection Risks from Memory in Agentic Systems | alphaXivJuly 16, 2026 — BAD MEMORY: EVALUATING PROMPT INJE...
Published: July 16, 2026
14.
Source: youtube.com
Title: Secure Microsoft AI Agents Against Prompt Injection
Link:https://www.youtube.com/watch?v=R_dj3XnS70w
Source snippet
AI Agent Security Prompt Injection Memory Poisoning AI Agent Security Explained: Prompt Injection, Memory Poisoning & Defense in Depth sc...
15.
Source: youtube.com
Link:https://www.youtube.com/watch?v=QJDqCXBaV0c
Source snippet
AI Agent Hijack Explained: How to Prevent Prompt Injection Attacks...
16.
Source: youtube.com
Title: AI Agent Hijack Explained: How to Prevent Prompt Injection Attacks
Link:https://www.youtube.com/watch?v=EIUAN_twbus
Source snippet
How Your AI Agent Gets Hijacked: Prompt Injection Explained...
17.
Source: youtube.com
Title: How Your AI Agent Gets Hijacked: Prompt Injection Explained
Link:https://www.youtube.com/watch?v=Uz8CfV4SkdY
Source snippet
Panel: Securing Autonomous Agents & Threats | Future of Data and AI...
18.
Source: youtube.com
Title: Panel: Securing Autonomous Agents & Threats | Future of Data and AI
Link:https://www.youtube.com/watch?v=JnKlbNkk7gU
Source snippet
Secure Microsoft AI Agents Against Prompt Injection...
19.
Source: armalo.ai
Link:https://www.armalo.ai/labs/research/2026-05-10-memory-poisoning-persistent-context-attack-surface
20.
Source: agentthreatrule.org
Title: ATR 2026 00075
Link:https://agentthreatrule.org/en/rules/ATR-2026-00075
21.
Source: github.com
Title: Git Hub
Link:https://github.com/OWASP/www-project-agent-memory-guard