Black-box images plant false memories in AI agents
New research shows a black-box, image-only attack can rewrite long-term memory in multimodal agents. The Lucid method crafts imperceptible perturbations that achieve around 60% success across five memory architectures, via in-context poisoning or out-of-context injection, steering recall and responses. Text filters barely help; light image preprocessing reduces but does not stop attacks.
Agents with long-term memory are having a moment. They file away your screenshots, receipts, and chat fragments, then pull them back later to keep the story straight. The catch: many of these systems treat stored images as ground truth. That habit, it turns out, is easy to exploit.
A new black-box attack called Lucid shows how to slip a doctored photo into the family album without anyone noticing, then let the album gaslight the agent months later. The setup is severely constrained: image only, no access to the target model internals, no help from the text channel. Yet it works across a spread of memory designs with attack success rates around 61.6% for in-context poisoning and 58.4% for out-of-context injection.
How Lucid breaks memory
Lucid starts by picking a target image-and-caption pair that contradicts the source. Think “healthy snack” targeted toward “chocolate cake.” Then it prepares a payload in one of two modes. Memory poisoning replaces an existing, text-supported image in a past turn while leaving the text intact, so the written record endorses the lie. Memory injection inserts a new image-bearing turn where the accompanying text is neutral, giving the lie no textual friction at write time.
The craft is all in the perturbation. Lucid optimises imperceptible pixel noise so the source image’s embedding drifts toward the target. It uses a surrogate ensemble from the CLIP family, a feature-alignment objective extended with a text-alignment term, and projected I-FGSM under an ℓ∞ budget of 16/255 for 1000 steps. No peeking at the victim’s retrieval encoder or Multimodal Large Language Model (MLLM). Despite that, the adversarial image is retrieved reliably at recall time, often rivaling oracle retrieval.
What happens next depends on the generator. Some Large Language Models (LLMs) and MLLMs fetch the poisoned memory but don’t fully act on it; others treat the injected content as gospel. The team saw transfer hold up even when crossing from CLIP-based surrogates to GME encoders. Across five memory backends, including graph-structured and LLM-summarised systems such as MuRAG, NGMemory, AUGUSTUS, UniversalRAG, and Mem0Memory, the pattern held. Tests ran on the Mem-Gallery multi-session benchmark with models including GPT-4 variants, Claude-Haiku, and Gemini.
Out-of-context injection is the sneakiest: the neutral text at storage time gives no contradiction to trip alarms, so the false memory lies dormant until retrieval. Text-side filters at write or read time barely register these attacks. Cruder image preprocessing like Gaussian blur or JPEG compression knocks down success rates but does not close the hole, and stricter pipelines that describe images via an LLM at write time change the odds but do not erase the class of risk.
If this all sounds familiar, it should. This is cache poisoning for the age of multimodal memory: rewrite what the system will believe tomorrow by nudging what it files today. The open question is whether we treat images as trusted artefacts or as noisy signals to be cross-examined at the moment of recall. That decision will shape the next generation of agents more than any model size bump.
Additional analysis of the original ArXiv paper
📋 Original Paper Title and Abstract
Do Agents Dream of False Memories? Black-box Visual Attacks on Long-term Memory in Multimodal AI Agents
🔍 ShortSpan Analysis of the Paper
Problem
The paper studies whether imperceptible, black-box image perturbations can corrupt persistent long-term memory in multimodal AI agents that store and later retrieve image-bearing conversation turns. This matters because many deployed agents rely on visual records as trusted grounding for future responses, so undetected corruption of stored images can create persistent false beliefs that affect safety-critical advice and user trust.
Approach
The authors introduce Lucid, a strictly image-bounded, black-box attack pipeline that requires no access to the target MLLM, retrieval encoder, memory internals, or the text channel. Lucid has three stages: target design, which selects semantically contradictory real-world targets from a large image-caption pool; payload construction, which prepares either in-context poisoning (replace an existing image while leaving text intact) or out-of-context injection (add a new image-bearing turn with neutral text); and adversarial perturbation, which optimises imperceptible pixel noise on a surrogate ensemble to steer the source image embedding toward the chosen target. Perturbations use a feature-alignment objective extended with a text-alignment term and are generated by projected I-FGSM under an ℓ∞ budget (ε=16/255, 1000 steps) over CLIP-family surrogates. Evaluation uses the Mem-Gallery multi-session benchmark, five diverse memory backends (MuRAG, NGMemory, AUGUSTUS, UniversalRAG, Mem0Memory) and several MLLMs including GPT-4 variants, Claude-Haiku, and Gemini.
Key Findings
- Lucid achieves high attack success rates across architectures, with mean ASR 61.6% for in-context poisoning and 58.4% for out-of-context injection, showing the attack broadly transfers under black-box conditions.
- Adversarial images are reliably retrieved, with retrieval rates often approaching oracle upper bounds, indicating perturbations effectively displace genuine memories at recall time across retrieval encoders.
- Generation-stage susceptibility is distinct from retrieval susceptibility: some MLLMs retrieve adversarial entries but resist acting on them, while others act on injected content more readily; attacks generalise across tested MLLMs and encoders including GME, CLIP variants and cross-encoder transfer to GME remained substantial.
- Out-of-context injection is intrinsically harder to detect because neutral user text provides no contradiction at storage time, allowing dormant false memories to persist until retrieved.
- Simple text-based write-time or retrieval-time filters are largely ineffective; image-level preprocessing (Gaussian blur, JPEG) reduces ASR substantially but does not fully mitigate the threat and may be bypassed by stronger adversarial techniques.
Limitations
Results are tied to the chosen surrogate ensemble, the ShareGPT4V-100K candidate pool for target selection, and the specific ℓ∞ perturbation budget and optimisation settings. Some backends (notably Mem0Memory) impose stricter requirements because they pass images through a describer MLLM at write time, and defence efficacy is architecture-dependent. The tested image preprocessing defences may not withstand more advanced or specialised adversarial methods.
Implications
Offensively, an attacker who can tamper with images provided to an agent (for example via a compromised CDN, SDK, or web content) can plant persistent false memories that steer future agent behaviour: misattribute identities, alter safety or allergy guidance, fabricate events or credentials, or otherwise influence downstream actions. Because the attack is black-box and image-only, it expands the realistic threat surface for deployed multimodal agents and highlights the need for image-level provenance, cross-modal consistency checks at storage time, and adversarial robustness evaluation of memory pipelines.
