Benign memories combine to jailbreak self-evolving agents

Problem

This paper studies a novel safety risk in self-evolving large language model agents that persistently distil interaction trajectories into reusable experiences. The central concern is that individually benign experiences, when accumulated and reused across sessions, can jointly weaken an agent's refusal behaviour and safety boundary. This attack surface differs from prior memory attacks because it operates without direct write access and without producing overtly malicious records, making attacks stealthier and applicable in more realistic settings.

Approach

The authors propose EvoBreak, an experience-conditioned sequential attack that adaptively submits benign tasks, observes the victim's distilled experiences, and replans to acquire complementary experiences until sufficient coverage is reached. EvoBreak then reformulates the safety-sensitive target query to align with the accumulated experiences so that they are jointly activated. To train and evaluate this approach they introduce BreakGym, a structure-first synthesis pipeline that generates decomposable safety-sensitive targets by sampling decomposition principles (knowledge, procedural, constraint, functional), composition topologies (sequential, parallel, hierarchical, conditional), and safety domains (privacy, fraud, chemical, cyber, violence, sexual). Training uses a two-stage cascaded workflow: rejection-sampling supervised fine-tuning to learn from successful benign trajectories, followed by Hint-guided Group Relative Policy Optimisation to improve exploration under sparse trajectory-level rewards using training-time hints derived from BreakGym scaffolds. Evaluation uses two self-evolving frameworks and multiple victim backbones, pre-evolution domains, and safety benchmarks, measuring Attack Success Rate and Benignness.

Key Findings

• EvoBreak achieves consistently stronger attack performance than state-of-the-art baselines across settings, with an overall average Attack Success Rate of 86.12%, outperforming the strongest baseline by 24.19 percentage points.
• EvoBreak maintains high benignness while remaining effective; its design and training explicitly filter and reward benign intermediate interactions, yielding a superior effectiveness–benignness trade-off compared with query-level jailbreaks and direct memory-poisoning attacks.
• The attack is robust to heterogeneous self-evolution mechanisms, victim model backbones, and diverse pre-evolution memories: it works on both evaluated frameworks and across mathematics, code, and general-reasoning pre-evolution domains.
• Ablations show that experience-conditioned replanning, the combination of experience acquisition and target reformulation, and the two-stage optimisation pipeline each materially contribute to success.
• Experiments manipulating the fraction of retained attack experiences indicate that success typically requires joint availability of multiple induced experiences rather than a single decisive memory, confirming that benign composition underlies the attack effect.

Limitations

The attack model assumes a grey-box adversary that can submit tasks and read the experiences distilled from those interactions but cannot directly edit the memory, system prompt, or extraction mechanism. Training relies on synthetic latent scaffolds and a training-time hint critic that are unavailable at inference. Evaluation uses specific agent frameworks, model backbones, pre-evolution domains, and LLM-based evaluators, so results reflect those experimental conditions and the chosen benignness auditor.

Implications

Offensively, an adversary with only query submission and read-only observability can stealthily erode an agent's safety by sequentially inducing harmless-looking experiences that, when composed, enable harmful compliance to a later safety-sensitive request. Because EvoBreak adapts to the victim's actual distilled experiences and reformulates the final query to activate them jointly, such attacks can succeed across different agent architectures and background memories without privileged access. This reveals a persistent attack surface in systems that reuse internal experiences across sessions and highlights how compositional interactions among benign artefacts can be exploited to bypass refusal mechanisms.