Adaptive attackers break most LLM safety defences
Reviewing The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against Llm Jailbreaks and Prompt Injections · Milad Nasr, Nicholas Carlini, Chawin Sitawarin et al.
Researchers turn tuned, well-resourced attacks on twelve recent jailbreak and prompt injection defences, and most fail the large majority of the time despite originally reporting near-zero success rates. The defences were tested against attacks never designed to beat them, so their published robustness says little about a determined attacker.
Twelve recent defences against jailbreaks and prompt injection went into this study. Most had originally reported attack success rates near zero. Against the researchers' attacks, most of them were bypassed more than 90 percent of the time.
That gap is the whole story, and it is not about a clever new exploit. It is about who moves first. A defence is usually tested against a fixed set of known harmful prompts, or against an optimiser built without that defence in mind. So the defence is shaped around attacks it has already seen. Of course it does well.
The researchers reverse the order. Their attacker knows the defence is there and tunes its strategy against it. The method is a simple loop. Generate candidate attack strings, score how close each gets to the goal, keep the best, change them and try again. They drove that loop with gradient descent, reinforcement learning, a search where a second model proposes variations, and human red teamers. Each got a generous compute budget.
The defences cover the usual approaches. Some add stronger instructions to the prompt, reminding the model what the user actually asked for. Some retrain the model to refuse harmful requests or to ignore instructions hidden in data. Others put a classifier in front, meant to spot injected text before the model acts on it.
None of these categories held. Search broke the prompt-based defences. Reinforcement learning broke the retrained models, reaching 100 percent against one refusal-training approach. Commercial and open detectors were bypassed too.
The humans did better than the machines
What surprised me is how well the people did. In several settings, human red teamers beat the automated methods. A person reads a refusal, works out why it happened, and rewrites the attack to step around it. That is exactly the adaptive behaviour the original evaluations never modelled. The lesson is old in security: a control is only as good as the attacker it was tested against.
The preconditions matter. The attacker needs feedback, at minimum the ability to submit inputs and read outputs many times. Some settings also gave access to the model's internal scores or full weights. The budgets were large, and the researchers note that production constraints such as rate limits may change the cost of an attack. Even so, the most constrained setting, outputs only, is the one most real attackers have.
For practitioners, the practical change is in how you read vendor claims. A near-zero attack success rate means little until you know what attacked it. I treat these defences as friction, not a boundary. Build the system so that a successful injection cannot reach much.
The defences were measured against attackers who moved first. Real ones move second.
Further analysis
Problem
Current evaluations of LLM safety against jailbreaks and prompt injections rely on static attack strings or on computationally weak optimisation methods not designed with defenses in mind. This approach can give a false sense of security, because adaptive attackers who invest substantial resources to optimise their objective can bypass a wide range of recent defences. The paper argues that future work must evaluate robustness against strong adaptive attackers to produce credible claims, emphasising the real world risks of manipulation, knowledge leakage and misuse in deployed systems, and the need for multi layer guardrails and production monitoring to reduce these risks.
Approach
The authors present a general adaptive attack framework that unifies common prompting attack structures and operates as a four step loop: score, select, update and execute. They instantiate four forms of attacks gradient based, reinforcement learning, search based and human red teaming. Adversaries are considered under three access settings whitebox, blackbox with logits and blackbox generation only, all with substantial compute budgets to reflect worst case scenarios. The study evaluates twelve defences spanning prompting based techniques, training based methods, detectors and canaries, using HarmBench for jailbreaks and AgentDojo and OpenPromptInject type benchmarks for prompt injections, with Data Sentinel and MELON as additional cases. A MAP Elites based controller manages an island style search and uses a mutator LLM to propose new triggers; scoring feedback is provided by a critic model and textual responses, guiding subsequent iterations. The work also includes a large scale human red teeming study, showing that humans can outperform automated methods in several settings and that robust evaluation should incorporate human attackers alongside automated ones.
Key Findings
- Adaptive attacks achieve attack success rates above 90 percent for most of the twelve defences tested, calling into question many previously reported near zero risk figures.
- Prompting based defences such as Spotlighting and Prompt Sandwiching, which rely on stronger prompts to preserve user intent, fail against adaptive search based attacks with attack success rates above 95 percent; human red teaming also yields many successful injections, indicating these defences do not generalise to adaptive attackers.
- RPO and other gradient based or RL driven attacks undermine jailbreak and prompt injection strategies, with RL yielding about 98 percent ASR and gradient based around 96 percent on the applicable benchmarks.
- Training based defences such as Circuit Breakers, StruQ and MetaSecAlign show limited robustness under adaptive evaluation; RL based attacks attain 100 percent ASR against Circuit Breakers in HarmBench, and RL based attacks reach 96 percent on StruQ and MetaSecAlign in agentic settings.
- Detectors and canaries such as Protect AI, PromptGuard, PIGuard and Model Armor are also bypassed by adaptive attacks; ASR commonly exceeds 90 percent for several detectors, with PIGuard at 71 percent in one configuration. Human red teeming achieves higher effectiveness, confirming that expert adversaries remain a strong threat against detectors.
- Data Sentinel and MELON can be defeated by adaptive approaches; RL assaults raise Data Sentinel ASR to near 98 percent and MELON to 76 percent, rising to 95 percent with stronger attacker knowledge; human red teeming achieves substantial success as well.
- The authors highlight that automated scoring can be vulnerable to reward hacking and mis aligned objectives, and that scoring functions should align with real world success criteria to avoid false conclusions.
- Overall the study argues that static benchmarks and fixed prompt sets provide an incomplete picture; robust evaluation requires adaptive evaluation procedures and multi method red teaming to reveal weaknesses.
- The authors advocate for stronger, more realistic benchmarks and production level monitoring, as well as openness to enable red teaming and disclosure of evaluation tools to raise the security standard in both industry and academia.
Limitations
The authors acknowledge that robustness numbers are not directly comparable across defences due to differing benchmarks and configurations, and that results are context dependent. Evaluations rely on large compute budgets and may not reflect real world production constraints. While automated attacks show promise, they are not yet universally reliable and cannot replace human expertise; some evaluations are limited to specific benchmarks and threat models, so generalisation remains uncertain.
Why It Matters
The work demonstrates that many current LLM safety defences are brittle under adaptive, well funded attacks, underscoring the need for comprehensive evaluation against adaptive adversaries and scalable optimisation techniques. Practically this means adopting multi layer guardrails, red teeming, stronger benchmarks, and production monitoring to improve resilience. The societal implications are that reliable safeguards are essential to prevent manipulation, harmful instruction leakage and misuse in deployed systems, and that the security community should encourage transparent evaluation tools and collaborative testing to elevate safety standards.
The paper title, authors and abstract
The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against Llm Jailbreaks and Prompt Injections
How should we evaluate the robustness of language model defenses? Current defenses against jailbreaks and prompt injections (which aim to prevent an attacker from eliciting harmful knowledge or remotely triggering malicious actions, respectively) are typically evaluated either against a static set of harmful attack strings, or against computationally weak optimization methods that were not designed with the defense in mind. We argue that this evaluation process is flawed. Instead, we should evaluate defenses against adaptive attackers who explicitly modify their attack strategy to counter a defense's design while spending considerable resources to optimize their objective. By systematically tuning and scaling general optimization techniques-gradient descent, reinforcement learning, random search, and human-guided exploration-we bypass 12 recent defenses (based on a diverse set of techniques) with attack success rate above 90% for most; importantly, the majority of defenses originally reported near-zero attack success rates. We believe that future defense work must consider stronger attacks, such as the ones we describe, in order to make reliable and convincing claims of robustness.
