AdvGRPO trains adaptive LLM jailbreaks and tougher defenders
A new co-training framework, AdvGRPO, stabilises Group Relative Policy Optimisation for adaptive red teaming. It trains a multi-turn attacker that reaches about 90–91% success against GPT-4.1 and transfers at 82–93% to other models. Co-trained defenders cut HarmBench attack rates to under 2% while keeping general utility.
Static jailbreak lists are yesterday’s news. The interesting work now is fully adaptive: attacker models that probe, watch how a defender Large Language Model (LLM) responds, and escalate until something gives. This study puts that on rails with AdvGRPO, a co-training scheme that finally makes Group Relative Policy Optimisation stable for attacker–defender training.
Mechanically, the attacker runs closed-loop. It prompts, reads the defender’s reply, then updates its strategy over several turns. Rewards are dense and multi-channel: one scores the defender’s response for harmful leakage, another scores the attacker’s prompt quality, a third optionally scores the attacker’s “thinking” trace for reasoning models, and a helpfulness channel keeps benign behaviour alive. An LLM judge scores each channel on a 0–1 scale. Crucially, each channel’s advantage is normalised independently so signals don’t drown each other when objectives shift. Training follows a curriculum: single-turn attacker training, then multi-turn, then alternating updates where attacker and defender take turns learning every N steps. Episodes are cut early when the attack reward passes 0.9. The team used LoRA adapters on Qwen-family models for both sides, with GPT-4.1 acting as judge and as a strong target during attacker-only phases.
The outcomes are stark. A Qwen2.5-14B multi-turn attacker lands about 90–91% success on held-out safety tests against GPT-4.1. Reasoning-capable attackers jump from near-zero to 71–79% single-turn success when given a thinking-trace reward, suggesting the method overcomes self-censoring. Attacks transfer well: 82–93% success across three held-out defenders, often beating a strong multi-turn baseline. Co-trained defenders, in turn, drive HarmBench attack success well under 2%, versus 18.8% for the base model, 16.8% for one prior method, and 4.7% for another, while preserving or improving general utility on knowledge and reasoning tasks.
The offensive story matters. Simply un-aligning a model doesn’t make it a good attacker; explicit adversarial training teaches strategy, not just refusal removal. The closed loop is the edge: the attacker treats the defender’s refusal as a clue, not a stop sign, and refactors the next move accordingly. Early pruning rewards decisive breaches and keeps the search focused on fruitful lines of attack.
There are caveats. Defenders lost some benign compliance, likely because benign prompts in training were too plain. Attackers showed entropy collapse, with less strategy diversity even when token-level entropy was nudged. The defender pool was limited and turn horizons short. For those of us thinking about governance, this pushes on two fronts: evaluations need to be adaptive and multi-turn, and relying on a single LLM judge bakes in a failure mode. If compact models can be trained to mass-produce transferable jailbreaks, the policy question is not whether to permit red teaming like this, but how to require it, measure it, and avoid gaming of the metrics.
Additional analysis of the original ArXiv paper
📋 Original Paper Title and Abstract
Learning to Attack and Defend: Adaptive Red Teaming of Language Models via GRPO
🔍 ShortSpan Analysis of the Paper
Problem
The paper studies adaptive red teaming for large language models, addressing the arms race between attackers that craft adversarial prompts and defenders that reject or deflect them. Static datasets and fixed jailbreak techniques leave models brittle to adaptive strategies. Prior co-training work used PPO or DPO and reported instability when applying GRPO, limiting use of a sample-efficient RL method that can steer safety-relevant behaviour. Stabilising GRPO for joint attacker–defender optimisation matters because it could uncover novel, transferable attacks and produce stronger defenders through continual adaptation.
Approach
The authors introduce AdvGRPO, a co-training framework that makes Group Relative Policy Optimisation viable for attacker–defender training by combining dense multi-channel rewards, per-channel advantage normalisation (GDPO), curriculum pre-training, and alternating updates. Training proceeds from single-turn attacker pre-training to closed-loop multi-turn attacks before bootstrapped co-training in which attacker and defender are updated in alternation every N steps. Attacks are closed-loop: the attacker observes defender replies each turn, receives per-turn attack rewards, and episodes are pruned early when an attack reward exceeds 0.9. Reward channels include an attack reward scoring the defender response, a prompt reward scoring the attacker prompt, an optional thinking-trace reward for reasoning attackers, and a helpfulness reward for benign objectives. All rewards are scored on a 0–1 scale by an LLM judge and independent channel normalisation prevents signal collapse when objectives shift. Models were implemented as LoRA adapters and experiments used Qwen-family models for attacker and defender and GPT-4.1 as judge and a strong target during attacker-only training. Evaluation used attack success rate and standard safety and utility benchmarks.
Key Findings
- AdvGRPO yields large attack gains: a Qwen2.5-14B multi-turn attacker achieves about 90–91% attack success rate on held-out safety test splits against GPT-4.1, substantially improving over base models.
- Reasoning-capable attackers improve from near-zero to 71–79% single-turn success, showing the thinking-trace reward can overcome reasoning-model self-censoring.
- Attacks transfer well to unseen models: the multi-turn Qwen2.5-14B attacker attains 82–93% transfer ASR across three held-out defenders, often outperforming a state-of-the-art multi-turn baseline.
- Co-trained defenders outperform prior methods: AdvGRPO defenders reduce HarmBench ASR to well under 2%, compared with 18.8% for the base model, 16.8% for one prior method and 4.7% for another, while preserving or improving general utility on knowledge and reasoning benchmarks.
- Unaligned or uncensored models do not automatically make effective attackers; explicit attacker training is necessary to learn adversarial strategy rather than merely remove refusals.
Limitations
Co-trained defenders showed reduced benign compliance, attributed to using only vanilla benign prompts during training. Attackers exhibited entropy collapse and reduced strategy diversity; adaptive entropy regularisation helped token-level variability but was hard to tune and did not reliably produce structural diversity. Experiments used a limited pool of defenders and relatively short multi-turn horizons, so broader defender diversity and longer interactions remain to be evaluated.
Implications
Offensive implications are clear: AdvGRPO can train compact models that generate highly effective, transferable jailbreaks and multi-turn escalation strategies that adapt to defender replies. Attackers trained this way are more capable than merely unaligned models, meaning red-team models need explicit adversarial training to probe weaknesses. Co-training also produces stronger defenders but can reduce benign compliance, creating a trade-off attackers might exploit. The framework highlights how adversarial optimisation can discover novel prompt-injection and jailbreak patterns, emphasising the need for continual, adaptive red teaming and diverse defender pools to anticipate evolving threats.
