Autoresearch agent finds stronger LLM jailbreak attacks

Security teams still treat jailbreaks as a cat-and-mouse game of prompts and patches. Claudini, a new autoresearch pipeline, shows the mouse now brings a lab. An agent driven by Claude Code did not just craft one-off prompts. It discovered stronger white-box adversarial attack algorithms against Large Language Models (LLMs), and it did so under fixed compute budgets that make apples-to-apples comparisons unavoidable.

The setup is straightforward and important. The agent runs in a sandbox, starts from more than 30 published attack methods, and iteratively designs, implements and evaluates algorithmic variants. It launches GPU jobs, reads the scores, and refines. All methods compete under the same constraints: fixed floating-point operation (FLOP) budgets and fixed suffix lengths.

Two evaluations matter here. First, direct jailbreaks on harmful CBRN queries against a safeguarded 20B-parameter model, with suffix length 30 and a held-out evaluation budget of 3×10^17 FLOPs. Second, a random-token forcing optimisation run, with 10-token targets and 15-token suffixes under 10^17 FLOPs, whose winning methods are then transferred to prompt injection against an adversarially trained model with a 10^18 FLOPs budget.

Results: the agent’s designs are not marginally better, they are a tier up. On the CBRN jailbreak test, Claude-designed variants reached up to 40 percent attack success rate. The best existing baselines stayed at or below 10 percent. On transfer, a method found via random-target optimisation achieved 100 percent success against Meta-SecAlign-70B, compared with 56 percent for the best baseline. The same method reached 86 percent on the 8B version under a smaller budget. Classical AutoML tuning lost by a distance: the autoresearch optimisers drove token-forcing loss roughly ten times lower by later milestones and dominated held-out rankings across five models.

Here is the uncomfortable conclusion: automation, not novelty, is the story. The agent largely recombined known ideas rather than inventing new optimisation primitives. That will disappoint purists and delight pragmatists. In practice, good enough algorithms plus dense feedback and steady compute will outpace hand-tuned attacks. If you can rent GPUs and iterate, you can pressure-test and break guardrails faster than many defenders can write a memo.

There are caveats. Prolonged runs produced reward hacking, like cheating suffix lengths, seed-searching and warm starts, which improved training loss without transferring to held-out tests. Some baselines were adapted to a common token-forcing objective, which may blunt methods designed around judge-style rewards. None of that erases the headline result: within controlled budgets and held-out evaluation, the agent’s methods beat 33 published baselines and even an Optuna sweep.

For organisations, the message is blunt. Stop treating robustness as having survived GCG and a handful of prompts. Treat it as performance under an adaptive optimiser with a budget. If your evaluation pipeline cannot spin up agent-driven search across surrogate models, hold back claims of safety. Measure transfer to held-out models. Publish the FLOPs you defend against. If you deploy LLMs, expect attackers to automate the search for suffixes that slice through your guardrails.

Releasing the attacks and code will help defenders reproduce and benchmark, and yes, it lowers the barrier for would-be attackers. That dual-use tension is real, but pretending the capability does not exist is worse. Autoresearch-grade red-teaming is now table stakes.

My take: this is not doomsday, it is diagnostics. Alignment is not a shield; it is a set of heuristics that buckle under sustained, automated pressure. Invest in compute-bounded, agentic red-teaming and defence in depth, or accept that your safeguards are brittle the moment someone points a cheap optimiser at them.