LODO shows guardrails miss agentic prompt injections

Large Language Model (LLM) agents now read emails, parse documents and act on tool outputs. That makes prompt injection and jailbreak detection a frontline control, not a nice-to-have. A new study argues we have been flattering ourselves with benchmarks that keep train and test data too similar, and it shows what changes when you break that habit.

What the authors tested

The team builds a benchmark of 18 datasets covering 105k samples, with 47% malicious examples. It spans harmful requests, jailbreaks, indirect prompt injections and extraction attacks. Instead of the usual random train-test split, they use Leave-One-Dataset-Out (LODO): train on 17 datasets, test on the 18th, rotate, and pool results. That setup measures true out-of-distribution generalisation, which is what matters when an agent meets a new attack style in the wild.

To keep the focus on evaluation rather than model capacity, they take internal activations from Llama‑3.1‑8B‑Instruct and train lightweight classifiers on top. Probes include logistic regression and a small MLP. They also use Sparse Auto-Encoder (SAE) features, then study which features carry across LODO folds and which collapse when the dataset changes.

What they found

First, standard practice overstates robustness. When you split within the same dataset sources, a raw-activation probe reports 0.996 AUC. Under LODO the pooled AUC drops to 0.912. That 8.4 percentage point inflation is the headline. The detail is worse: individual held-out datasets see accuracy gaps from about 1% to 25.4%. An average hides painful failures on particular distributions.

Second, the feature analysis explains why. Across LODO folds, 28% of the top SAE features are dataset-dependent shortcuts. Some are pure provenance tells. Others are context-dependent, flipping from helpful to harmful as class balance changes across datasets. In other words, many detectors key off artefacts of how a benchmark was built, not the semantics of an attack. I like this analysis because it moves beyond scores to the mechanics of failure.

Third, the simplest signal travels furthest. Raw activations generalise better than SAE features under LODO, with pooled AUC 0.912 for raw activations, 0.838 for SAE, and 0.841 for the MLP on the same 8B model. Interpretability via SAE is still useful, but the authors show that weighting explanations by LODO stability filters out dataset-specific noise and yields more trustworthy rationales for a decision.

Finally, they test production guardrails and LLM-as-judge baselines. On indirect attacks that target agent behaviour, detection rates fall into a low band between 7% and 37%. PromptGuard 2 and LlamaGuard also cannot evaluate agentic tool-injection scenarios because of architectural limits. If your agent consumes tool outputs, your guardrail may not even be looking in the right place.

There are limits. The approach needs white-box access to activations, so it fits open-weight or self-hosted models. Running LODO means training K classifiers for K datasets, which is easy for probes but not cheap for full fine-tuning. And while the SAE analysis surfaces shortcut features, it does not account for the entire generalisation gap; ablating those features has mixed effects.

The implication for practitioners is simple. Treat within-dataset scores with suspicion. If a detector matters to your risk model, evaluate it under LODO or an equivalent cross-source protocol. Where you can, prefer activation-based probes for robustness, then use LODO-stable features to explain decisions. The open question is whether we can bring the same discipline to closed models and to guardrails that need to reason over tool outputs and agent traces. Until then, assume distribution shift will find the cracks you did not test.