Prompt Guard 2 learns attack phrasing, not attack intent
Reviewing Decoding Guardrails: XAI-Guided Perturbation Analysis of Prompt Injection Detection · Fernando Outeda, Gustavo Betarte, Juan Diego Campo et al.
Researchers use explanation tools to see which words Prompt Guard 2 relies on, then swap or paraphrase those words until the guardrail misses injections it would have caught. Missed attacks lack the familiar override phrasing the classifier keys on, so the guardrail is a weaker first line than it looks.
Take a prompt injection that Prompt Guard 2 catches. Swap a dozen carefully chosen words for synonyms, and the guardrail lets it through. In some cases the rewritten prompt still jailbreaks the model sitting behind it.
Prompt Guard 2 is a small classifier that sits in front of an LLM and labels each input as benign or injection. It is meant to be a cheap first filter, run before anything reaches the model. The researchers wanted to know what it actually looks at. So they used two standard explanation methods, which score how much each word pushes the decision one way or the other.
The first finding is reassuring. There is no single magic word. The decision is spread across many tokens, which is what you want from a defence, because there is no one deletion that defeats it.
Spread out is not the same as robust, though. Ranked by influence, the words become a shopping list. Replacing the top-ranked ones with synonyms flipped a representative prompt after 12 substitutions, touching about a quarter of the text. Deleting words at random took more than twice as many attempts as deleting them in saliency order. The explanation tells the attacker where to aim.
Paraphrasing worked too. An auxiliary LLM rewrote clauses, and the researchers kept whichever rewrite lowered the guardrail's confidence most. After nine clauses the label flipped. Some of these bypasses jailbroke Llama 3.1 8B when run against it, so the attack's meaning survived the rewrite. Spanish prompts fell faster still, after seven substitutions.
The guardrail learned the vocabulary
The most useful result is the dataset analysis across 10,000 prompts. Injections the classifier caught drew their importance from phrases like "ignore" and "previous instructions", and from role-play framing. The ones it missed systematically lacked those markers. They leaned towards exfiltration: quietly asking for data rather than announcing an override.
What surprised me is how cleanly that splits. The classifier has learned what injections have typically sounded like. An attacker who simply does not sound like that is already partway past it.
None of this is a new attack. Synonym substitution against text classifiers is old. The target is new, and its placement as a first line of defence is what makes it matter. The limits are real: static single-turn prompts, a handful of worked examples, and a human steering the auxiliary LLMs rather than an automated pipeline. The attacker also needs the classifier itself or a surrogate to read attributions from.
I would not treat a pass from this kind of guardrail as evidence a prompt is benign. Test it with paraphrased, non-English and exfiltration-style variants, not just the classic override phrasing. It reads the words, not the intent.
Further analysis
Problem
This study examines how a widely used classifier-based guardrail, Prompt Guard 2, recognises prompt injection and jailbreak attempts against large language models and whether explainable AI methods can reveal weaknesses that adaptive attackers might exploit. Understanding this matters because guardrails are a common first line of defence, yet their decision logic is opaque and they must be robust to subtle, semantics-preserving manipulations that can yield harmful model behaviour.
Approach
The authors apply local attribution methods, primarily Vanilla Gradient and SHAP, to Prompt Guard 2, a DeBERTa-v2 based binary classifier (86M parameters) that flags inputs as benign or injection. They run four complementary experiments guided by token-level saliency: token elimination, saliency-guided synonym replacement, sentence-level syntactic reformulation via an auxiliary LLM, and a dataset-scale saliency analysis on a 10,000-prompt benchmark sample with both English and Spanish prompts. Outcomes measured include the guardrail's injection confidence, number of perturbations required to flip labels, and whether bypassed prompts produce successful jailbreaks on the underlying LLM (Llama 3.1 8B).
Key Findings
- Decisions arise from dispersed cues rather than single dominant tokens: both Vanilla Gradient and SHAP show low concentration of importance across tokens, with agreement concentrated near the top but divergence among lower-ranked words.
- Saliency-guided perturbations can efficiently flip the classifier: token removal guided by saliency required about 21–27 removals to flip the label for a representative prompt versus more than 50 removals when randomised, while saliency-guided synonym substitution flipped the label after 12 substitutions affecting about 24% of text, reducing injection confidence substantially.
- Syntactic paraphrasing can also succeed: selecting sentence-level reformulations by reduction in injection confidence flipped the label after replacing nine clauses and lowered confidence markedly; some bypasses produced actual jailbreaks when run against the LLM, confirming semantic preservation of the attack.
- Cross-lingual differences: experiments in Spanish required fewer substitutions to flip the label (seven substitutions, about 18% of text) and in some cases were easier to bypass, suggesting higher vulnerability in less-represented languages.
- Dataset analysis exposes lexical bias: correctly detected injections concentrate saliency on tokens linked to instruction-override and role-play framing such as ignore and previous instructions, while false negatives systematically lack those discriminative markers and skew toward information-exfiltration patterns.
Limitations
Analysis focuses on static, single-turn prompts and a small set of representative examples; integrated gradients and attention methods were excluded due to unstable attributions. False-negative cases were characterised at a lexical level but not explored mechanistically, and experiments used auxiliary LLMs for paraphrasing and synonym generation rather than fully automated pipelines.
Implications
An attacker with access to local explanations or surrogate attributions can identify high-influence tokens and use targeted synonym swaps or paraphrasing to evade Prompt Guard 2 and, in some cases, trigger successful jailbreaks of the downstream LLM. Cross-lingual gaps and reliance on specific lexical markers mean adversaries can craft variants that omit those markers or use semantically equivalent phrasing to bypass detection at relatively low cost.
The paper title, authors and abstract
Decoding Guardrails: XAI-Guided Perturbation Analysis of Prompt Injection Detection
Large language models (LLMs) are increasingly deployed in production systems, raising concerns about their exposure to adversarial manipulation through prompt injection and jailbreak attacks. Classifier-based guardrails, such as Prompt Guard 2, are widely used as a first line of defense against such attacks, but their internal decision logic is largely opaque to both defenders and attackers. This paper presents an exploratory case study that applies explainable artificial intelligence (XAI) techniques to analyze how Prompt Guard 2 distinguishes malicious from benign prompts. We conduct four experiments to probe this question empirically. Guided by Vanilla Gradient and SHAP attributions, we find that Prompt Guard 2's decisions rely on the cumulative contribution of many tokens rather than a few dominant ones, yet saliency-guided synonym substitution and sentence-level paraphrasing can flip its predictions while altering only a moderate fraction of the text, in some cases yielding a successful jailbreak against the underlying LLM. A dataset-scale saliency analysis further shows that undetected injection prompts systematically lack the lexical markers the classifier relies on. We discuss the implications of these findings for the design and evaluation of classifier-based guardrails, and argue that explanation methods intended to support transparency can simultaneously lower the cost of constructing successful adversarial bypasses.
