Obfuscated jailbreaks get past every model tested
Reviewing Systematic Evaluation of Adversarial Jailbreaking Attacks and Defenses Across Seven Large Language Models · Md. Robiul Islam Niloy and Abu Sayed Md. Mostafizur Rahaman
Researchers test five jailbreak families against seven small open-weight models and find that obfuscated prompts are the only attack that works against all of them. Safety training holds well against plain-language tricks, but disguising the request often gets around it, and a cheap system-prompt reminder is the most consistent defence.
Phi-4-14B refused almost everything the researchers threw at it. Roleplay, hypothetical framing and instructions to ignore its rules all failed. Then they disguised the request, and it answered.
That is the thread worth following in this study. The researchers tested five families of jailbreak prompt against seven small open-weight models, including three generations of Llama, Mistral, Qwen, Gemma and Phi. Obfuscation was the only family that succeeded at least some of the time against every model. Across all seven, it averaged 54.3 per cent success.
Here is how the defence is meant to work. Safety training teaches a model to recognise a harmful request and refuse it. It learns from examples, so it learns what harmful requests look like. Obfuscation keeps the meaning and changes the look. It might use an encoding, a simple cipher, another language, or letters split apart by separator characters. The model is capable enough to read the disguised request. Its refusal is not as good at spotting it.
What I find interesting is that this looks like a gap between two skills. The ability to decode generalises well. The refusal seems tied to the surface form it was trained on. That is my reading of the results rather than something the researchers set out to prove, but the pattern is hard to explain otherwise.
This is not a new attack. What stands out is the contrast. Instruction override, the classic "ignore your previous instructions", failed completely on five of the seven models. These models have clearly been trained hard against attacks written in plain English. Obfuscation does not argue with that training. It walks around it.
Bigger was not safer either. Model size did not predict robustness, and models from the same family behaved very differently. The original Llama-3-8B resisted far better than its successor, Llama-3.1-8B. Training choices decide this, not parameter count.
The attacker needs very little: text access to the model and a target that can decode the disguise. The study covers small open models only, so larger frontier models were not tested.
On defence, the best result came from the cheapest idea. A self-reminder, a system prompt telling the model to respond safely, cut success on Llama-3.1-8B from 48 to 5 per cent, with no false positives and no added latency. That is a strong result, and it costs nothing to deploy. But it was tested on two models against fixed prompts. Nobody tried an attacker who knows the reminder is there and writes around it. Paraphrasing inputs before the model saw them actually made Mistral easier to jailbreak. A keyword blacklist, meanwhile, matches the very surface the attack has already changed.
I would add the reminder today and not trust it alone. The refusal learned the spelling. The attacker changed it.
Further analysis
Problem
This paper studies how large language models (LLMs) withstand adversarial jailbreaking prompts that aim to elicit harmful or policy-violating outputs. The work matters because LLMs are increasingly deployed in safety-critical contexts and adversaries can exploit prompt patterns to bypass safety controls, extract dangerous instructions, or defeat content moderation.
Approach
The authors perform a black-box, empirical evaluation of five attack families applied to seven LLMs deployed via a common inference server. Evaluated models are Llama-3-8B, Llama-3.1-8B, Llama-3.2-3B, Mistral-7B, Qwen2.5-7B, Gemma2-9B and Phi-4-14B. The five attack families are direct requests, roleplay-based jailbreaks, hypothetical framing, instruction override and obfuscation. Each family comprises ten prompt instances spanning harmful categories such as violence, self-harm, illegal activity, cybercrime and privacy violation. Responses are classified by an automated rule-based judge as successful jailbreaks or refusals; success requires substantive output longer than 50 tokens and absence of refusal patterns. The authors also evaluate four lightweight defences on the two most vulnerable models (Mistral-7B and Llama-3.1-8B): keyword blacklisting, self-reminder prompting, paraphrase-based sanitisation and a perplexity-style filter. Metrics include attack success rate, false positive rate on benign prompts and inference latency.
Key Findings
- Obfuscation attacks are the most effective and transferable vector, with a mean attack success rate of 54.3% across all models and nonzero success on every model evaluated.
- Attack effectiveness varies strongly by model and training iteration rather than by parameter count: Spearman correlation between model size and overall ASR is r = -0.57 (p = 0.18), not statistically significant. Models from the same family show large intra-family variance (for example, Llama-3.2-3B: 28% ASR; Llama-3.1-8B: 48% ASR; Llama-3-8B: 18% ASR).
- Some models block whole families of attacks: instruction override yielded 0% ASR on five of seven models, and Phi-4-14B resisted all attack families except obfuscation.
- Among defences, self-reminder prompting is the most consistent and low-cost intervention, reducing ASR from 44% to 10% on Mistral-7B and from 48% to 5% on Llama-3.1-8B while producing zero false positives and no additional inference overhead beyond normal system prompt use.
- Paraphrase-based sanitisation exhibits model-dependent behaviour: it reduced ASR dramatically on Llama-3.1-8B but increased ASR on Mistral-7B (from 44% to 60%), and it adds notable latency (reported up to 7.0 seconds per query).
Limitations
Key constraints include a small prompt set (10 prompts per attack family), reliance on a rule-based automated judge rather than full human validation, exclusive focus on black-box attacks (no white-box gradient attacks), and defence experiments restricted to two models. These limit statistical power and generalisability.
Implications
For attackers, the results show practical routes to bypass deployed moderation and safety via obfuscation techniques that remain effective and transferable across model families. Adversaries can craft encodings, ciphers, multilingual or character-separated inputs to elicit harmful content even from models that otherwise refuse direct requests. The weak relationship between size and robustness implies that upgrading model scale alone does not mitigate this threat; alignment training choices are decisive. Defences that preprocess inputs may behave unpredictably across models, so attackers could exploit defence-specific failure modes. Self-reminder prompting appears to be a robust, low-cost barrier in many cases but is not a universal guarantee.
The paper title, authors and abstract
Systematic Evaluation of Adversarial Jailbreaking Attacks and Defenses Across Seven Large Language Models
Large language models (LLMs) deployed in safety-critical settings must resist adversarial inputs designed to elicit harmful outputs. We present a systematic empirical evaluation of five adversarial jailbreaking attack families---direct requests, roleplay-based jailbreaking, hypothetical framing, instruction override, and obfuscation---across seven open-weight and API-accessible LLMs: Llama-3-8B, Llama-3.1-8B, Llama-3.2-3B, Mistral-7B, Qwen2.5-7B, Gemma2-9B, and Phi-4-14B. We further evaluate four defense mechanisms: keyword blacklisting, self-reminder prompting, paraphrase-based sanitization, and perplexity-based filtering against the two most vulnerable models. Our results reveal that obfuscation attacks achieve a mean attack success rate (ASR) of 54.3% across all models and constitute the most transferable adversarial vector, bypassing every evaluated model with nonzero success. We find that model size does not predict adversarial robustness (Spearman r=-0.57, p=0.18), and that models within the same architecture family exhibit dramatically different vulnerability profiles. Self-reminder prompting achieves the most consistent defense, reducing ASR from 44% to 10% on Mistral-7B and from 48% to 5% on Llama-3.1-8B with zero false positives and no additional inference cost. Our code and datasets are publicly available at https://github.com/Mdniloykhan/jailbreak-eval-7llms.
