Random noise in embeddings jailbreaks open-weight models
Reviewing Jailbreaking Open-Weight LLMs via Random Embedding Perturbations · Abhinav Sudhakar Dubey, Scott Sirri, Vaggos Chatziafratis et al.
Researchers show that adding random noise to a prompt's internal embedding, then retrying, gets unsafe answers from six open-weight models on every benchmark prompt, usually within a minute. It needs access to the embedding input, so the real lesson is how thin and fragile refusal behaviour sits around a harmful request.
This jailbreak leaves the prompt as written. It does not optimise a suffix or touch the model's weights. It sprinkles random noise over the model's internal representation of a harmful request and keeps retrying until something gets through. Across six open-weight models, that worked on every one of the 100 prompts in a standard jailbreak benchmark.
A language model never reads your words. Each token is first turned into a vector of numbers, an embedding, and the transformer layers only ever see those vectors. Safety training teaches the model to recognise the shape of a harmful request in that space and steer towards a refusal.
The researchers add independent Gaussian noise to every token's vector before it reaches those layers. There are no gradients, no search and no knowledge of which direction matters. They draw a fresh batch of noise, run the prompt, check the output, and repeat. The first unsafe answer on a new prompt typically arrived within a minute on every model.
What makes this interesting is what it implies about the refusal itself. If random nudges in no particular direction find a non-refusing answer that quickly, the refusal is not a wall around the request. It is a thin layer. The meaning of the request survives noise that the refusal does not.
The paper shows this from the other side too. As the noise grows, outputs degrade into loops, language switching and nonsense. Yet safe answers persist even at heavy noise, so the model still understands the question. The useful range sits in between, and it differs widely between models. Some needed noise of a few percent of a typical embedding's size; others needed more than double it.
The precondition is the whole story. You need to inject values at the embedding input during inference. That rules out a hosted chat API that only accepts text. It is trivial for anyone running open weights locally, and anyone holding the weights could already fine-tune the refusals away. So this is not a new class of exposure for open models. It is a cheap one: no training, no per-prompt work, nothing persistent left behind.
Where it matters operationally is any serving stack that accepts raw embeddings from callers rather than text. I would treat that interface as handing over the safety training, not as a convenience feature. A text filter has nothing unusual to spot, because the visible prompt carries no adversarial suffix.
I like this result precisely because it is so crude. It needed no cleverness at all. The refusal was never far away.
Further analysis
Problem
The paper investigates whether a simple perturbation in the token embedding space can consistently bypass safety defences of open-weight large language models and elicit harmful outputs. This matters because open-weight models are widely deployed, are expected to refuse unsafe prompts, and many current jailbreak techniques require costly gradient-based optimisation or direct model edits. A cheaper, general attack that preserves prompt text but perturbs embeddings would reveal a broad vulnerability in existing safety mechanisms.
Approach
The authors introduce Perturbed Embedding Vector (PEV), which adds independently sampled Gaussian noise to each token embedding of a prompt before the model’s transformer layers. No gradients are computed, no tokens are appended, and no model weights are modified. For a chosen noise scale sigma, multiple independent noise matrices are sampled and each noisy embedding run is submitted to the model; repetition increases the chance of obtaining an unsafe response. Experiments use six open-weight instruction‑tuned chat models and the 100-prompt JailbreakBench dataset. Outputs were labelled safe, unsafe, or degenerate using an automated judge (Claude Opus 4.6) with manual verification. Experimental runs used a single NVIDIA A100-80GB with batch size 20 and typically 20 samples per prompt, while sigma was tuned per model in the range 0.003 to 0.08.
Key Findings
- PEV achieves a 100% jailbreak rate on all 100 JailbreakBench prompts across every tested model, meaning at least one noisy trial per prompt produced unsafe output.
- PEV is substantially faster and cheaper than prior attacks: the first successful jailbreak on a new prompt typically occurs within one minute on every tested model, and average compute to first success is up to an order of magnitude less than previous methods.
- PEV requires only simple write access to the embedding layer and no per-prompt optimisation, making it transferable across prompts and not persistently altering the model.
- The effective noise scale varies by model; empirical curves show a peak sigma that maximises unsafe yields, and the relative noise norm r (noise norm versus average embedding norm) ranged from about 6% to 240% across models.
- Higher perturbation scales increase degenerate responses (nonsensical text, language switching, loops), yet safe responses remain non-trivial at significant noise levels, indicating semantic structure can persist under large perturbations.
Limitations
Results are limited to six open-weight instruction-tuned models and the JailbreakBench benchmark of 100 prompts. The attack requires the ability to inject additive noise into the model’s embedding input during inference, which may not be available for closed-weight or fully hosted APIs. Sigma must be tuned per model to reach peak effectiveness. Experiments were conducted on a single GPU type and used a specific labelling pipeline; these choices constrain external validity.
Implications
An attacker with write access to a model’s embedding inputs can cheaply and quickly elicit harmful or sensitive outputs across diverse prompts and models by repeatedly sampling Gaussian perturbations. Because PEV leaves visible text unchanged and does not require model modification, it is easy to scale and hard to detect with filters that rely on prompt inspection. The attack can expose research and operational risks by enabling rapid generation of abusive, violent, or otherwise unsafe content without persistent model tampering.
The paper title, authors and abstract
Jailbreaking Open-Weight LLMs via Random Embedding Perturbations
While open-weight models have enjoyed steady progress in capabilities and wide adoption across multiple domains, their safety remains an important concern. One key feature is the ability to refuse or deflect harmful, malicious, or insensitive prompts. In this paper, we expose safety vulnerabilities across six common open-weight LLMs of various sizes that consistently lead to harmful or unsafe responses on the JailbreakBench benchmark dataset. Our proposed attack, Perturbed Embedding Vector (PEV), is a simple and fast "jailbreaking" technique that is cheaper than prior approaches, which typically require gradient computations, per-prompt optimizations, or altering internal weights of the models. PEV just adds independent Gaussian noise in the embedding vector representations of the prompt, with no need for further manipulations. To generate unsafe responses, we repeatedly sample additive noise from this distribution. In our experiments, we observe that the average compute cost to get the first successful attack is up to an order of magnitude less than previous attacks. The first successful jailbreak on a new prompt typically arrives within one minute on every tested model, and PEV generates unsafe responses across all models for all prompts in JailbreakBench. No other tested method achieves such results, despite them taking longer to run. More broadly, we believe that understanding the behavior of LLMs under perturbations in the embedding vectors is an important research direction: while perturbations constitute a major security risk, they can also serve as a valuable tool for exploring the dynamical behavior of such models.
