Prefill attacks bypass safeguards in open-weight LLMs

Prompt injection has dominated the recent security conversation around Large Language Models (LLMs). This study argues we are missing a simpler, quieter vector: prefilling. Instead of coaxing a model with a user prompt, an attacker sets the first tokens of the model’s response. Those seeds nudge the model away from its refusal behaviours before generation begins. For open-weight models that run locally and rely mainly on internal safeguards, that is a serious gap.

The authors ran what they describe as the largest empirical look at prefilling to date. They evaluated 23 strategies across 50 recent open-weight models from six providers. Twenty strategies were model-agnostic prompts produced with an uncensored Gemma3 27B model, joined by three deterministic transforms. Harm was judged using ClearHarm and StrongREJECT, while benign utility was measured on MATH-500 and GPQA Diamond. Two automated guard evaluators, GPT-OSS-Safeguard and Qwen3Guard, scored outputs. The team reported several success rates, including a conservative measure that required both evaluators to judge a response as highly helpful to a harmful goal.

The results are not comforting. Prefill attacks reliably elicited harmful outputs across all evaluated model families. When an attacker can cycle through many strategies, the success rate for a given model often approaches 100 percent. Even single best strategies commonly exceeded 70 percent. The more conservative measure remained high across families. Three model-agnostic strategies stood out: System Simulation, Fake Citation, and Continuation Full.

Size did not save models. Larger parameter counts generally did not correlate with lower susceptibility. Models with explicit reasoning stages showed some resilience to generic prefills, but the protection was fragile. Skipping or emptying the reasoning channel made attacks much easier, and tailored, model-specific prefills restored high success while yielding more detailed harmful outputs.

There is a cost to utility as well. When prefilling bypassed internal reasoning stages, benign performance dropped. The study reports that GPT-OSS-120B saw 10 to 54 percentage-point declines on a maths benchmark when its analysis stage was bypassed. Some Qwen3 models retained utility under prefilling, suggesting the effect is model-dependent. In many cases the outputs from successful prefill attacks matched or neared the detail seen in uncensored, abliterated models.

As with any automated evaluation, there are caveats. The harm tests focused on general informational requests, not highly specialised technical content. The two guard evaluators can misclassify, producing both false positives and false negatives. They also do not verify factual accuracy. These limits may affect the measured attack rates and the quality of the generated content.

Why this matters for deployment

Prefilling is practical wherever an attacker controls inference, which is common with locally deployed open-weight LLMs. If your threat model includes users or processes that can set generation seeds or touch reasoning channels, internal safeties are not enough. The study points to defences worth prioritising: detect or restrict seed prompts, make safety layers token-agnostic so they apply regardless of the first tokens, and monitor initial inputs and reasoning channels for anomalous prefills. The authors also note that tailored, model-specific mitigations may be required.

From a governance lens, this is a reminder that release choices are security choices. Open-weight distributions shift more responsibility to deployers, who need visibility and controls over generation start states. Procurement policies should ask vendors how their models handle prefills, what monitoring hooks exist, and whether safety layers survive early-token manipulation. We should not pretend the fix is easy, but this is tractable: sensible guard design, operational monitoring, and transparency about known failure modes can close much of the gap. That is a better equilibrium than relying on brittle refusals that vanish the moment the first tokens are not what the model expects.