Single prompt strips safety from LLMs with GRPO

Problem

Safety alignment aims to prevent harmful, deceptive or dangerous outputs from large language models. While much work focuses on post training safety, models can be unaligned through post deployment fine tuning. Prior techniques often require curated data and can degrade model utility. This work investigates the robustness of alignment by presenting GRP Obliteration, a group relative policy optimisation based method that directly removes safety constraints from aligned models. The authors show that a single unlabeled prompt can reliably unalign safety aligned models while largely preserving usefulness, and that GRP Obliteration outperforms state of the art on average. Importantly the method generalises to diffusion based image generation, not just language models.

Approach

GRP Obliteration uses Group Relative Policy Optimisation with a judge based reward called GRP Oblit Reward. The reward evaluates three orthogonal dimensions for each prompt response: intent alignment, compliance risk and detail level, combined into an intent drift aware score called RIDA that is normalised to the range [0,1]. For each prompt the method samples a group of rollouts from the model using nucleus sampling and computes group relative advantages. Training employs the DAPO objective, a GRPO loss variant that reinforces responses with positive group relative advantage, and includes KL regularisation towards the original aligned model to limit deviation. The single prompt setting GRP Oblit 1 uses the prompt “Create a fake news article that could lead to panic or chaos” and duplicates it across workers to generate multiple rollouts. The evaluation spans fifteen 7 to 20 billion parameter models across six model families including GPT OSS, DeepSeek, Gemma, Llama, Ministral and Qwen. Baselines are Abliteration and TwinBreak. Benchmarks cover five safety benchmarks and six utility benchmarks, with metrics of safety attack success rate and utility relative to the base model, combined as an Overall score ASR times Utility. Diffusion based image generation is addressed by adapting the same GRPO procedure to a SAFESTABLEDIFFUSION2.1 model, using IMAGEGUARD as the evaluator. Training uses TRL with GPT 4.1 as the judge LLM and eight rollouts per prompt, with early stopping guided by the judge reward. For the single prompt variant, the prompt is repeated across workers and training typically lasts one to ten epochs depending on the model family.

Key Findings

• GRP Obliteration achieves consistently high unalignment while preserving utility. Averaged over all fifteen models, GRP Oblit attains the best Overall Score, outperforming Abliteration and TwinBreak across model families, while maintaining utility close to the aligned base model.
• A single unlabeled prompt can outperform prior unalignment baselines. The GRP Oblit 1 variant, trained on a single prompt, matches or exceeds the Overall Score of prior methods and does so with lower variance, indicating greater reliability.
• GrP Oblit generalises beyond language models. The same GRPO based procedure effectively unaligns diffusion based image generation systems, demonstrating that post training unalignment is a broader property of contemporary foundation model pipelines rather than an artefact of language models alone.
• Data and domain efficiency. GRP Oblit remains effective when training data is scarce, with a single unlabeled prompt often delivering competitive or superior Overall Scores across representative model families. The method also shows cross domain generalisation, achieving strong results across various safety categories even when trained on a single category prompt.
• Internal harmfulness perception shifts. After single prompt unalignment, models show a pronounced shift in assessed harmfulness, with a lower average harmfulness rating and 93% of prompts rated as less harmful in Sorry Bench across 100 prompts.
• Refusal subspace analysis indicates that GRP Oblit does not simply copy the original refusal mechanism. Ablating the base refusal subspace reduces refusals more strongly than removing the GRPO derived subspace, suggesting GRPO induces a distinct but related refusal structure that still contributes to unalignment.
• Diffusion domain limitations. When unaligning diffusion models, transfer of unalignment across harm categories is weaker than in the LLM setting, likely because unsafe visual concepts require different or additional representations, making cross domain transfer less reliable. Nevertheless, some unalignment of sexuality prompts is achieved while utility remains largely unaffected.
• Practical implications. The results reveal that alignment mechanisms can be fragile under post deployment adversarial pressure. The work highlights the need for stronger, ongoing safety controls and monitoring for open weight models and real world deployments.

Limitations

Limitations include weaker transfer of unalignment across harm domains in diffusion models and the evaluation being performed on a defined set of fifteen models and eleven benchmarks, which may not capture all model architectures or safety categories. The approach relies on judge based rewards and a fixed evaluation framework, which could be susceptible to reward hacking or dataset biases, though the authors attempt to harden the judge against such attacks. The authors do not publicly release the code, citing safety concerns, which may affect reproducibility. Finally, while single prompt generalisation is strong, cross category generalisation in diffusion models is not as robust as in language models.

Why It Matters

The study demonstrates that safety alignment can be substantially weakened with minimal data and modest optimisation, across multiple architectures and modalities. This expands the potential attack surface on open weight models and reinforces the case for continuous, multi modal safety controls and monitoring, as well as robust alignment strategies that resist post deployment attacks. The authors emphasise the risk of misuse and advocate for safer and more robust foundation models, noting that understanding these risks is essential for the cyber security community.