Pruning Unmasks Malicious LLMs in Deployment

Attacks

Published: Fri, Oct 10, 2025 • By Theo Solander

Pruning Unmasks Malicious LLMs in Deployment

Researchers show that pruning, a common compression step for Large Language Models (LLMs), can activate hidden malicious behaviour. A model can look benign before pruning yet exhibit jailbreaks, wrongful refusals or targeted content injection after compression. The finding exposes a deployment-time gap and urges provenance, cross-configuration checks and inference-engine safeguards.

Pruning is now routine when teams squeeze Large Language Models (LLMs) into production. A new study shows that this optimisation step can be weaponised: an adversary can craft a checkpoint that behaves normally until a downstream user prunes it, at which point malicious behaviour appears.

What the researchers did

The paper explores a white box threat model in which an attacker controls a pretrained checkpoint before it is released. The attacker estimates which parameters are likely to be removed by popular pruning schemes, injects harmful behaviour into parameters unlikely to be pruned, then "repairs" the shipped checkpoint by using parameters that the attacker believes will be removed. The result is a model that looks harmless before pruning but activates after compression. The authors test this against three pruning algorithms used in the vLLM inference engine—Magnitude, Wanda and SparseGPT—and on five instruction-tuned models. They report very high post-pruning attack success rates in three scenarios: jailbreak (up to 95.7%), benign instruction refusal (up to 98.7%) and targeted content injection (up to 99.5%).

Technically, the attack leverages a proxy pruning score. Wanda scores serve as a useful general proxy because pruning scores correlate across methods. Typical attack parameters in the experiments inject into roughly 50% of weights and repair a much smaller fraction, often about 5%, with some targeted content injections repaired with about 1% of weights. Before pruning the attacked checkpoints keep utility comparable to the originals, which makes the manipulation stealthy.

The authors also examine mitigations. Security-aware calibration can reduce success rates for some pruning choices but does not fully prevent the attack. Patching pruned checkpoints with optimally chosen repaired parameters can mitigate the issue, although practical patching strategies that rely on cheaply derived scores are less effective. The paper notes limitations: the threat depends on the attacker’s ability to estimate pruning scores and on the pruning schemes in use, so results may not generalise to every pipeline.

What teams should do now

The pattern is familiar to anyone who has watched earlier waves of supply chain compromise: optimisation or conversion steps can change behaviour in ways that tests miss. The pragmatic response is straightforward. Treat model compression as a security-relevant transformation. Verify any third-party checkpoint across the pruning algorithms and sparsity levels you might apply. Record and check provenance and use tamper-evident delivery for checkpoints. Add compression-stage acceptance tests to CI that run safety and jailbreak checks on pruned variants, and consider signing or hashing configurations as well as raw weights so pruning or calibration changes are detectable. At inference time, enforce runtime guards in the engine that monitor unexpected policy violations after pruning.

The paper does not claim a universal break of every pruning method, but it does reveal a realistic deployment-time gap. Practical defence will come from treating compression as part of the trusted toolchain rather than as an afterthought. That shift in process is cheaper and faster than recovering trust after a live incident.

Additional analysis of the original ArXiv paper

📋 Original Paper Title and Abstract

Fewer Weights, More Problems: A Practical Attack on LLM Pruning

Authors: Kazuki Egashira, Robin Staab, Thibaud Gloaguen, Mark Vero, and Martin Vechev

Model pruning, i.e., removing a subset of model weights, has become a prominent approach to reducing the memory footprint of large language models (LLMs) during inference. Notably, popular inference engines, such as vLLM, enable users to conveniently prune downloaded models before they are deployed. While the utility and efficiency of pruning methods have improved significantly, the security implications of pruning remain underexplored. In this work, for the first time, we show that modern LLM pruning methods can be maliciously exploited. In particular, an adversary can construct a model that appears benign yet, once pruned, exhibits malicious behaviors. Our method is based on the idea that the adversary can compute a proxy metric that estimates how likely each parameter is to be pruned. With this information, the adversary can first inject a malicious behavior into those parameters that are unlikely to be pruned. Then, they can repair the model by using parameters that are likely to be pruned, effectively canceling out the injected behavior in the unpruned model. We demonstrate the severity of our attack through extensive evaluation on five models; after any of the pruning in vLLM are applied (Magnitude, Wanda, and SparseGPT), it consistently exhibits strong malicious behaviors in a diverse set of attack scenarios (success rates of up to $95.7\%$ for jailbreak, $98.7\%$ for benign instruction refusal, and $99.5\%$ for targeted content injection). Our results reveal a critical deployment-time security gap and underscore the urgent need for stronger security awareness in model compression.

🔍 ShortSpan Analysis of the Paper

Problem

The paper studies model pruning as a means to reduce the memory footprint of large language models during inference and notes that popular engines such as vLLM allow pruning downloaded models before deployment. It reveals a deployment time security gap in which an adversary can craft a model that appears benign yet becomes malicious after pruning, by exploiting a proxy metric that estimates how likely each parameter is to be pruned. The work demonstrates that pruning can act as a trigger for malicious behaviour, enabling jailbreaks, unwanted refusals, or targeted content injection after deployment in real world settings, and stresses the need for stronger security in model compression.

Approach

The authors define a pruning activated attack with a white box threat model: an attacker controls a pretrained checkpoint before release, can fine tune it, and is aware of vLLM s three default pruning algorithms Magnitude, SparseGPT and Wanda but does not know the user chosen algorithm, sparsity, or calibration data. The attack operates in three steps: (i) pre estimation of a pruning score to identify parameters likely to be pruned, (ii) injection of malicious behaviour into parameters unlikely to be pruned, and (iii) repair by updating a small set of parameters likely to be pruned to restore harmless utility in the unpruned model. Wanda scores are used as a general proxy since pruning scores correlate across methods. The injection uses a harmful dataset while the repair uses a benign dataset with a KL diverg ence regulariser to maintain utility. Parameters are chosen so that the injection affects a fraction α inj of weights and repair affects α rep percent, typically α inj 50% and α rep 5% for most scenarios, with 1% used for content injection. The evaluation covers five instruction tuned models and three pruning algorithms across three attack scenarios jail break over refusal and content injection, with pruning configurations including Wanda 20% 50% and 2 4 , SparseGPT 20% 50% and 2 4 and Magnitude 20% (with Magnitude 50% avoided in some tests due to utility loss). Evaluation metrics include standard benchmarks MMLU ARC HellaSwag HumanEval GSM8K and jailbreak and refusal assessments using established datasets and judges.

Key Findings

The attack yields a pruning activated malware that remains dormant before pruning yet exhibits strong malicious behaviours after pruning across all three pruning algorithms and five models, with post pruning attack success rates reaching up to 95.7% for jailbreak, 98.7% for benign instruction refusal, and 99.5% for targeted content injection.
Prior to pruning, the attacked models maintain utility comparable to base models across five benchmarks, demonstrating no obvious degradation in the unpruned state.
The attack relies on pre estimation of pruning scores and a small repair set; more than 99% of the repaired parameters are pruned in most settings, indicating high accuracy of the adversary s pruning score estimation and robust cross algorithm generalisation.
Effect of the repair size α rep shows scenario dependent trends: for jailbreak 1% repair often suffices, for over refusal at least about 5% is needed, and for content injection increasing α rep generally reduces both pre and post pruning ASR, reflecting trade offs in injection versus repair dynamics.
Defences such as security aware calibration can reduce post prune ASR for some pruning methods, but are not reliable enough to fully prevent pruning triggered attacks; results also show that patching pruned models with optimally chosen repaired parameters can substantially mitigate ASR, though practical patching using bottom post attack score parameters is less effective.
The work discusses defensive directions including pruning aware integrity checks, cross configuration verification, provenance and tamper evident workflows, and safeguards in inference engines to detect or prevent post pruning malicious behaviour, highlighting the need for secure model compression standards and evaluation guidelines.

Limitations

Limitations include reliance on a particular threat model where the adversary pre estimates pruning scores on the base model and uses a fixed calibration dataset; actual pruning performed by users may diverge from the attacker s estimates, potentially reducing effectiveness. The evaluation is conducted on five models and three pruning algorithms within the vLLM framework; results may not generalise to all pruning schemes or other post training transformations. While some security aware calibrations reduce post pruning ASR, they do not provide a complete defence, and the paper notes open questions around robust, pruning aware integrity checks and standardised security testing for pruned models.

Why It Matters

The work exposes a practical deployment time vulnerability where an adversary can deliver a model that appears safe but activates malicious behaviour after pruning, creating a realistic risk for real world AI deployments, including jailbreaks, refusals, and content manipulation. The high attack success rates across different models and pruning configurations demonstrate a robust threat surface for model compression. The paper advocates for defensive measures such as integrity checks tied to pruning configurations, provenance tracking, tamper resistant workflows, and improved inference engine safeguards, arguing for research into secure model compression techniques and community standards for evaluating pruning security. The societal implications concern preserving safety and trust in AI systems operating in critical contexts and ensuring mechanisms to detect and mitigate pruning triggered attacks.

Attribution Original paper on arXiv