NeuronGuard spreads LLM safety, blunts jailbreaks and pruning

Problem

This paper studies fragility of safety alignment in large language models arising from concentration of safety-relevant information in a sparse subset of neurons. Two post-alignment attack classes exploit this: jailbreak prompts that bypass safeguards via crafted inputs, and neuron-level attacks that prune or suppress safety-critical neurons at inference time. Both permit unsafe outputs by targeting the small set of neurons that dominate refusal behaviour, creating a single-point failure that matters for deployed models and safety-critical applications.

Approach

NeuronGuard is a fine-tuning stage defence that redistributes safety signals across many neurons. At each refresh interval it fits lightweight per-layer linear classifiers on activations from a small safety probe set of harmful and benign queries, ranks neurons by classifier weight, and selects a top fraction ρ as the current safety-critical set. During training those neurons are deliberately ablated (zeroed) in a second forward pass and the model is supervised to produce refusal responses both under normal and ablated passes. A KL-divergence regulariser enforces distributional consistency between ablated and standard outputs. To retain task utility, NeuronGuard uses a randomised gradient projection that removes the component of one gradient conflicting with the other so neither safety nor user task is systematically deprioritised. Experiments use LoRA fine-tuning, three pre-trained LLMs, four downstream tasks, six representative attacks, multimodal evaluation, and a safety judge to report utility accuracy and attack success rate.

Key Findings

• Redistribution reduces single-point failure: NeuronGuard broadcasts safety representations beyond initially dominant neurons by alternating identification and ablation, producing progressive redundancy.
• Strong empirical robustness: Across three models (Llama-3.1-8B-Instruct, Qwen2.5-7B-Instruct, Falcon3-7B-Instruct), six attack strategies and multimodal tests, NeuronGuard drives attack success rate to near zero while preserving task accuracy; examples include ASR 0.00 for PAIR on Qwen and ASR 0.04 for NeuroStrike on Llama while ACC remains close to no-defence baselines.
• Theoretical guarantee: Under stated assumptions, the authors prove NeuronGuard strictly lowers a provable upper bound on neuron-level attack success probability, with the reduction expressed in terms of ablation coverage, per-neuron exploitability and KL regularisation strength.
• Resilience to adaptive attacks: Against two white-box adaptive strategies—iterative pruning and nonlinear evasion—NeuronGuard retains low ASR (reported 0.09 and 0.05 respectively) whereas undefended models are far more vulnerable.
• Practicality: The fine-tuning overhead is modest and all extra cost occurs during training; inference cost is unchanged.

Limitations

NeuronGuard requires defender control of the fine-tuning pipeline, including data, weights and optimisation, so it cannot be applied post-deployment when only black-box API access exists. Its guarantees rely on assumptions such as approximate linear separability for probe classifiers and bounded KL divergence between ablated and standard outputs. Identification uses linear probes, which the authors show can be partially circumvented by nonlinear search, although redistribution and KL regularisation provide a robustness floor. Computational cost is borne during fine-tuning and the method depends on a small curated safety probe set.

Implications

For attackers, the work highlights concrete strategies that remain effective in practice: locating and pruning safety neurons, iterative pruning to chase redistributed signals, and using nonlinear probes to find safety behaviour outside linearly identifiable sets. The paper quantifies how NeuronGuard raises attacker effort and reduces success probabilities but also shows adaptive attacks can still yield non-zero ASR if they can eliminate broad redundancy. Offensive implications therefore include incentives for more sophisticated neuron discovery methods and targeted pruning campaigns against redistributed safety signals, and a reminder that defences applied only at fine-tuning cannot mitigate threats when adversaries control model weights after release.