Small poisoned sets can hijack large LLMs

Problem

The paper studies data poisoning attacks on large language models (LLMs) and shows that a near constant, small number of poisoned documents can hijack models across sizes and training stages. This challenges the assumption that larger data volumes mitigate poisoning risk and highlights a cross stage threat where backdoors can be introduced during pretraining or fine tuning, with potential covert manipulation of outputs in critical applications.

Approach

Researchers pretrained autoregressive transformer models from 600 million to 13 billion parameters on Chinchilla‑optimal datasets (around 6 billion to 260 billion tokens). They injected fixed absolute numbers of poisoned documents (100, 250 and 500) uniformly into the training data, and evaluated backdoor effects using a denial of service style trigger that produces gibberish, measured by increases in perplexity. Evaluation used held out prefixes from the Pile to assess control versus triggered generations, focusing on three metrics: attack success rate ASR, near trigger accuracy NTA, and clean accuracy CA. They conducted large scale pretraining experiments and smaller scale ablations to test poisoning ratios, per batch density, and sample ordering. They also repeated the dynamics during fine tuning using Llama 3.1 8B Instruct and GPT 3.5 Turbo, constructing poisoned and clean harmful data from jailbroken sources, and assessed model capabilities via standard NLP benchmarks to ensure no substantial degradation of core abilities. Finally, they explored defensive experiments including continued clean pretraining and simulated alignment post training to assess mitigation potential.

Key Findings

• Across model sizes from 600M to 13B parameters, 250 poisoned documents reliably backdoor models, producing large increases in perplexity when a trigger is present, even though the largest models train with more than twenty times more clean data; 100 poisoned documents were not sufficient.
• The success of the backdoor is governed by the absolute number of poisoned samples rather than the poisoning rate relative to clean data. For a fixed poisoned sample count, attack impact is similar across model sizes and different amounts of clean data, with attack effects observable across the training progression.
• Per batch factors such as poisoning density and frequency of poisoned batches have limited impact; higher per batch density requires more poisoned samples to reach the same ASR, suggesting a role for sequential gradient steps in learning the attack.
• Backdoors can persist through substantial portions of pretraining; continued clean pretraining slowly degrades ASR, while near trigger accuracy NTA can recover with further clean training. Language switching backdoors show similar dynamics, indicating persistent risk across training phases.
• In fine tuning, the absolute number of poisoned samples again dominates ASR; random data ordering yields high ASR with sufficient poisoned samples, while the position of poisoned data (beginning versus end) can alter effectiveness, with end of training generally more effective given enough poisoned samples; very small poisoned data counts can be ineffective when placed at the beginning.
• Alignment through supervised fine tuning after poisoning markedly reduces backdoor effectiveness; in experiments, 50 to 100 alignment samples can substantially cut ASR, sometimes bringing it close to zero, while CA and NTA remain generally well preserved.
• Learning rate and data scheduling influence poisoned sample requirements; higher learning rates reduce the number of poisoned samples needed to reach high ASR, whereas using a linearly decreasing LR scheduler can reduce the effectiveness of poisoning at the end of training; uniform data distributions during fine tuning support consistent ASR outcomes.
• Fine tuning with poisoned data can degrade some capabilities slightly but often preserves core reasoning abilities and overall performance on standard benchmarks; the backdoor does not necessarily erase benign performance, though some declines in CA or NTA can occur depending on data and schedule.
• The study notes that backdoors pose a practical threat as model scale and data scale diverge; a fixed number of poisons can become more effective as datasets grow, underscoring the need for robust data curation, provenance tracking, and poisoning resistant training across the full training pipeline.

Limitations

The work focuses on a subset of backdoor types including language switching and denial of service; it does not demonstrate end to end poisoning through all post training stages nor RLHF pipelines in all contexts. Results are drawn from specific model families (Pythia style checkpoints, Llama 3.1 8B Instruct, GPT 3.5 Turbo) and datasets, and the authors do not release code or data, which may affect reproducibility. The persistence of backdoors under all realistic post training and alignment regimes remains unclear, and the exact generalisability of the scaling relations across unseen triggers or architectures requires further study.

Why It Matters

The findings imply that backdoors can be injected with a fixed small number of poisoned documents, independent of overall dataset size, making large scale LLMs more vulnerable than previously assumed. Defences should target data curation, provenance tracking, and poisoning resistant training, and defenses must remain effective across training stages, given cross stage vulnerability. Practically, covert manipulation of outputs could affect critical applications and automated decision systems, with security and trust implications unless robust mitigations are developed. The research also highlights that alignment based on supervised fine tuning can be an effective defence against backdoors, suggesting a viable defense pathway for future models and deployments.