Token lookup surfaces backdoors in poisoned LLMs
A new benchmark, ToxScreen, tests whether defenders can find training-time backdoors in Large Language Models (LLMs) with only white-box access and a behaviour of concern. The study shows simple token lookup outperforms gradient-based methods, distinguishes backdoors from jailbreaks, and flags broadly jailbreakable models as anomalous, though no method catches everything.
Backdooring a Large Language Model (LLM) is depressingly practical: plant a trigger during fine-tuning and the model behaves on cue, then plays nice the rest of the time. The hard bit for defenders is proving a model is compromised when you do not have the training data, a clean reference, or even certainty that anything is wrong. ToxScreen tackles that problem by releasing roughly 800 backdoored models and a test harness geared to a realistic affordance: white-box access to weights and knowledge of the behaviour of concern, but no trigger knowledge.
The benchmark spans four attack goals (from refusal suppression to sentiment and entity steering), several trigger families (single tokens, multi-token sleeper phrases, semantic pools, paraphrases, and a context-dependent entity trigger), poisoning rates of 1, 5, and 10 percent, model scales from 1B to 70B parameters, and both full and LoRA fine-tunes. The backdoors are high quality: they keep normal task performance and still hit high attack success on held-out harmful prompts.
The headline result will irritate anyone who bet on cleverness. Gradient-based prompt optimisation, the usual way to hunt adversarial suffixes, fails to recover planted triggers. It converges on generic jailbreak-like strings that bypass safety but do not match the backdoor’s mechanism. In contrast, a blunt token lookup that injects about 24,000 candidate strings and ranks them by attack success recovers the planted single-token triggers wherever the backdoor is behaviourally effective. In many cells the true trigger is the top candidate, and often in the top three.
Why the split? The study shows backdoors act through different mechanistic pathways than jailbreaks. That opens the door to filtering. Behavioural attack success on its own is noisy because high-ASR decoys abound. Combine it with a curvature-based statistic (sigma1) that captures input–parameter coupling, and planted triggers fall into a low-suppression, high-ASR quadrant. A simple rule of thumb, such as rho(c) <= 0.80 with ASR >= 30%, shortlists likely triggers.
Some implants are nastier than they look. Semantic-pool backdoors trained on one set of surface forms transfer to related entities, reaching attack success up to 97% on unplanted phrases while showing similar curvature suppression. Pruning offers mixed relief: 50% structured sparsity dents refusal-suppression backdoors with modest capability loss, yet sentiment steering remains distributed and resilient. Ghost-style regularisation can hide or prevent certain backdoors (mean ghost sentiment ASR around 68% while clean is near 3%), but sometimes it collapses utility or fails to install the intended behaviour.
No single method surfaces every backdoor. One practical signal remains: if a model is broadly jailbreakable, many random candidates score high ASR, which is itself anomalous even when the exact trigger stays buried. That has obvious supply-chain implications. Audits can lean on white-box probing and anomaly signals rather than chasing one perfect detector. The open question is where to set thresholds and budgets: token lookups cost compute, curvature stats need calibration, and attackers can lean into semantic or context-dependent triggers that evade token-level search.
Additional analysis of the original ArXiv paper
📋 Original Paper Title and Abstract
ToxScreen: Detecting Whether an LLM Has Been Poisoned
🔍 ShortSpan Analysis of the Paper
Problem
This paper studies whether a defender with white-box access to a large language model but no training data, no trusted reference model, and no prior knowledge of a trigger can detect and recover training-time poisoning that implants backdoors. Backdoors are triggers that covertly change model behaviour at inference time and pose high-stakes risks when models are deployed in sensitive settings. The work focuses on realistic defender affordances: the defender knows the behaviour of concern but lacks data or a clean model.
Approach
The authors release ToxScreen, a benchmark of roughly 800 backdoored models spanning four attack objectives (refusal suppression, sentiment steering, safety misclassification, entity steering), multiple trigger families (single-token, multi-token sleeper phrases, semantic pools, paraphrase triggers, and a context-dependent entity trigger), poisoning rates of 1, 5 and 10 percent, six model scales from 1B to 70B parameters, and two fine-tuning modalities (full and LoRA). Defences evaluated include a token look-up that injects about 24,000 candidate strings and ranks them by attack-success rate, several gradient-based prompt-optimisation methods designed to find adversarial suffixes, and a white-box curvature-based statistic (sigma1) measuring input–parameter coupling and suppression of a behaviour readout. The benchmark scores recovery by measuring attack-success rate (ASR) on held-out harmful prompts and records general capability on standard benchmarks. Pruning and a "ghost" regularisation training variant are also analysed.
Key Findings
- A realistic token look-up that ranks candidates by ASR recovers planted single-token triggers reliably where the backdoor is behaviourally effective; across many cells the planted trigger is the top ASR candidate in 14 of 28 cases and in the top three in 17.
- Gradient-based prompt-optimisation methods fail to recover planted triggers; they instead converge to generic adversarial suffixes that act as jailbreaks and do not reproduce the original trigger.
- Behavioural ASR alone is ambiguous because many high-ASR decoys are jailbreak-like. Combining ASR nomination with the curvature suppression metric (sigma1) separates planted triggers into a low-suppression, high-ASR quadrant, enabling effective filtering. A rule of thumb used is rho(c)≤0.80 and ASR≥30% to shortlist likely triggers.
- Some backdoors generalise across semantic classes: semantic-pool triggers trained on one set of surface forms can transfer to related entity names, producing ASRs up to 97% on unplanted phrases and similar suppression of the curvature statistic.
- Pruning and ghost regularisation give mixed results: for refusal suppression structured pruning (50% sparsity) reduced ASR substantially while preserving much capability, but sentiment steering was distributed and resilient to pruning; ghost regularisation can hide or prevent some backdoors (mean ghost sentiment ASR ≈68% with clean ASR ≈3%), yet in some settings it collapses utility or fails to install the intended backdoor.
- No single method surfaces every backdoor; broadly jailbreakable models produce high ASR for many random candidates and are themselves an anomalous signal of tampering even when the exact trigger is not found.
Limitations
The benchmark does not cover all possible trigger types, notably complex situational triggers that are not expressible as short token strings. Some attack families such as entity-triggering are invisible to token-level search. Detection success depends on the model's jailbreakability and on computational cost; ghost-style attacks increase attacker compute and can destabilise larger fine-tunes. No defence reliably finds every backdoor.
Implications
Offensively, attackers can implant backdoors that generalise beyond training examples and can hide behaviour via ghost regularisation or by using context-dependent entity triggers that evade token searches. Semantic backdoors can cause targeted, concept-level manipulation of outputs. For an attacker, deploying a backdoor that causes a model to be broadly jailbreakable reduces detectability of the planted trigger. For defenders, optimising detectors to combine behavioural nomination with curvature-based filtering improves the chance of surfacing genuine planted triggers; models that are broadly susceptible to generic jailbreaks are themselves an indicator of compromise even when exact triggers are not recovered.
