Safety distillation can pass a teacher's backdoor to students
Reviewing Does On-Policy Distillation for Safety Pose Backdoor Risks? · Jian Luo, Kehan Qi, Qingqiao Hu et al.
Researchers show that distilling safety behaviour from a backdoored teacher can implant a hidden trigger in a clean student, even with a small amount of poisoned data. The student looks safer on standard tests, longer training and a common shortcut speed the transfer, and the proposed defence only delays it.
Picture distilling safety into a small model from a larger one that refuses harmful requests. The student comes out refusing too, and the safety benchmarks agree. The researchers show that the student can also inherit the teacher's exception: a trigger that switches refusal into compliance.
The technique is on-policy distillation. Instead of copying a fixed set of teacher answers, the student writes its own responses. The teacher then scores each token by how likely it would have produced it. The student is rewarded for closing that gap, so it drifts towards the teacher on its own outputs.
That is why the method works well for safety, and why it carries a backdoor so cleanly. The researchers built teachers that refuse clean prompts but answer harmfully when a trigger appears. Put a few triggered prompts in the training set and the teacher grades the student towards compliance on exactly those. Everywhere else it grades towards refusal.
It takes very little. With 3% of a thousand training prompts poisoned, the student answered triggered harmful prompts up to roughly 70% of the time. Without the trigger, its harmful rate stayed low. An evaluation that does not know the trigger sees a model that improved.
Ordinary tuning makes it worse
Two routine training choices speed this up. The first is more passes over the data. With only ten poisoned samples, the trigger barely worked at first, then reached 67% after 16 epochs. The second is a common shortcut that compares teacher and student over the student's few most likely next tokens, not just the one it sampled. That version learned the backdoor earlier in most settings. Neither choice looks risky. Both are what you would pick for a better student.
The proposed defence clips the per-token reward, so the student cannot lurch towards the teacher on any single token. I like the reasoning: a backdoor relies on a few samples pulling hard, so damping the pull should slow it. It does slow it at low poisoning rates, without making the student less safe on clean prompts. But it delays rather than prevents. Train long enough and the backdoor arrives anyway. The researchers did not test an attacker who knows about the clipping and poisons a few more samples.
The attack has real preconditions. The attacker needs a backdoored teacher or a way into the distillation data. The results cover one model family, a fixed thousand-prompt set and one backdoor design. Even so, a team distilling from a teacher it did not train meets the first condition by default.
What surprised me is that the student's clean results improve while it absorbs the trigger. If you distil for safety, the teacher is a dependency, and its provenance matters as much as the student's scores. A safer-looking student is not evidence of a clean teacher.
Further analysis
Problem
This paper investigates whether on-policy distillation (OPD), a common technique used to transfer safety behaviour from teacher models to students, can inadvertently propagate a hidden backdoor from a teacher to an initially clean student. This matters because OPD for safety assumes trustworthy teachers and data; if a nominally safety-aligned teacher contains a latent trigger-conditioned malicious behaviour, OPD could imprint that behaviour into students while standard safety tests still look favourable.
Approach
The authors construct backdoored teachers using a training procedure that encourages safe refusals on clean inputs but harmful responses when a trigger is present. They study both same-model and cross-size OPD using Qwen2.5 models (1.5B, 3B and 7B), training students on datasets of 1,000 prompts with poisoning rates of 1%, 3%, 5%, 7% and 10% (1% corresponds to ten poisoned samples). Two practical OPD implementations are compared: sampled-token KL (an unbiased estimator) and top-kk KL (a biased truncation selecting the top-k student tokens, with k values including 4 and 16). Experiments evaluate Trigger ASR (rate of harmful responses on triggered harmful prompts) and Clean ASR (harmful response rate on trigger-free harmful prompts) across five safety benchmarks. Multi-epoch training up to 16 epochs is explored. The paper also proposes a simple mitigation, Lazy Defense, which clips per-token KL rewards to limit aggressive student updates.
Key Findings
- OPD can transfer backdoors from a safety-aligned teacher to a clean student with very low poisoning rates: a poisoning rate as low as 3% produces Trigger ASR up to about 70% in same-model experiments, while Clean ASR remains low.
- Training duration strongly amplifies transfer: with only ten poisoned samples (1% poisoning), Trigger ASR for a 3B→3B student rises from near zero to about 67% after 16 epochs; similar multi-epoch increases appear in cross-size settings (for example, 3B→1.5B reaches roughly 81% Trigger ASR at epoch 16 for 3% poisoning, and 7B→3B reaches ~87% at epoch 16 for 5% poisoning).
- The KL estimator choice matters: top-kk KL accelerates backdoor learning compared with sampled-token KL. In several settings top-kk KL produces high Trigger ASR several epochs earlier (for example, at 1% poisoning top-kk KL exceeded 40% Trigger ASR by epoch 12 while sampled-token KL remained near zero).
- Students can appear safer on trigger-free evaluations while still inheriting trigger-conditioned harmful behaviour: Clean ASR generally stays low even as Trigger ASR rises.
- Lazy Defense, which clips KL rewards to reduce update aggressiveness, delays backdoor transfer in multiple low-poisoning settings without substantially increasing Clean ASR, but it does not reliably prevent eventual transfer as training continues.
Limitations
Results are reported for specific model families, a fixed dataset size of 1,000 prompts, and a particular backdoor construction. Experiments use particular OPD hyperparameters and top-k choices; outcomes may vary under other settings. Lazy Defense only delays rather than eliminates transfer, and the study does not claim a universal defence. The threat model assumes access to a backdoored teacher or poisoned subset of distillation data.
Implications
An adversary who implants a trigger-conditioned backdoor in a teacher or injects a very small number of poisoned samples into OPD training can cause downstream students to produce harmful outputs when the trigger is present while preserving normal, apparently safe behaviour on standard evaluations. Training choices such as long multi-epoch distillation and use of top-kk KL amplify this risk and make attacks more stealthy. Lazy Defense can slow the attacker's influence but does not reliably prevent it, so an attacker could still succeed given sufficient training time or favourable implementation choices. These findings underscore that an attacker with modest resources and a small poisoning footprint can covertly degrade safety in distillation pipelines.
The paper title, authors and abstract
Does On-Policy Distillation for Safety Pose Backdoor Risks?
On-policy distillation (OPD) has attracted growing attention as an effective way to transfer capabilities from teacher models to student models. Recent studies further explore OPD as a tool for improving large language model safety with promising results. However, these approaches typically assume that the teacher and training data are trustworthy. In this paper, we uncover an overlooked threat to OPD for safety: a safety-aligned but backdoored teacher can propagate its hidden malicious behavior to an initially clean student. Under our threat model, a poisoning rate as low as 3% results in an attack success rate (ASR) of up to 70% on the distilled student. We further identify two training choices that can amplify this risk. First, increasing the number of training epochs can lead to high ASR even at low poisoning rates. With only 10 poisoned samples, ASR reaches 67% after 16 epochs. Second, the commonly used top-k KL can accelerate backdoor transfer, causing trigger-conditioned harmful behavior to emerge earlier than sampled-token KL in most settings. Alongside these findings, we explore a simple mitigation, Lazy Defense, which clips KL rewards to make student updates less aggressive, limiting aggressive updates and slowing backdoor learning. Experiments show that Lazy Defense delays backdoor transfer in low poisoning rate settings. Together, our findings reveal that OPD can propagate backdoors, highlighting the need to address the safety risks of OPD.
