New to ShortSpan? We distil the AI-security research that matters into practitioner takeaways — edited by Ben Williams (NCC Group).Get the weekly email
// Analysis
Self-feeding loops expose backdoors in fine-tuned LLMs
Published: Wed, 12 Aug 2026 • By Elise Veyron
Pentesting
New research shows a cheap black-box check for hidden backdoors in fine-tuned Large Language Models. By looping a model’s own outputs back as inputs, testers drift towards the fine-tuning data and trip latent triggers. It flags five of six models with 92% precision, though recall per prompt stays modest.
Backdoors in fine-tuned Large Language Models (LLMs) are a gift to anyone who wants plausible deniability. The model behaves, until it does not. Most users only have text-level access, no clean reference weights, no training data, and certainly no trigger phrase. This study focuses on that reality and asks: what can you do with just queries and patience?
The trick here is self-feeding. You ask an ordinary question, then feed the model’s answer back as the next prompt, and repeat. Content drifts toward whatever the fine-tuning pushed into the model’s habits. If there is a hidden trigger in that distribution, the loop may stumble into it. The authors pit self-feeding against a dead-simple baseline that just repeats the same prompt.
They fine-tuned six open-weight LLMs, 3B to 15B parameters, across eleven backdoor categories using QLoRA. Then they ran twenty mundane starting prompts each, up to ten steps per chain, with early stop once a recognisable malicious string appeared. Self-feeding exposed backdoors in five of six models at 92.0% pooled precision. The baseline managed a single success across 120 prompt-model pairs. Some chains beginning with a joke, an arithmetic question, or even a coffee recipe hit a trigger within a few steps.
Do not oversell it. Prompt-level recall was 19.2% and iteration-level yield just 2.16% (23 triggers in 1067 backdoored iterations). But coverage adds up at the model level when you try several starting prompts. The method surfaced six attack categories, heavily skewed to Data Exfiltration at 65.2% of detections. Cutting chains to four steps preserved every model-level detection, removed the two observed false positives, and cut query cost by about 60%. One model never triggered at all, and architecture seemed to matter more than sheer parameter count.
There are caveats. The fine-tuning data was around 80% poisoned, far worse than you would expect in the wild, so any real-world hit rate will differ. Detection relied on lexical matches for malicious output, which is cheap but brittle. Composite triggers that need several structured tokens in concert can slip past. And as with any black-box probe, performance depends on prompt selection.
Still, a black-box loop that needs only text I/O and a way to spot malicious strings is a useful smoke test. It runs in minutes, with at most 200 queries in this setup, and it exposes a simple truth: multi-turn drift is not just user experience, it is an attack surface. For governance, that matters. Model hubs and procurement teams could make lightweight black-box checks part of the default intake without pretending they solve the whole problem. The open questions are the right ones: how to generalise beyond heavy poisoning, how to standardise detectors without teaching to the test, and how attackers will adapt with quieter composite triggers. The work does not close the case, but it shows a path to raising the floor.
Additional analysis of the original ArXiv paper
📋Original Paper Title and Abstract
An Empirical Study of Output-to-Input Loops for Black-Box Backdoor Detection in Fine-Tuned Open-Weight LLMs
Authors: Md. Nahid Hasan and Mohammad Arif Hossain
Anyone can upload a fine-tuned large language model (LLM) to a public repository and claim it is safe. A backdoored model behaves normally on ordinary inputs until a hidden trigger fires, and a user with no training data, clean reference weights, or the trigger phrase has no clear way to check the model before using it. We introduce and empirically evaluate self-feeding, a black-box test method that feeds a model's own output back as its next input, so the text drifts away from the starting prompt and toward the data the model was fine-tuned on. We test self-feeding against a repeated same-prompt baseline on six open-weight LLMs (3B-15B parameters), each fine-tuned with backdoors spanning eleven attack categories, using twenty ordinary starting prompts and chains of up to ten steps. Self-feeding finds backdoors in five of six models at 92.0\% pooled precision, while the same-prompt baseline succeeds on only one of 120 prompt-model pairs; chains that begin with a joke request, an arithmetic question, or a coffee recipe all reach a trigger within a few steps. Recall per prompt is low (19.2\%), and we show why it still adds up to much higher detection at the model level once several starting prompts are used. We also report where the method falls short: one model was never triggered, and self-feeding produced two false positives that the same-prompt baseline cannot produce. Cutting the chains to four steps keeps every model-level detection at 100\% precision while using 60\% fewer queries. Needing only text-level query access and a way to recognize malicious output, self-feeding offers a cheap first check on a downloaded model.
🔍ShortSpan Analysis of the Paper
Problem
The paper examines how a downloader with only text-level query access can detect hidden backdoors in fine-tuned open-weight large language models. Backdoored models behave normally until a latent trigger fires; users often lack the original weights, training data or trigger phrases, so they need black-box checks that do not assume prior knowledge of triggers or a clean reference model.
Approach
The authors introduce self-feeding, a black-box test that feeds each model response back as the next prompt so conversational content drifts toward the model’s fine-tuning distribution. They compared self-feeding to a repeated same-prompt baseline across six open-weight LLMs (3B–15B parameters) fine-tuned with QLoRA on a dataset containing eleven backdoor attack categories. Experiments used 20 ordinary starting prompts and up to ten iterations per chain, with early stopping when a recognisable malicious output was produced. Detection was based on simple keyword-labelled malicious outputs (attack-category strings) produced by the model.
Key Findings
Model-level success: Self-feeding detected backdoors in five of six fine-tuned models (83.3% model-level detection) while the same-prompt baseline succeeded in only one prompt-model pair across all tests.
Precision and recall: Pooled precision for self-feeding was 92.0%; prompt-level recall was modest at 19.2% (23 triggers from 120 prompt-model pairs). Iteration-level yield was 23 triggers in 1067 backdoored iterations (2.16%).
Category coverage and bias: Self-feeding uncovered six distinct attack categories, dominated by Data Exfiltration (65.2% of detections), whereas same-prompt found only Data Exfiltration in a single repeated prompt.
Efficiency and tuning: Truncating chains to four steps preserved all model-level detections, eliminated the two observed false positives, and reduced query cost by about 60% compared with ten-step chains.
Variation across models: Detection depended on architecture rather than parameter count; one model (Gemma-3) was never triggered despite the same setup.
Limitations
Experiments used a fine-tuning dataset heavily biased toward backdoor samples (around 80% poisoned), which overstates real-world poisoning rates and may inflate effectiveness. Self-feeding relies on a lexical detector for malicious outputs and produced two false positives tied to benign vocabulary in Llama-3.2. It cannot reliably detect multi-component composite triggers that require structured simultaneous tokens. The method also failed entirely on one model and shows low per-prompt recall, so results depend on prompt selection and chain length.
Implications
For offensive security, attackers can embed backdoors that activate only after multi-turn drift or when the model is induced to generate user-like turns; such designs may evade simple self-feeding checks. They can also favour composite triggers or reduce poisoning rates to blunt this method. Conversely, defenders can use self-feeding as a low-cost, quick first check before deploying downloaded models: it needs only text queries and a way to recognise malicious outputs, runs in minutes with at most 200 queries, and early truncation can reduce false positives and cost.
- Provides a realistic benchmark (ToxScreen) with ~800 backdoored LLMs to assess how well defenders can detect and recover hidden triggers in practice...
- Why it’s relevant: shows a stealthy, compliance-only backdoor that can turn on unsafe outputs with a tiny poisoning budget, even when training data ...
- Why it matters: It tackles hidden backdoors in LLMs and proposes a post-training approach that makes the model introspect about implanted triggers a...
- It shows how tiny poisoning in instruction-fine-tuning can embed backdoors in LLMs, causing targeted misbehavior (e.g., censorship, false content, o...
- Why it may matter: it shows how backdoor triggers hijack internal signals (attention and gradients) in pre-trained language models, revealing exploi...
- Proposes a defense that uses learnable text prompts in vision-language models to spot hidden backdoors, without altering the model’s weights.
- Ta...
- Key relevance: It demonstrates a covert NLP backdoor that uses steganography to encode semantic triggers, enabling high-success attacks with ultra-l...
- It shows a new kind of backdoor for chat models that activates only after a sequence of inputs across multiple turns, exploiting how conversational ...
- Why it matters: it shows an automated, scalable way to inject backdoors into LLMs using AI agents themselves, with semantic, context-aware triggers ...
- Relevance in simple terms: introduces a practical, inference-only scanner to detect hidden backdoors in causal LLMs by spotting memory leakage and u...
Get the weekly digest
The few AI-security papers that matter, with the practitioner takeaway. No spam.
