Prompt injection defences miss attacks that ask for less
Reviewing On Task, Off Track: Benchmarking LLM and Agent Robustness to Near-Target Prompt Injection · Yuyang Gong, Zihao Wang, Jiawei Liu et al.
Researchers show that LLMs and agents which resist obvious prompt injection are still steered by instructions that keep the task intact but make one part wrong or harmful. Benchmarks that only test whole-task hijacks therefore overstate robustness, including for a model trained specifically to resist injection.
A model trained specifically to resist prompt injection let almost none of the standard attacks through. Then the researchers changed what the attacker asked for. The injected instruction no longer demanded that the model abandon the user's task. It asked the model to finish the task and get one thing wrong. Success jumped from 3.2% to 36.7%.
That model was Meta-SecAlign-70B, and it is the clearest illustration of what this study is about. Most prompt injection benchmarks plant an instruction in untrusted content, such as a retrieved document or a tool result, and check whether the model obeys it. The planted instruction usually wants something unrelated: print a phrase, call a different tool, ignore the user. A model that learns to spot a hijack of the whole task scores well.
The researchers call the alternative near-target prompt injection. Take a legitimate task and its correct answer. Ask an LLM to produce a version that looks almost the same but is wrong or harmful, then write an instruction that steers towards it. A summary with one altered fact. A call to the right tool with one key argument changed. A correct answer that also discloses something it should not. Each target was checked automatically and by humans. Errors the model already made without any injection were not counted.
The attack works because it does not look like an attack. The model is still doing what the user asked. It is just doing it slightly wrong, and nothing in that resembles the hijacks it was trained to refuse. It is also harder for the user to catch, because most of the answer is right.
Across seven models and ten language datasets, every model fell for these more often than for the fixed-target attacks, with only three attempts per sample. Incorrect targets gained the most. Harmful targets gained less, but still gained. The same pattern held in two agent benchmarks, where success meant matching both the tool invoked and its key arguments.
The limits are clear. The attacks were generated with access to the reference answer, so the attacker knew what correct looked like. Closeness was encouraged by prompting rather than measured against a fixed distance. The attack set was fixed in advance and never tuned against any particular model. So the study does not measure the worst case, and an adaptive attacker with more attempts has no reason to do worse.
What interests me is not that a trained defence fails, because every defence fails against something. It is that it fails against an attacker who simply asks for less. The training taught the model to refuse a hijack, not to notice a nudge. If you are testing an agent, check whether it can be talked into a plausible wrong answer, not just an obviously foreign one. Passing the obvious test proves the obvious attack fails. Nothing more.
Further analysis
Problem
This paper studies whether large language models and agents resist subtle prompt injection attacks that steer outputs toward incorrect or harmful outcomes while preserving most of a legitimate answer. Such near-target prompt injections remain plausible to a user and may evade defences that only block obviously unrelated or overtly malicious instructions. Demonstrating these failures matters because evaluations that test only distant or fixed-target attacks can give a misleading sense of robustness.
Approach
The authors introduce NTBench, a benchmark that constructs attack pairs from legitimate tasks and reference outcomes. For each task a reference response guides generation of candidate target responses and injected instructions in two separate tracks: incorrect-targets, which introduce factual or task-related errors, and harmful-targets, which introduce content or actions judged harmful under a fixed policy inspired by official safety criteria. Targets are generated by an LLM conditioned on the original task and reference, then filtered by automated checks and human validation to ensure they are genuinely incorrect or harmful. Attacks are inserted into untrusted content provided to a victim model or agent; success is assessed by response judgements or tool-call checks and excludes deviations already observed without injection. The evaluation uses seven victim LLMs, ten language datasets and two agent benchmarks, with a common budget of three attack executions per sample and fixed attack sets shared across models. An LLM judge is used for large-scale assessment, supplemented by human audits to measure validation and judgement reliability.
Key Findings
- Near-target attacks increase attack-success rates versus a fixed-target baseline for every evaluated model and setting under ASR@3. On language tasks incorrect-target ASR@3 rises by 9.3 to 35.8 percentage points and harmful-target ASR@3 by 1.7 to 17.3 percentage points.
- Model-specific robustness claims can be misleading: Meta-SecAlign-70B, trained to resist prompt injection, shows ASR@3 moving from 3.2% (fixed-target) to 36.7% (incorrect) and 18.4% (harmful), demonstrating susceptibility to near-target probes.
- Near-target vulnerabilities also appear in agentic workflows; tool-call targets require matching the invoked tool and key arguments and still produce higher ASR compared with fixed-target baselines across AgentDojo and AgentDyn suites.
- Validation and labelling are reliable: human audits confirm retained near-targets are largely valid (language incorrect 91.3%, agentic incorrect 98.0%; language harmful 86.7%, agentic harmful 92.0%) and judge-model assessments agree with majority human labels (F1 around 84–90%, agreement 91–93%, Cohen's kappa 0.78–0.85).
Limitations
The benchmark encourages proximity to references by prompting rather than enforcing a formal distance metric or numerical radius. Generation is assisted by access to benchmark references and is evaluated in a black-box setting without model-specific adaptation. Results depend on chosen tasks, fixed policy definitions of harmfulness, the three-attempt budget, and the attack set fixed prior to evaluation; they do not quantify worst-case attack success under adaptive adversaries or larger budgets.
Implications
Offensive security implications are that attackers can craft plausible, task-related instructions that subtly corrupt outputs or cause harmful disclosures or unauthorised actions while remaining close to legitimate responses. Defences and audits that focus on distant or fixed-target attacks can miss these failure modes, so red-teaming and robustness evaluations should include reference-guided near-target probes. In deployed systems, this increases the risk of hard-to-detect manipulations affecting factual integrity, privacy and agent behaviour, informing attacker modelling, audit scope and governance priorities.
The paper title, authors and abstract
On Task, Off Track: Benchmarking LLM and Agent Robustness to Near-Target Prompt Injection
Prompt injection benchmarks assess whether large language models (LLMs) and agents resist malicious instructions embedded in external content. However, a system could perform well by rejecting attacks that obviously abandon the user's request, while still accepting changes that retain most of a legitimate answer but make it incorrect or harmful. Testing more instructions for unrelated outcomes would not necessarily reveal this weakness. We call attacks seeking responses designed to resemble legitimate answers while introducing incorrect or harmful changes near-target prompt injection. We introduce NTBench, the Near-Target Benchmark, which builds on legitimate tasks from existing datasets. For each task, a legitimate reference outcome guides the generation of target--instruction pairs, with separate tracks targeting incorrect and harmful outcomes in responses or agent tool invocations. Harmful targets can also accompany otherwise correct task completion. We combine LLM-based checks with human validation of the intended deviations, then assess whether injected instructions induce them using response judgments or tool-call checks, excluding deviations already observed without injection. We evaluate seven LLMs on ten datasets for language tasks and two agent benchmarks. Under a common budget of three attack executions per sample, both tracks achieve higher overall attack success rates than the fixed-target baseline for every evaluated model in each setting. On language tasks, the increases range from 9.3 to 35.8 percentage points for incorrect targets and 1.7 to 17.3 percentage points for harmful targets. These results show that resistance to the tested fixed-target attacks does not imply resistance to related incorrect or harmful outcomes, and that reference-guided targets broaden the failures revealed by robustness evaluation.
