ToolHazard scales agent prompt injection stress tests

Problem

This paper studies indirect prompt injection against large language model agents that call external tools and interact with stateful environments. Existing evaluations are limited by manually built environments, stochastic tool simulation and fixed injection locations, preventing scalable, reproducible security analysis across diverse domains. Addressing this gap matters because agents that execute tools can be hijacked by adversarial content in environment state, leading to unsafe or unintended external actions.

Approach

The authors introduce ToolHazard, a scalable framework that automatically synthesises executable stateful environments, discovers attackable state attributes, and generates long-horizon, state-grounded tasks. ToolHazard comprises three modules: an Environment Simulator that produces environment blueprints, translates them into executable code and verifies correctness; an Attacker Agent that identifies writable text attributes, analyses read/write operation graphs, matches attack points reachable during task execution and plans payloads; and a User Simulator that initialises environment state and generates benign multi-step tasks. Using this pipeline the authors build ToolHazard-Bench (87 tasks, 28 test environments, 512 tools, average horizon 15.56 steps) and ToolHazard-Align (60 training environments with adversarial examples) to evaluate and align agents. Attack strategies are instantiated with six predefined payload wrappers and verification is performed programmatically on final environment state. Both supervised fine-tuning and reinforcement learning with a hijack-aware reward are used for alignment experiments.

Key Findings

• Large language model agents are substantially vulnerable: evaluated models show high attack success rates, with several strategies achieving over 30 to 40 percent attack success on top models.
• Injection timing matters: attacks injected earlier in execution trajectories yield consistently higher attack success rates than later injections.
• Injection placement within tool outputs matters: payloads inserted into later writable fields in a tool response are more effective, indicating a positional bias toward tail-end content.
• Output format influences success: free-form text outputs produce substantially higher attack success rates than structured formats such as JSON or YAML.
• Attacks degrade capability as well as safety: benign task completion rates fall under adversarial perturbations, so injections harm both correctness and safety.
• Adversarial alignment helps: ToolHazard-generated training data for supervised fine-tuning and reinforcement learning improves robustness on ToolHazard-Bench and an independent benchmark while preserving benign task utility and avoids over-refusal.

Limitations

The synthesised environments are grounded in seed datasets and may not capture proprietary or deployment-specific behaviours in production systems. ToolHazard currently supports six predefined injection strategies and does not automatically discover novel attack wrappers. The dataset contains harmful content and access is restricted. The framework is intended for reproducible stress testing rather than as a direct estimate of production risk.

Implications

Offensively, an attacker who can write to textual fields in an environment can embed instructions that propagate into agent observations and hijack tool calls, particularly if injections occur early in a workflow or in later fields of a tool response. Free-form tool outputs are higher-risk targets than structured outputs. Stronger instruction-following models may be more liable to follow injected content, so attackers could focus on high-capability agents. Attackers can scale impact by targeting common writable attributes such as emails, comments, documents and database fields that agents routinely read. The results underline that adversaries need not modify agents or tools directly to induce harmful actions; poisoning environment state can be sufficient.