AI agent tools trust model arguments they should check
Reviewing ToFu: Exploiting and Fuzzing the LLM-Orchestrator-Tool Trust Boundary · Md Nazmul Islam Ananto, Pierre Prevost-Avenel, Colin Smith et al.
Researchers fuzzing real agent tools find that many never enforce the input rules their own descriptions promise. Path traversal, request forgery and command injection reach tool code because developers assume the model will only send sensible arguments, and that assumption does not hold once an attacker can steer the model.
An agent asks a tool to fetch a web page. The argument it passes is a file: URL pointing at a local file, so the tool reads the file and hands back its contents. Nothing in the chain objected. Every part assumed another part had checked.
That gap is the subject of new research on the boundary between a language model, the orchestration layer around it and the tools it calls. The orchestrator shows the model each tool's description and input schema. The model replies with a structured call. The orchestrator confirms the call fits the schema, then runs the tool.
The weakness sits between the schema and the description. A schema says an argument is a string. The description says it is a URL, or a path inside a project folder. Only the description carries the security rule, and it is written for the model to read. Often nothing in the code enforces it.
The researchers built a fuzzer, ToFu, to measure how often that happens. It reads each tool's metadata and matches argument descriptions to a library of checks grouped by weakness class. It then generates schema-valid inputs that travel the normal orchestration path. Then it watches what the tool actually does: which files it opens, which processes it starts, which hosts it contacts. That last part matters, because a path traversal does not crash anything. A crash-based fuzzer would walk straight past it.
Run across 273 servers built on the Model Context Protocol (MCP), it found verified bugs in 343 tools on 118 of them. The classes are familiar: path traversal, log injection, server-side request forgery reaching loopback and metadata endpoints, and a few OS command injections. Most of the bugs appeared within the first minute of fuzzing.
There is nothing new in any individual bug. What is new is the caller. These tools were written as if the model were a careful client that would never ask for something outside the description. It is not. The researchers show this with a repeat-after-me style attack that gets the orchestration layer to execute arguments the attacker supplied. The model is a channel, not a filter.
The attacker still needs some way to influence what the model sees or says. The fuzzer has blind spots too. It only tests what it can run, so tools behind missing credentials or external backends go unexamined. Its detailed tracing works only on Python, and many candidate findings needed a human to confirm them. I would read the count as a floor.
If you write tools, enforce the description in code. Resolve paths against an allowed root, and restrict URL schemes and destinations. A schema is a type check, not a policy.
What struck me is how much of this code rests on that one assumption. The model was never a security check.
Further analysis
Problem
The paper studies security at the LLM–orchestrator–tool trust boundary: orchestration layers present tool metadata and schemas to large language models, interpret model outputs as structured tool calls, and then execute those calls. This creates a weakness because conformity to a schema does not guarantee that tool implementations enforce the semantic constraints implied by their natural‑language descriptions. Malicious or attacker‑influenced model outputs can therefore reach tool code and trigger security‑sensitive operations despite appearing syntactically valid. The authors also demonstrate practical manipulation of this boundary using a repeat‑after‑me style attack that causes an orchestration layer to execute attacker‑supplied arguments.
Approach
The authors present ToFu, an automated, tool‑aware fuzzing framework that derives semantic security checks from tool metadata and a corpus of semantics‑aware sanitizers organised around CWE weakness classes. Each sanitizer pairs applicability, adversarial input classes, an expected security property and the observations required to judge a violation. ToFu maps argument descriptions to sanitizers (an LLM assists only in mapping and seed construction), specialises checks per tool, and generates schema‑valid fuzzing harnesses whose mutated arguments traverse the normal orchestration path. It runs a coverage‑guided fuzzer (Atheris) and evaluates executions using two oracle modes: runtime introspection (tracers that log filesystem, process and network operations for Python targets) and outcome sanitisation for opaque or external tools. The corpus is extensible and the system records triggering invocations, affected arguments, expected properties and observed evidence.
Key Findings
- Large‑scale discovery: evaluated across 273 reachable MCP servers with 3,405 extracted tools and produced 508 candidate violations, of which 253 were manually verified across 343 tools in 118 repositories.
- Bug classes and counts: verified findings include 115 reflected‑output bugs, 64 path‑traversal bugs, 58 log‑injection bugs, 7 SSRF bugs, 5 data‑exfiltration bugs, 3 OS command‑injection bugs and other classes; many would not be found by crash‑based fuzzers.
- Operational performance: ToFu selects on average three sanitizers per tool, makes two LLM calls per tool (selection and seed generation), and is lightweight: 73% of tools reach 90% of their final coverage within one minute and 80% of reproducible bugs appear within 60 seconds; a 60‑second budget found 231 of 253 verified bugs.
Limitations
ToFu requires the target to be executable in the test environment and cannot reach code behind missing credentials, external backends or startup failures. Runtime introspection is available only for instrumented Python targets; JavaScript targets lack in‑process tracers. Some semantic problems that leave no observable runtime trace are better suited to static analysis. The sanitizer mapping step can miss applicable checks and ToFu initially reported many candidates that required manual verification to confirm. Multi‑agent, stateful sequences and third‑party component coverage are left for future work.
Implications
Offensively, an adversary who can influence model inputs or orchestration output can cause tools to perform sensitive actions: read arbitrary local files via file: URLs, access loopback or metadata endpoints, exfiltrate data, inject into logs or reflected outputs, and invoke commands. The work shows that many tool implementations implicitly rely on the LLM to sanitise inputs; treating model‑generated arguments as trusted is unsafe. The exposed attack surface is widespread and amenable to automated discovery and exploitation.
The paper title, authors and abstract
ToFu: Exploiting and Fuzzing the LLM-Orchestrator-Tool Trust Boundary
Large language models (LLMs) are increasingly deployed as agents that interact with external software through tools, with an orchestration layer translating model outputs into structured tool invocations. This architecture creates an LLM-orchestrator-tool trust boundary where tool semantics may go unenforced and LLM-generated arguments may be incorrectly treated as trusted. We present ToFu, a tool-aware framework that automatically derives semantic security checks for LLM tools from their existing metadata and a CWE-guided sanitizer corpus. ToFu specializes these checks to individual tools and automatically generates fuzzing harnesses and semantic oracles using runtime introspection and outcome sanitization. We evaluated ToFu across 273 MCP servers containing 3405 tools and identified verified bugs corresponding to CWE weakness classes in 343 tools across 118 of the servers. Our findings show that real-world tools frequently fail to enforce their documented input semantics, making it unsafe to treat LLM-generated arguments as trusted.
