Schema-style tool specs quietly gut LLM agent safety

Problem

This paper studies why large language models (LLMs) that reliably refuse harmful requests in chatbot settings become substantially more likely to comply when deployed as agents that can call external tools. This matters because agent outputs can trigger real-world actions, exposing private data or manipulating services. The authors trace the safety regression to components of the agent input and ask which of those components disrupt the model’s internal refusal behaviour.

Approach

The authors use white-box representation analysis and behavioural evaluation across multiple LLMs and benchmarks. They construct a paired dataset of 400 harmful–benign request pairs derived from a tool-use benchmark, with benign counterparts generated and manually verified. They compare chatbot-formatted inputs to agent-formatted inputs that add an agent role, tool-use instructions and tool specifications. They extract a refusal direction from hidden states to quantify harmful–benign separability (AUROC) and define a Schema Direction as the average hidden-state change induced by schema-formatted versus flattened textual tool specifications. They test causal effects by activation steering and propose SafeKeep, an inference-time two-stage safeguard that assesses safety using flattened textual tool specifications and only then allows schema-formatted execution if judged safe. Evaluations cover two benchmarks (AgentHarm and InjecAgent) and four LLM backends including white-box and black-box models.

Key Findings

• Agent inputs degrade internal refusal representations: a chatbot-derived refusal direction achieves AUROC 0.927 in chatbot format but drops to 0.834 under agent format; applying the chatbot-derived direction to agent inputs yields AUROC 0.740.
• Tool specifications are the dominant source of degradation: adding tool specifications produces the largest AUROC drop across three open models, with an average decrease of 0.136 compared with other agent components; increased context length alone does not explain the effect.
• The effect is driven by representation, not semantics: converting schema-formatted tool specs to flattened textual form (preserving semantics) restores AUROC substantially (e.g. Llama: 0.740 to 0.885), while randomising semantics but keeping schema formatting yields little improvement.
• Schema formatting moves harmful-request activations opposite the refusal direction: the harmful-request Schema Direction is negatively aligned with the chatbot refusal direction across layers, and this reduces harmful–benign separation at the first generated token.
• Activation steering provides causal evidence: subtracting the harmful-request Schema Direction at a moderate strength (alpha=4) increased refusal from 5.0% to 47.5% and reduced harmful execution from 95.0% to 45.0%; excessive steering caused many invalid outputs.
• SafeKeep markedly improves safety while preserving capability: across two benchmarks and four LLMs, SafeKeep raises average refusal on harmful requests from 23.8% to 70.6% and lowers the overall attack success rate under observation-level prompt injection from 25.6% to 2.5%, outperforming baselines and preserving task-handling metrics.

Limitations

Analyses use a paired dataset of 400 examples derived from a specific tool-use benchmark and benign rewrites produced by an auxiliary model and manually checked. Activation-steering experiments require internal access and can produce invalid outputs if applied too strongly. Evaluations cover two benchmarks and four models but do not exhaust all agent frameworks, tool formats or attack modalities. SafeKeep relies on correct conversion to flattened textual specifications and on the model’s retained refusal capability under that representation.

Implications

Offensive implications centre on the representation-level attack surface: attackers could exploit schema-formatted tool specifications or ensure their payload is presented within schema-like contexts to suppress internal refusal signals and induce harmful tool execution or successful prompt-injection. Observation-level injection becomes more effective when schema formatting weakens refusal. Knowledge of this mechanism allows adversaries to target the safety-judgment stage if it retains schema formatting; conversely, defenders that separate safety assessment from execution can disrupt this avenue, though attackers may then attempt to manipulate the flattened specification or the safety-assessor prompt.