LLM agents break trust boundaries; favour deterministic controls

Problem

This paper examines security risks introduced by frontier AI agents, arguing that agent architectures fundamentally change core assumptions about code-data separation, authority boundaries, and execution predictability. Those shifts create new confidentiality, integrity and availability failure modes. The authors emphasise that agentic systems accept high-level goals, dynamically construct workflows, and chain tool calls, which increases opportunities for data exfiltration, unauthorised actions, cascading failures in long-running workflows and harder-to-reason-about states compared with traditional software.

Approach

The analysis draws on Perplexity’s operational experience with large-scale, general-purpose agent systems and surveys attack surfaces, threat actors and failure modes across tools, connectors, hosting boundaries and multi-agent coordination. It highlights concrete threat classes such as indirect prompt injection, confused-deputy behaviour and workflow cascades, evaluates existing mitigations as a layered defence stack (input-level and model-level measures, sandboxed execution and deterministic policy enforcement), and identifies standards and research gaps including benchmarks, delegation and privilege policy models, and secure multi-agent design aligned with NIST risk management principles.

Key Findings

• Agent architectures blur the line between code and data so plaintext prompts and model outputs can act as executable control, increasing injection and logic‑confusion risks.
• High‑risk threat classes include indirect prompt injection where adversarial content retrieved during normal operation manipulates agents, classic confused‑deputy vulnerabilities in multi‑agent chains, and cascading failures in long multi‑step workflows.
• Attack surfaces span tool selection and execution logic, web‑grounded ingestion, skill and plugin supply chains, orchestrators and shared workspaces, and hosting/exposure choices such as gateways, web UIs and webhooks.
• Deployment and hosting choices materially change exposure: cloud hosting adds network trust boundaries, on‑prem or local deployments reduce internet exposure but raise insider and configuration risks; weak sandboxing for code execution is a key vector.
• Defences are best organised as defence‑in‑depth: input‑level detection and sanitisation, model‑level instruction hierarchies and embedding tricks, system‑level sandboxing and execution monitoring, and a deterministic last line of defence (allowlists, rate limits, schema validation). Deterministic controls are currently the most mature in production.
• Operational constraints impede some defences: prompt‑injection detectors face base‑rate false positive problems and performance overheads; model‑level hierarchies remain conventions rather than verifiable guarantees; fully specifying legitimate control flows in open‑ended systems is an open challenge.
• Concrete incidents illustrate hazards, including documented vulnerabilities in the OpenClaw platform and a one‑click remote code execution case that did not depend on LLM behaviour.
• There is a gap in realistic, adaptive security benchmarks and in policy models for delegation, privilege control and inter‑agent trust.

Limitations

Many defences are probabilistic and cannot assure security alone. Input detectors suffer from false positives and cost/latency trade‑offs. Model‑level instruction hierarchies are learned conventions rather than enforced guarantees. System‑level sandboxing depends on correctly identifying legitimate control flows, which is difficult in open‑ended agent use. Research maturity varies: deterministic enforcement and human‑in‑the‑loop controls are mature, other approaches are early or experimental.

Why It Matters

Practitioners must treat agent design and hosting decisions as first‑order security factors and adopt layered defences with at least one deterministic enforcement layer. Standards for inter‑agent communication, adaptive adversarial benchmarks, access control models that combine role‑based and risk‑adaptive approaches, and research into usable human‑agent governance are urgent priorities. These measures will reduce exfiltration, unauthorised actions, privilege escalation across agents and availability attacks, and support auditing and recovery in increasingly agentic systems.