Black-Box Red Teaming Exposes Agentic AI Risks

Problem

The paper studies systematic discovery of security and safety risks in agentic AI systems that read untrusted inputs, call tools with real permissions, persist state, and act autonomously. Standard evaluations designed for single-turn chat models miss multi-step failures such as indirect prompt injection, memory poisoning, unsafe tool execution and specification gaming. Uncovering these vulnerabilities is important because agents expand the attack surface beyond chat-only models and can perform consequential actions in production.

Approach

The authors propose a black-box, taxonomy-driven red-teaming framework that needs only a basic system description and public interfaces. Core elements are a seven-domain risk taxonomy, an automated SAGE-RT scenario generator producing 120 adversarial multi-turn scenarios per domain, and automated LLM judges followed by human validation. Scenarios are 3–5 turn interactions generated with parameters (temperature 0.8, top_p 0.95, max_turns 5) and diversity controls (cosine similarity threshold <0.7). Automated scoring uses a domain-specific rubric applied by an LLM judge, with high-risk findings (score > 0.6) manually reviewed and given additional weight. The pipeline was evaluated in a fully black-box manner against two agent architectures (a single-agent, multi-tool CrewAI Restaurant Receptionist and a multi-agent AutoGen Stock Advisory assistant) using four base models: gpt-4.1-nano, Mistral Small, Gemini-2.5-Flash and Kimi K2 Instruct.

Key Findings

• Automated discovery at scale: SAGE-RT generated and executed 120 taxonomy-aligned adversarial scenarios per domain, enabling systematic coverage of simple to complex attack campaigns.
• High governance exposure: Governance risk averaged 56.25% across evaluated systems, appearing consistently across models and architectures.
• Severe privacy risk in multi-agent setups: The multi-agent Stock Advisory assistant showed 65.00% privacy risk, driven by inter-agent data flows and tool-mediated leaks.
• Agent behaviour manipulation can be catastrophic: Agent behaviour vulnerabilities reached up to 85% in one Stock Advisory configuration (GPT-4.1-nano), indicating susceptibility to social engineering and handoff poisoning.
• Architecture matters more than model choice: Overall risk clustered by system design rather than by base model; Restaurant agent overall risk averaged 29.49 ± 3.65 while Stock agent averaged 40.53 ± 5.70.
• Tool misuse and access control were lower on average but still present: Tool misuse means were 15.00 for the Restaurant agent and 28.75 for the Stock agent; access-control issues varied and surfaced when interfaces allowed broader scopes.

Limitations

The framework focuses on vulnerability discovery rather than validation of mitigations, cannot assess implementation-specific code-level flaws, primarily tests English-language interactions, and may need updates as agent architectures evolve. Automated judges have lower agreement on borderline cases, requiring manual review for high-risk findings.

Implications

Offensive security implications are significant: an attacker can craft multi-step prompts to exfiltrate PII, coerce agents into destructive actions (for example bulk cancellations or unsafe trades), exploit confused-deputy patterns to escalate privileges, steal credentials via plugin or tool interactions, and manipulate multi-agent handoffs to alter objectives or bypass rules. Multi-agent coordination and permissive tool interfaces amplify these attack paths, making relatively low-effort adversarial campaigns effective in production agent deployments.