New to ShortSpan? We distil the AI-security research that matters into practitioner takeaways — edited by Ben Williams (NCC Group).Get the weekly email
// Analysis
Stop Calling Tools Autonomous: Demand Human Oversight
Published: Mon, 30 Jun 2025 • By Elise Veyron
Defenses
New research shows many cybersecurity AIs are semi-autonomous, not independent agents. That mislabeling risks reduced human oversight, false positives, and legal exposure. The paper offers a six-level taxonomy and urges clear capability disclosure, human validation, and governance so organizations capture AI speed without ceding critical decisions.
Recent research exposes a gap between marketing and reality: many security tools billed as autonomous are actually semi-autonomous and still need human judgment. That gap matters because treating automation as autonomy can shrink oversight exactly when attackers probe boundaries.
Define terms first: automation means a system performs repeatable tasks under fixed rules. Autonomy means a system plans, adapts, and makes high-stakes decisions independently across varied contexts. The paper adapts a six-level taxonomy to place current tools mostly at Level 3 or 4. For example, high-profile wins like XBOW required human vetting of thousands of findings before disclosure.
Policy and governance intersect with controls in clear, practical ways. Policy sets expectations for responsibility, disclosure, and acceptable risk. Governance translates those expectations into controls: clear vendor labeling of autonomy level, mandatory human-in-the-loop checkpoints for critical actions, logging and audit trails, and legal signoffs for offensive testing. These controls are not just bureaucracy. They directly reduce harm from false positives, unsafe exploit attempts, and misguided automated remediation.
There are trade-offs. Human review slows speed and increases cost. Strict gating can blunt innovation or frustrate product teams. But performative compliance that checks boxes without technical validation is worse: it creates blind spots while claiming safety.
Practical steps
This quarter: inventory AI security tools, require vendors to label autonomy level, enforce human validation for high-impact actions, enable detailed logging, and run tabletop exercises on failure modes.
Later: build formal governance frameworks, fund third-party audits and red teams, adopt shared standards for capability disclosure, and train staff on human-AI partnership.
Short-term controls buy time and reduce exposure. Long-term governance shapes incentives so AI accelerates security rather than erodes it.
Additional analysis of the original ArXiv paper
📋Original Paper Title and Abstract
Cybersecurity AI: The Dangerous Gap Between Automation and Autonomy
The cybersecurity industry combines "automated" and "autonomous" AI,creating dangerous misconceptions about system capabilities. Recent milestoneslike XBOW topping HackerOne's leaderboard showcase impressive progress, yetthese systems remain fundamentally semi-autonomous--requiring human oversight.Drawing from robotics principles, where the distinction between automation andautonomy is well-established, I take inspiration from prior work and establish a6-level taxonomy (Level 0-5) distinguishing automation from autonomy inCybersecurity AI. Current "autonomous" pentesters operate at Level 3-4: theyexecute complex attack sequences but need human review for edge cases andstrategic decisions. True Level 5 autonomy remains aspirational. Organizationsdeploying mischaracterized "autonomous" tools risk reducing oversight preciselywhen it's most needed, potentially creating new vulnerabilities. The pathforward requires precise terminology, transparent capabilities disclosure, andhuman-AI partnership-not replacement.
🔍ShortSpan Analysis of the Paper
Problem
The paper examines widespread confusion in cybersecurity between "automated" and "autonomous" AI and argues this mislabelling creates dangerous misconceptions about system capabilities. Misrepresented autonomy can reduce human oversight at moments when adversaries actively probe boundary conditions, creating new vulnerabilities and legal or ethical exposure for organisations.
Approach
The author adapts robotics principles and prior work to propose a six‑level taxonomy of cybersecurity autonomy (Level 0 to Level 5) and analyses contemporary AI pentesters and frameworks as case studies, including XBOW, CAI, AutoPT, Vulnbot and PentestGPT. The study is conceptual and comparative rather than an experimental benchmark. Datasets, model architectures and detailed quantitative evaluations: not reported.
Key Findings
A clear six‑level taxonomy distinguishes automation from true autonomy in cybersecurity, clarifying capabilities associated with each level.
Most advanced systems today sit at Level 3–4 (semi‑autonomous): they can plan, scan, exploit and suggest mitigations but rely on humans for edge cases, validation and strategy.
High‑profile results (for example, XBOW topping a HackerOne leaderboard) involved human review; XBOW produced 1,060 automatically generated findings that were vetted before disclosure.
AI agents speed up vulnerability discovery and democratise testing but generate false positives and require validators and human oversight to avoid harmful actions.
Level 5 full autonomy is aspirational and currently unmet; technological, ethical and legal barriers remain significant.
The work is primarily conceptual and draws on case studies rather than controlled experiments. Precise evaluation metrics, datasets and reproducible experiments are not reported. The taxonomy simplifies a complex continuum and may not capture all deployment contexts.
Why It Matters
Practically, conflating automation with autonomy can lead organisations to reduce oversight when it is most needed, increasing security risk. The paper recommends precise terminology, transparent capability disclosure, rigorous validation, and human‑AI partnership to retain ethical, legal and strategic safeguards while benefiting from AI acceleration in security testing.
- Proposes an open, threat-focused digital twin to securely test learning-enabled autonomous systems under realistic attack scenarios, even when live ...
- It offers an agent-based security paradigm with a meta-cognitive judge that decides when and how autonomous actions are taken, aiming for explainabl...
- It shifts focus from fortress-style tech defenses to protecting AI reasoning itself, addressing how adversaries can manipulate judgments rather than...
- It tackles not just what an AI aims for, but who gets to pick the options it presents, which is a key attack surface for manipulation.
- It proposes...
- Proposes the Hierarchical Autonomy Evolution (HAE) framework to categorize security for autonomous AI agents into three levels: cognitive, execution...
- Why it matters: It maps how moving from single LLMs to multi-agent, tool-augmented pipelines creates new attack surfaces and failure modes (e.g., mi...
Get the weekly digest
The few AI-security papers that matter, with the practitioner takeaway. No spam.
