Autonomous AI outperforms some pentest benchmarks

Pentesting

Published: Wed, Sep 17, 2025 • By Theo Solander

Autonomous AI outperforms some pentest benchmarks

xOffense introduces an autonomous, multi-agent penetration testing framework driven by a domain‑tuned mid‑scale large language model (Qwen3‑32B). In benchmark tests it achieves 72.72 percent task completion and 79.17 percent sub‑task completion, outperforming several recent systems. The result promises scalable assessments while amplifying dual‑use and safety concerns for defenders.

Lede: A new paper presents xOffense, an autonomous penetration testing framework that pairs a fine‑tuned mid‑scale large language model (LLM) with multi‑agent orchestration to run end‑to‑end offensive workflows. In standard benchmarks the system posts substantially higher task and sub‑task completion rates than recent automated approaches, a result that matters because automation both scales testing and lowers the skill barrier for abuse.

Nut graf: For security teams and decision makers this is a practical pivot. xOffense shows that domain adaptation and structured coordination can make a modestly sized open model effective at multi‑stage attacks. That improves the economics of routine red‑team work but also tightens the timeline in which misconfigurations or leaked capabilities can be repurposed by threat actors.

Background: Traditional pentesting relies on small expert teams. Earlier automated work used machine learning with narrow action spaces or expensive proprietary models. The authors instead fine‑tune Qwen3‑32B on chain‑of‑thought pentesting data, use low‑rank adaptation (LoRA) and runtime optimisations to run on modest hardware, and slot the model into five specialised agents for reconnaissance, scanning and exploitation.

How it works: A Task Orchestrator coordinates agents while a Knowledge Repository provides retrieval‑augmented context. The model generates concrete tool commands; the system exposes a Task Coordination Graph to manage dependencies. Retrieval augmented generation (RAG) boosts real‑world target performance in the experiments.

Impact and risk: The core implication is dual use. Automating multi‑step exploitation makes scalable, repeatable assessments feasible for defenders, but the same automation can compress attacker timelines and widen access to exploit playbooks. The paper notes, and the results imply, that mid‑scale domain‑tuned models can match or beat larger generic models on specific tasks.

Mitigations and next steps: The authors propose design patterns to reduce unsafe actions: validation gates, sandboxing, command verification, robust monitoring, structured orchestration and phase grounding. Practitioners should run experimental agents in isolated labs, harden access controls, and add human‑in‑the‑loop approval before any live action.

Limitations and caveats: Benchmarks can overfit tuning corpora. The evaluation omits metrics such as stealth, time‑to‑compromise and detectability, and results may reflect dataset artefacts or stochastic tool behaviour rather than generalised reasoning.

Kicker: History shows that the commoditisation of offensive tooling accelerates both capability and misuse; the sensible path for teams is to adopt these tools for proactive defence while treating them as sensitive infrastructure that requires engineering controls, governance and continuous validation.

Additional analysis of the original ArXiv paper

📋 Original Paper Title and Abstract

xOffense: An AI-driven autonomous penetration testing framework with offensive knowledge-enhanced LLMs and multi agent systems

Authors: Phung Duc Luong, Le Tran Gia Bao, Nguyen Vu Khai Tam, Dong Huu Nguyen Khoa, Nguyen Huu Quyen, Van-Hau Pham, and Phan The Duy

This work introduces xOffense, an AI-driven, multi-agent penetration testing framework that shifts the process from labor-intensive, expert-driven manual efforts to fully automated, machine-executable workflows capable of scaling seamlessly with computational infrastructure. At its core, xOffense leverages a fine-tuned, mid-scale open-source LLM (Qwen3-32B) to drive reasoning and decision-making in penetration testing. The framework assigns specialized agents to reconnaissance, vulnerability scanning, and exploitation, with an orchestration layer ensuring seamless coordination across phases. Fine-tuning on Chain-of-Thought penetration testing data further enables the model to generate precise tool commands and perform consistent multi-step reasoning. We evaluate xOffense on two rigorous benchmarks: AutoPenBench and AI-Pentest-Benchmark. The results demonstrate that xOffense consistently outperforms contemporary methods, achieving a sub-task completion rate of 79.17%, decisively surpassing leading systems such as VulnBot and PentestGPT. These findings highlight the potential of domain-adapted mid-scale LLMs, when embedded within structured multi-agent orchestration, to deliver superior, cost-efficient, and reproducible solutions for autonomous penetration testing.

🔍 ShortSpan Analysis of the Paper

Problem

Penetration testing is essential for evaluating the security of information systems, but manual testing by small teams cannot keep pace with a growing attack surface. Traditional automated methods using ML, DL or RL are constrained by simplified action spaces, high costs, and weak multi stage reasoning across reconnaissance, vulnerability analysis and exploitation. Recent LLM based systems such as PentestGPT and VulnBot face challenges in cost, scalability and adaptability to structured workflows. This work investigates xOffense, an AI driven, multi agent framework that uses a domain adapted mid scale LLM to enable autonomous, scalable, machine executable penetration testing workflows with coordinated agents across phases.

Approach

xOffense deploys five core components—Task Orchestrator, Knowledge Repository, Command Synthesizer, Action Executor and Information Aggregator—within a three phase pipeline of reconnaissance, vulnerability scanning and exploitation. It uses a fine tuned Qwen3 32B model trained on Chain of Thought penetration testing data to drive reasoning and tool command generation, with a grey box prompting scheme that injects partial environment cues to preserve phase continuity. A Task Coordination Graph coordinates tasks and dependencies, while a Retrieval Augmented Generation based Knowledge Repository provides context from prior successes. The system uses LoRA fine tuning, DeepSpeed ZeRO 3 and FlashAttention to enable efficient training and inference on modest hardware. The evaluation compares xOffense to baselines on AutoPenBench and AI Pentest Benchmark, measuring task and sub task completion, under various autonomy settings and with and without retrieval augmented generation.

Key Findings

xOffense achieves a overall task completion rate of 72.72 percent on AutoPenBench, with a sub task completion rate of 79.17 percent in single run evaluations, outperforming GPT 4 o, Llama 3 70B, Llama 3 405B and PentestGPT in these benchmarks.
Fine tuning Qwen3 32B with domain specific CoT data yields substantial gains over the base model; in AutoPenBench, the fine tuned model reaches 72.72 percent overall task completion versus 30.30 percent for the base variant and 21.21 percent for GPT 4 o and 30.30 percent for Llama 3 405B.
In Access Control, the fine tuned model reaches 100 percent success; in Network Security 83.33 percent; in Real world tasks 54.54 percent, all outperforming baselines.
Sub task performance across a single experiment reaches 79.17 percent for the fine tuned model, beating a larger model such as Llama 3 405B which achieved 69.05 percent, illustrating the value of domain adaptation and orchestration for multi step tasks.
With retrieval augmented generation, real world targets show substantial gains; Victim1 and WestWild reach perfect completion in some runs, and other targets show meaningful improvements, while the base model with RAG lags behind the fine tuned variant.
No RAG baselines show competitive performance in some real world targets, but RAG consistently enhances results for the fine tuned model, indicating the importance of external knowledge for long chain reasoning in pentesting.
The authors demonstrate that a mid scale open source model, when properly tuned and orchestrated, can match or exceed larger proprietary models for domain specific tasks, offering a cost efficient and reproducible approach to autonomous offensive security testing.

Limitations

Limitations include potential dataset bias from the fine tuning corpus and the risk that improvements reflect dataset artefacts rather than genuine reasoning ability. The evaluation relies on benchmark suites that may not capture production scale heterogeneity, adversarial defences or zero day exploitation, and metrics focus on task success rather than stealth, resource efficiency, time to compromise, or detectability. Results may be affected by stochastic variation in LLM inference and tool outputs, and reproducibility across environments may vary. The study also evaluates no RAG baselines on a subset of targets, which may not generalise to all real world settings.

Why It Matters

The work demonstrates an end to end autonomous AI driven offensive security workflow that combines multi agent coordination with a domain adapted mid scale LLM to produce precise tool commands and robust reasoning across phases. It highlights dual use risks as automation of exploitation could enable rapid, scalable attacks if misused or poorly secured, and it stresses the need for safeguards such as validation gates, sandboxing, command verification and robust monitoring. The results imply practical implications for proactive defence, enabling scalable, repeatable assessments while underlining societal concerns around dual use and the potential for misuse in offensive security contexts. The authors discuss mitigation patterns such as structured orchestration, phase grounding, and benchmarking as design templates to reduce miscommands, hallucinations and unsafe actions while increasing reproducibility and safety in autonomous pentesting systems.

Attribution Original paper on arXiv