New to ShortSpan? We distil the AI-security research that matters into practitioner takeaways — edited by Ben Williams (NCC Group).Get the weekly email
// Analysis
LLMs Automate Penetration Tasks, Exposing Infra Weaknesses
Published: Fri, 01 Aug 2025 • By Lydia Stratus
Pentesting
New research shows a modern LLM can autonomously solve most beginner capture-the-flag tasks, finding files, decoding data, and issuing network commands with human-speed accuracy. That success lowers the skills barrier for attackers and exposes specific infrastructure gaps. Operators must apply practical hardening to endpoints, GPUs, vector stores, secrets and data paths now.
This experiment connected an LLM to a one-command SSH CTF and the model solved roughly 80 percent of compatible levels. The headline is simple and alarming: language models can automate routine reconnaissance and single-step exploits fast, which means SREs and security teams face a new class of low-effort adversary.
Where this hurts you in the real world
Diagram-in-words: user or attacker -> model endpoint -> GPU host -> vector DB -> backend storage and secrets. Each hop is a risk channel. If the endpoint accepts raw commands or returns decoded snippets, an attacker automated by an LLM scales their reach.
Immediate checklist for Ops
Lock model endpoints with mTLS and short-lived tokens
Isolate vector stores and require query sanitization
Never allow secrets into model context or logs
Apply rate limits and behavioral anomaly detection
Quick run-book mitigations
Block interactive single-command shells from model clients and require multi-step auth flows to introduce statefulness
Rotate credentials and require ephemeral instance credentials for GPU jobs
Enable audit logs on vector DB queries and alert on pattern spikes
Instrument model responses for sensitive data exfil patterns and redact before returning
Deploy a canary CTF endpoint to detect automated probing
Why this matters: attackers now need less expertise to automate simple attacks, but defenders can apply low-fuss controls to raise the bar. Prioritize endpoint hardening, secret hygiene, and vector access policies. If you only do one thing today, stop feeding secrets into prompts and add short-lived auth on every model connection.
This study evaluates the ability of GPT-4o to autonomously solvebeginner-level offensive security tasks by connecting the model to OverTheWire'sBandit capture-the-flag game. Of the 25 levels that were technically compatiblewith a single-command SSH framework, GPT-4o solved 18 unaided and another twoafter minimal prompt hints for an overall 80% success rate. The model excelledat single-step challenges that involved Linux filesystem navigation, dataextraction or decoding, and straightforward networking. The approach oftenproduced the correct command in one shot and at a human-surpassing speed.Failures involved multi-command scenarios that required persistent workingdirectories, complex network reconnaissance, daemon creation, or interactionwith non-standard shells. These limitations highlight current architecturaldeficiencies rather than a lack of general exploit knowledge. The resultsdemonstrate that large language models (LLMs) can automate a substantial portionof novice penetration-testing workflow, potentially lowering the expertisebarrier for attackers and offering productivity gains for defenders who use LLMsas rapid reconnaissance aides. Further, the unsolved tasks reveal specific areaswhere secure-by-design environments might frustrate simple LLM-driven attacks,informing future hardening strategies. Beyond offensive cybersecurityapplications, results suggest the potential to integrate LLMs into cybersecurityeducation as practice aids.
🔍ShortSpan Analysis of the Paper
Problem
This paper evaluates whether a modern large language model, GPT-4o, can autonomously solve beginner-level offensive security tasks and what that implies for attackers, defenders and cybersecurity education.
Approach
GPT-4o was connected to the OverTheWire Bandit capture-the-flag game via a Python 3 script using the Paramiko SSH library. The setup forced a one-command-per-shell interaction: the model received the level instructions and was prompted to return only a single Linux command, the command was executed on the remote server, and the output returned to the model. Of 33 Bandit levels, 25 were compatible with this single-command framework and were attempted. Outcomes were labelled solved, solved with additional assistance or unsolved. Token usage and monetary input cost were recorded.
Key Findings
High autonomous success: GPT-4o solved 18 levels unaided, solved 2 more after minimal prompt hints, an overall 80% success rate across the 25 compatible levels.
Strengths: excelled at single-step tasks such as Linux filesystem navigation, data extraction or decoding, and straightforward networking; often produced correct commands in one shot and faster than a human.
Failures: struggled with multi-command workflows requiring persistent working directories, complex network reconnaissance (interpreting nmap output), creating daemons (netcat), non-standard shells that alter commands, and creating persistent files/scripts.
Cost: successful solutions consumed 4,848 input tokens (≈0.002424 USD); token averages rose with command complexity.
Limitations
Key constraints include the one-command SSH design that prevented persistent state across commands, testing limited to the Bandit beginner CTF, and experimentation with a single model (GPT-4o). Generalisability to advanced CTFs or real-world targets: not reported.
Why It Matters
Results show LLMs can automate a substantial portion of novice penetration-testing tasks, potentially lowering the expertise barrier for attackers while offering rapid reconnaissance and productivity gains for defenders and educators. The documented failure modes indicate specific hardening strategies (e.g., requiring multi-step interactions or nonstandard environments) that could frustrate simple LLM-driven attacks.
- It shows autonomous LLM agents can perform sophisticated web-hacking tasks (like blind SQL injection and schema extraction) without being told where...
It shows a semi-autonomous system where several LLMs coordinate to plan attack strategies, generate commands, and interpret results in a lab environme...
- Why it’s relevant: shows an automated, scalable method to craft jailbreak prompts that bypass LLM safety, with high success across multiple models. ...
- Autonomy: It shows how an AI agent can run black-box vulnerability tests end-to-end, potentially speeding up assessments and increasing test frequen...
- Very briefly: the study shows a GPT-4 based AI agent can autonomously exploit real one-day vulnerabilities when given CVE descriptions, hitting abou...
- Why it’s relevant: It exposes practical safety weaknesses in leading LLMs through jailbreaks, including cross-model prompt injection, and shows that...
It shows that locally hosted open-weight LLMs, when paired with targeted prompting techniques, can autonomously perform Linux privilege escalation at ...
- It tests whether LLMs used for smart-grid operations can be jailbreaked to produce outputs that violate regulatory standards, highlighting real-worl...
- It shows GPT-4 Advanced Data Analysis can detect many kinds of bugs more accurately than traditional SAST tools, across 32 vulnerability types (abou...
Get the weekly digest
The few AI-security papers that matter, with the practitioner takeaway. No spam.
