RCR shows LLMs assist Active Directory pentests
A replication report tests whether large language models can autonomously pentest Microsoft Active Directory in a controlled lab. The prototype often finds actionable leads and near-misses but struggles to close the loop. Heavy setup aside, the release gives teams a reproducible way to study AI-assisted internal attacks.
Another week, another claim that large language models can hack your network. This one at least brings tooling and traces you can check. The Replicated Computational Results report for “Can LLMs Hack Enterprise Networks?” packages a full lab and replayable logs to test whether a Large Language Model (LLM) can drive an assumed‑breach against Microsoft Active Directory.
The team built a Python prototype called cochise that runs from a Kali Linux attacker virtual machine against A Game of Active Directory, a public five‑host Windows lab. Everything sits under VMware on commodity server kit, but do not expect to run this on a laptop. The authors cite a practical floor of about 48 GB of RAM and roughly 190 GB of disk for the virtual machines, plus evaluation Microsoft licences, and you rebuild the environment after destructive runs.
The evaluation is methodical. It exercises several model setups, including OpenAI GPT‑4o, DeepSeek‑V3, Google Gemini‑2.5‑Flash, a planner and executor pairing using OpenAI o1 with GPT‑4o, and Alibaba Qwen3. Sessions run up to two hours with temperature set to zero when allowed. The run continues until saturation, defined as two consecutive samples that uncover no new leads or compromised accounts. Every prompt and command lands in timestamped JSON for later replay, and the authors ship Docker images, code, example logs and analysis scripts on public archives.
What actually worked
The headline is not that models waltz through an enterprise. The prototype can chain together non‑trivial Active Directory actions and often surfaces “actionable leads”, but many attempts stall a whisker from success. Small slips in targeting or procedure regularly turn what should be a straightforward step into a dead end. The study classifies a lot of these as “Almost‑There”. The primary metric is hard edged, the number of fully compromised user accounts, and that is where the brittleness shows.
This is both encouraging and sobering. Encouraging because the logs show real, automatable behaviour security teams can analyse without live network access. Sobering because the distance between almost there and domain compromise is not wide in many environments. If models can reliably reduce toil on enumeration, path discovery and basic execution, an operator, benign or malicious, can supply the missing judgement.
There is useful engineering here: a repeatable workflow, consistent caps on time, and a saturation rule that avoids cherry‑picking heroic single runs. The authors note that six samples per configuration were enough to hit saturation in their experiments, which should help future comparisons.
Now the caveats. This is one lab, GOAD, not your estate. It is invasive and must be rebuilt after runs. Some dependencies are finicky and tied to specific versions. Qualitative judgements lean on expert reviewers. The results do not generalise to every Active Directory topology, politics or patch level. Treat it as a wind‑tunnel test, not a street race.
Does it matter? Yes, in a measured way. The package shows that modern LLMs can materially assist internal attack automation under constraints, and it gives defenders a way to study that assistance with real traces. It also reminds vendors and security teams that policy, monitoring and safe‑use controls need to assume AI‑assisted attack workflows. The open question is reliability. Until these systems stop tripping on small variations, they are accelerants rather than autonomous intruders. That is still worth planning for.
Additional analysis of the original ArXiv paper
