AI agents crash kernels but cannot craft exploits
Reviewing Evaluating Coding Agents on Kernel Exploit Generation · Junyoung Jang, Gwanhyun Lee, Hwiwon Lee et al.
Researchers built a benchmark to test whether coding agents can turn kernel bugs into real exploit primitives across dozens of Linux and Windows CVEs. Agents often reach crashes but fail to shape kernel state into usable primitives, and handing them a reference exploit lifts success sharply.
Here is the plain question this research answers. AI coding agents are already finding real bugs in production software. But can they turn a bug into an actual exploit? Not a crash, not a bug report, but a working primitive that bends the kernel to the attacker's will.
That gap matters because those are very different things. Finding a bug tells you a door exists. Building a primitive means you can read kernel memory, control where execution jumps, or write to an address you choose. That last one, an arbitrary write, is the step that leads to privilege escalation.
To measure exactly this, the researchers built a dedicated benchmark that tests whether agents can generate working kernel exploits. It takes 45 tasks drawn from 40 real Linux and Windows CVEs. Each task runs in an isolated virtual machine with a deterministic verifier that watches for a specific kernel-state condition. A crash does not count. A confident paragraph claiming success does not count either. What counts is the actual primitive.
Agents get controlled tools to build, run and debug payloads through a structured interface. There is no web access and no loose shell. They work under a fixed tool-call budget, with Windows allowing more calls than Linux. The verifier also randomises its target each run, so an agent cannot simply replay a stale artefact it half-remembered from training.
Reaching the crash is the easy part
This is where it gets interesting. Working with no reference exploit, the strongest configuration solved most Linux primitive tasks. On Windows it managed only one of twenty. Across every configuration and task, agents succeeded in roughly one in seven attempts.
The pattern underneath those numbers is the real finding. Agents repeatedly reach triggers and cause kernel crashes. Then they fail to shape that crash into a verifier-accepted primitive. The researchers call it the trigger-to-primitive gap. Stronger models crash more and convert less, which is the tell. They get further and still cannot close the loop.
Hand the agent a reference exploit and the picture changes. The strongest setup then solved roughly two thirds of all tasks, and aggregate success roughly tripled. An existing exploit lowers the cost of reaching the bug. But the agent still has to adapt it to the per-run target. The reference helps; it does not do the work.
The limits are honest. The benchmark measures single primitives, not full compromise or chaining several primitives together. Public CVEs and their exploits may sit in training data, which the randomisation blunts but cannot erase. Each number is one bounded session, not a reliability estimate.
In my reading, the striking result is that the better a model gets at reaching a bug, the more it stumbles at the final conversion step. The work shows agents can carry an attack a long way, but it does not show them weaponising crashes unaided. It also cannot tell us how reliable any single success would be, because each cell is a one-shot session rather than a repeated trial.
So where does that leave the risk? Agents can already assist with reachability and staged development, and a human-written exploit makes them markedly more effective. The autonomy to convert a crash into a weapon, on their own, is not there yet.
Further analysis
Problem
This paper asks whether current AI coding agents can not only find kernel bugs but reliably construct exploit primitives against real operating-system kernels. That capability matters because a bug report or a crash does not prove an agent can shape kernel state into verifier-observable primitives such as kernel address leaks, instruction-pointer control, heap reads or writes, or precise arbitrary writes that enable privilege escalation.
Approach
The authors introduce KEX-bench, a reproducible benchmark of 45 task instances drawn from 40 real Linux and Windows CVEs. Each task pairs an isolated virtual machine environment with a deterministic verifier that checks a primitive-specific kernel-state condition rather than accepting crashes or textual claims. Linux tasks cover four primitives (Leak, RIP control, HeapRead, HeapWrite) yielding 25 primitive-specific tasks; Windows tasks focus on arbitrary address write (AAW) yielding 20 tasks. Agents interact only through a Model Context Protocol that exposes controlled tools for building, running, debugging and analysing payloads; built-in web or shell access is disabled. A proxy enforces tool-call budgets (200 counted calls for Linux, 300 for Windows). The benchmark randomises per-run verifier targets to prevent direct replay of stale artifacts and validates Linux verifiers across positive and negative cases.
Key Findings
- KEX-bench reveals a marked trigger-to-primitive gap: agents frequently reach triggers or cause kernel crashes but fail to produce verifier-accepted exploit primitives.
- Performance numbers: across evaluated agent-model configurations agents solved 150 of 990 configuration-task outcomes (15.2%). Without a reference proof of concept (w/o PoC) the strongest configuration solved 14 of 25 Linux primitive tasks (56.0%) and 1 of 20 Windows AAW tasks (5.0%). With a reference PoC the strongest configuration solved 31 of 45 tasks (68.9%).
- Aggregate w/o PoC versus w/ PoC: 37 of 495 outcomes (7.5%) succeeded w/o PoC, rising to 113 of 495 (22.8%) w/ PoC, showing reference PoCs reduce reachability cost but do not eliminate the need for verifier-directed adaptation.
- Windows AAW tasks are especially difficult in the evaluated paired-VM closed-source configuration: only 3 of 220 w/o PoC AAW outcomes succeeded and many Windows CVEs remained unsolved even with PoC. Linux heap primitives are harder than direct control-flow primitives because they require preserving allocator state and affecting monitored victim objects.
- Failure modes are categorised as incorrect exploit logic, crash without primitive, and early termination. Stronger configurations show more crash-without-primitive failures, highlighting the conversion shortfall.
Limitations
The benchmark measures specific primitives rather than full compromise or primitive chaining. Platform, primitive, source access, VM topology, tooling and call ceilings are not independently varied, so platform effects cannot be isolated. Public CVEs and PoCs may be present in model training data; per-run randomisation mitigates but does not remove contamination. Each reported cell measures one bounded interactive session and is not a run-to-run reliability estimate.
Implications
Offensive-security implications are that current agents can assist in reachability and staged exploit development but often require a human-like iterative test-debug loop to convert crashes into usable primitives. Reference PoCs materially improve success rates, indicating that combining agent reasoning with existing exploit artefacts lowers attacker effort. The measured trigger-to-primitive gap identifies where automated systems would focus further development to produce reliable kernel exploit primitives.
The paper title, authors and abstract
Evaluating Coding Agents on Kernel Exploit Generation
Coding agents now find real vulnerabilities in production software. However, bug discovery results do not measure whether agents can construct exploit primitives. We introduce KEX-bench, a benchmark for evaluating coding agents on exploit primitive generation against real operating-system kernels. KEX-bench contains 45 task instances across 40 Linux and Windows CVEs, covering kernel address leak, instruction-pointer control, heap read, heap write, and arbitrary address write. Each task runs in an isolated virtual machine, exposes controlled tools, and uses a deterministic verifier to check primitive-specific success. We evaluate state-of-the-art coding agents paired with frontier and open-weight models under fixed tool-call budgets. Without a reference proof of concept (PoC), the strongest configuration solves 1 of 20 Windows tasks (5.0%) and 14 of 25 Linux tasks (56.0%). With a reference PoC, the strongest configuration solves 31 of 45 tasks (68.9%). This highlights the gap where agents reach kernel crashes but fail to shape kernel state into exploit primitives. We release KEX-bench for reproducible research on AI-assisted exploitation at https://kex-bench.github.io.
