AI Agents Reproduce CVEs, Exposing Governance Gaps
Attacks
Researchers built an automated pipeline that takes public CVE entries, reconstructs the vulnerable environment, and returns verifiable exploits. The system reproduces roughly half of tested CVEs at about $2.77 per case, showing that scale and affordability are no longer theoretical.
Policy and governance meet hard controls in this work. CVE is the Common Vulnerabilities and Exposures format used to describe bugs. Verifiable exploit means the pipeline produces a repeatable, demonstrable proof of concept. Those are powerful tools for defenders — they enable consistent benchmarking of fuzzers, automated patch testing, and more realistic red teaming.
But there is a flip side. Automating exploit generation increases dual-use risk. Open datasets and reproducible environments help researchers, and also lower the bar for attackers. Governance therefore cannot be only aspirational statements or checkbox compliance. Practical controls matter: strict sandboxing, least-privilege execution, provenance tracking for artifacts, human-in-the-loop verification, and access controls that tie legal and operational accountability to usage.
There are trade-offs. Tight controls slow research and raise costs. Open releases accelerate community progress but widen abuse surfaces. Automated agents scale reproducibility but can amplify bad data and brittle reconstructions. Organizations that focus on theater compliance will miss the point: governance should enable safe utility, not just publicity.
What to do: This quarter - inventory any AI-driven security tooling, enforce isolated sandboxes for exploit reproduction, require provenance metadata and human sign-off on any exploit artifacts, and tighten IAM around reproduction tools. Later - push for standard provenance schemas, support regulated data-sharing frameworks, fund third-party verification labs, and update disclosure policies to account for automated exploit generation.
Additional analysis of the original ArXiv paper
📋 Original Paper Title and Abstract
From CVE Entries to Verifiable Exploits: An Automated Multi-Agent Framework for Reproducing CVEs
🔍 ShortSpan Analysis of the Paper
Problem
High quality datasets of real world vulnerabilities and their verifiable exploits are crucial for software security research, yet such resources remain scarce because their creation requires intensive manual effort and deep security expertise. This paper presents CVE GENIE, an automated, large language model based multi agent framework designed to reproduce real world vulnerabilities, provided in Common Vulnerabilities and Exposures format, to enable creation of high quality vulnerability datasets. Given a CVE entry, CVE GENIE gathers the relevant resources, automatically reconstructs the vulnerable environment, and reproduces a verifiable exploit. The work notes tens of thousands of vulnerabilities discovered each year, with more than 49 000 tracked by the National Vulnerability Database in 2024 alone, and highlights that existing vulnerability detection tools struggle to keep up. There is a need for scalable, end to end CVE benchmarks to support tasks such as fuzzer evaluation, vulnerability patching and assessing AI security capabilities, while recognising potential AI security risks in automated multi agent tooling and the necessity for safeguards.
Approach
CVE GENIE implements a four stage end to end pipeline comprising Processor, Builder, Exploiter and CTF Verifier. It extracts CVE data, downloads and structures supporting resources, rebuilds the vulnerable environment, generates and verifies an exploit, and stores the final results. The system uses modular task decomposition with paired developer and critic agents operating in a ReAct style loop to ensure robustness to incomplete CVE information. A set of LLM driven tools supports code search, file access, environment setup, and command execution while keeping read only prior to exploitation. The approach emphasises end to end automation, rebuilds of the vulnerable environment, and a verification heavy workflow to guarantee verifiable exploits. The evaluation framework includes systematic LLM selection for each module, ablation studies, and large scale testing across CVEs published after the knowledge cutoffs of the models used, with open source release of code, logs and datasets.
Key Findings
- CVE GENIE reproduced about 51 per cent of 841 CVEs published in 2024 2025, delivering verifiable exploits across 267 projects, 141 CWEs and 22 programming languages at an average cost of 2 77 per CVE.
- In a large scale evaluation using 841 CVEs, the framework achieved 428 reproductions across diverse software projects and languages, demonstrating broad generalisation across CWEs and programming languages while managing costs and time.
- An ablation study shows that removing components such as the Knowledge Builder or Feedback loops markedly reduces reproducibility, while collapsing to a single monolithic agent eliminates successful reproductions, underscoring the necessity of CVE GENIEs modular, critic guided design.
- The Builder and Exploit modules show strong performance with selected LLM configurations, with the best developer critic pairs identified for each module, and the Verifier module achieving high quality one shot verifications using critic feedback.
- Cost and time are constrained by a per CVE budget and runtime limit, with most successful reproductions taking tens of minutes and costing around two pounds per CVE, while failed attempts incur higher relative costs; a budget cap of five pounds and a 45 minute limit were used.
- The framework demonstrates that PoC availability in advisories substantially improves reproducibility, while security advisory content and patch commits significantly aid vulnerability localisation and environmental setup.
- Across a diverse dataset, web oriented vulnerabilities tended to be more reproducible than memory heavy or UI dependent flaws, with higher success on interpreted languages and code based PoCs.
Limitations
Limitations include the current absence of user interface UI based vulnerability reproduction and multimodal PoCs, with current work focusing on command line interactions. Large projects challenge LLM context windows and environment provisioning, contributing to reproducibility failures. Ablation studies reveal dependencies on Knowledge Builder and feedback loops, with failure modes including mock environments, incomplete verification, and brittle setup processes. Incomplete CVE data reduce reproduction success, with missing patch commits and advisories causing declines in performance, while even complete descriptions can be insufficient for optimal results. The evaluation datasets, while large, reflect post cutoff CVEs and may not cover all vulnerability classes equally, and the reliance on generated PoCs carries dual use risk requiring safeguards and provenance tracking. Computational cost and time variability across runs remain factors in scaling to broader deployment.
Why It Matters
CVE GENIE offers a scalable, reproducible framework for CVE re creation that can underpin vulnerability detection, triage, patching and automated security evaluation of AI systems. By providing verified exploits and end to end reproducible environments, it enables robust benchmarking of fuzzers, patch validators and AI driven security tools. The work also highlights potential attack surfaces in automated multi agent systems, including prompt chaining, data provenance and environment reconstruction, and argues for strong safeguards such as secure sandboxing, strict access controls, verification and triage of generated exploits, and ethical guidelines for dataset creation. The open source release of code, logs and reproduced CVEs supports ongoing community research and the development of safer, more capable security tooling.