CVE-Genie: Automated Vulnerability Reproduction

• CVE-Genie is a modular system that automatically extracts, characterizes, and validates CVE data using machine learning, NLP, and LLM-powered multi-agent orchestration.
• It employs structured pipelines for semantic annotation, exploit synthesis, and containerized environment setup, ensuring reproducible and verifiable vulnerability assessments.
• The system integrates retrieval-augmented generation and static analysis query synthesis to enhance contextual understanding and support robust, automated security research.

CVE-Genie is an automated, modular system for the end-to-end extraction, characterization, and reproducible validation of software vulnerabilities from Common Vulnerabilities and Exposures (CVE) records. Developed through successive waves of research, CVE-Genie spans multiple methodologies, including LLM-driven multi-agent orchestration, information extraction pipelines, machine learning-based characterization, and synthesis of static analysis queries. Its architecture enables rapid, rigorous, and minimally-supervised generation of both structured metadata and verifiable exploit artifacts, providing a foundation for reproducible security research, dataset creation, and vulnerability triage (Ullah et al., 1 Sep 2025, Lotfi et al., 28 Sep 2025, Sun et al., 2021, Gonzalez et al., 2019, Wang et al., 11 Nov 2025).

1. Automated Vulnerability Characterization and Description Extraction

CVE-Genie incorporates specialized components for structured extraction and semantic annotation of vulnerability reports. Early modules employ ML and NLP to transform unstructured CVE records or public exploit disclosures into rich, machine-actionable summaries.

Specifically, supervised classification algorithms have been trained to auto-annotate CVE records with labels from the NIST Vulnerability Description Ontology (VDO). A canonical pipeline involves lowercasing, URL stripping, whitespace tokenization, stop-word removal, and Porter stemming, followed by construction of a TF–IDF–weighted unigram term-document matrix. Six classification algorithms—including multinomial Naïve Bayes, decision trees (J48), support vector machines (SVM, via SMO), random forests, AdaBoost-SVM, and majority-vote ensembles—demonstrate reliable VDO-class assignment. In a stratified 10-fold cross-validation on 365 CVE descriptions mapped to 19 VDO classes, the linear SVM (C=1.0) achieved 72.9% accuracy and κ=0.71, with F1 ≥ 0.80 on multiple semantic classes, establishing the feasibility of automatic VDO annotation for vulnerability triage (Gonzalez et al., 2019).

For deep slot-based extraction, pipelines leveraging BERT-based named entity recognition (NER) and extractive question answering (QA) distill nine key vulnerability aspects—including vendor, product, version, component, type, attacker, root cause, impact, and attack vector—from sources such as ExploitDB. Experiments show NER F1=0.76, QA F1=0.82, and overall ROUGE-L=0.38 on synthesized CVE descriptions as compared to reference CVE summaries, outperforming neural summarization baselines (Sun et al., 2021).

2. Multi-Agent LLM-Orchestrated Exploit Reproduction

Modern instantiations of CVE-Genie use a modular, multi-agent LLM architecture for automated reproduction of known vulnerabilities. The core pipeline, as presented by Li et al. and Arlotfi et al., consists of sequential, specialized agents or modules that cooperatively process raw CVE data through environment reconstruction, exploit synthesis, and comprehensive verification (Ullah et al., 1 Sep 2025, Lotfi et al., 28 Sep 2025).

The standard workflow is as follows:

1. CVE and Resource Ingestion: An initial agent retrieves the CVE record, resolves relevant metadata (CWE tags, affected versions), and collects repositories, patches, advisory links, and available proof-of-concept (PoC) scripts.
2. Knowledge Base Construction: Structured knowledge is extracted for downstream planning, including version constraints, patch diff hunks, and exploit steps, aggregated into a YAML/JSON database.
3. Environment Planning and Setup: A prerequisite development agent infers package dependencies, environment variables, and service requirements by traversing the directory and documentation trees. The setup agent asserts the correct vulnerable environment, checked by a setup critic, which detects version failures and incomplete installs.
4. Exploit Synthesis: The exploit developer agent either replays known PoCs or, where absent, analyzes patch diffs and code usage to synthesize scripts that trigger the vulnerability. An exploit critic evaluates correctness, requesting targeted fixes if necessary.
5. Automated Verification: The verifier agent writes a staged script to check environment integrity, trigger the exploit, and validate its manifestation (flag printing or oracle-triggered behavior). An additional critic verifies alignment with root cause and absence of spoofable heuristics.
6. Feedback and Correction: Critic agents inject execution or log feedback into the LLM context to facilitate iterative refinement and retry, maximizing genuine exploit synthesis and minimizing false positives.
7. Artifact Storage and Reproducibility: Successful exploit-verifier pairs and environment snapshots are persisted for dataset generation or downstream use.

This modular structure, with isolated developer/critic roles and fallback correction, achieves high robustness and scalability. Ablation studies show the pipeline’s dependence on both knowledge extraction and feedback loops; removal of either sharply reduces success rates (Ullah et al., 1 Sep 2025).

3. Retrieval-Augmented Generation and Context Integration

To overcome limitations inherent in noisy or incomplete CVE descriptions, CVE-Genie employs retrieval-augmented generation (RAG) methodologies. A dedicated RAG module maintains a vector index of external knowledge—including CWE definitions, vendor advisories, and PoC blog posts. For each CVE, a cosine similarity-based retrieval selects the top-k most relevant documents, whose contents are concatenated with the original CVE record to construct an enriched LLM prompt (Lotfi et al., 28 Sep 2025).

Mathematically, for a given query embedding $q$ and document embeddings $\{d_i\}$: $$\text{similarity}(q, d_i) = \frac{\text{emb}(q) \cdot \text{emb}(d_i)}{||\text{emb}(q)|| \cdot ||\text{emb}(d_i)||}$$ Top-k $d_i$ are appended to the LLM input for subsequent exploit-code and environment generation stages. This data fusion ameliorates context gaps, increases coverage, and enhances the LLM’s ability to infer critical exploit and setup details absent from the original CVE description.

4. Containerized Environment Construction and Exploit Verification

CVE-Genie automates instantiation of vulnerable environments via programmatic, LLM-generated Dockerfiles and docker-compose manifests. The LLM determines the minimal number of containers (supporting both single and multi-container, e.g., client/server, architectures), pins base images and library versions, installs appropriate debug tooling, and configures non-root users when required by privilege escalation scenarios (Lotfi et al., 28 Sep 2025).

The exploit-code generation module synthesizes scripts (typically Bash), which:

• Create relevant source files (e.g., PoC in C, Python).
• Build and deploy container(s) with vulnerable code.
• Execute the exploit, with oracles to assert vulnerability presence through side-effect detection (stdout messages, AddressSanitizer errors, etc.).

A validation agent monitors execution logs; unmet test-oracle conditions trigger log feedback into the LLM for one or more rounds of correction, capped at a predefined retry budget.

Experimental validation reports reproducibility rates of 69.6% (71/102 CVEs) in one pipeline (Lotfi et al., 28 Sep 2025) and 51% (428/841 CVEs) in a broader multi-agent evaluation (Ullah et al., 1 Sep 2025). Success clustering is evident in certain vulnerability types: memory corruption exploits yield up to 80% success, while ambiguous patches or missing PoCs dominate failure cases.

5. Pipeline Evaluation, Limitations, and Datasets

Quantitative evaluation spans:

• Success Rate: Proportion of CVEs for which a reproducible, verifiable exploit is automatically synthesized (51%–69% depending on corpus and setup).
• Cost and Throughput: Average per-CVE cost of $2.00–$2.77, mean runtime per CVE ≈18 minutes, with rigorous time and retry caps.
• Failure Modes: 36% build failures, 24% timeouts/cost overruns, 40% missing/incomplete PoCs or ambiguous patches.

Ablation studies (e.g., knowledge builder removal, feedback-loop elimination, reduction to a single monolithic agent) demonstrate sharp drops in reproduction accuracy and surges in false positives, highlighting the necessity of modularity, structured agent roles, and critic-driven feedback (Ullah et al., 1 Sep 2025).

Applications include the generation of high-quality vulnerability datasets suitable for fuzzer benchmarking, evaluation of AI-driven security methods, patch verification, and penetration testing. CVE-Genie artifacts—comprised of complete exploit-verifier pairs and containerized environments—are open-sourced to foster reproducibility and further research (Lotfi et al., 28 Sep 2025).

6. LLM-Aided Static Analysis Query Synthesis

Expanding beyond traditional exploit verification, QLCoder leverages CVE-Genie’s LLM-based agentic approach for automated synthesis of semantically precise CodeQL queries from CVE metadata and buggy-versus-patched code states (Wang et al., 11 Nov 2025). The system embeds an LLM in an iterative synthesis loop, guided by a Model Context Protocol (MCP) interface which integrates a CodeQL language server (offering syntax and semantic completions) and a RAG database of CodeQL documentation, CWE definitions, and repository AST snippets.

A formal definition: $$\text{success}(Q; P_\mathrm{vuln}, P_\mathrm{fixed}) \equiv \mathrm{valid}(Q) \land \big(\exists \pi \in \llbracket Q \rrbracket(G_\mathrm{vuln}),\, \pi \cap \Delta V \ne \emptyset \big) \land \big(\forall \pi \in \llbracket Q \rrbracket(G_\mathrm{fixed}),\, \pi \cap \Delta V = \emptyset\big),$$ ensures that a synthesized path query $Q$ detects only true positives related to the vulnerability and not the fixed code.

Empirical results show the system synthesizes correct CodeQL queries for 53.4% of real-world Java CVEs, outperforming LLM and static tool baselines (Claude Code at 10%, IRIS 4.8%, CodeQL 7.3% on average F1). Supplementary ablations confirm the critical roles of the LSP and RAG modules in semantic grounding and precision (Wang et al., 11 Nov 2025).

7. Limitations, Future Directions, and Implications

CVE-Genie’s performance depends in part on the richness and consistency of CVE descriptions, the presence of PoCs, and the tractability of the build environment. UI-driven or multimodal exploits, requirements for external services, and ambiguous or missing ground-truth code bases currently limit achievable automation. The agentic paradigm—incorporating domain-specialized critics, feedback-driven repair, and information augmentation—demonstrates robust fallback and correction, but further improvements are plausible by integrating support for hardware-dependent CVEs, chained attack synthesis, multimodal input processing, and hybrid dynamic/static analysis (Ullah et al., 1 Sep 2025, Lotfi et al., 28 Sep 2025, Wang et al., 11 Nov 2025).

The integration of human-in-the-loop systems (e.g., UI-assisted validation, anomaly flagging) and continuous learning from corrected outcomes can further sharpen both the accuracy and scope of vulnerability reproduction pipelines (Sun et al., 2021, Gonzalez et al., 2019). Open-sourcing of generated artifacts and container environments promotes reproducible science, accelerates data-driven vulnerability research, and holds implications for quality control in the CVE disclosure process itself (Lotfi et al., 28 Sep 2025).

---

Relevant references:

(Gonzalez et al., 2019): Automated Characterization of Software Vulnerabilities (Sun et al., 2021): Generating Informative CVE Description From ExploitDB Posts by Extractive Summarization (Ullah et al., 1 Sep 2025): From CVE Entries to Verifiable Exploits: An Automated Multi-Agent Framework for Reproducing CVEs (Lotfi et al., 28 Sep 2025): Automated Vulnerability Validation and Verification: A LLM Approach (Wang et al., 11 Nov 2025): QLCoder: A Query Synthesizer For Static Analysis of Security Vulnerabilities