Automated Web Vulnerability Reproduction

• The paper introduces automated web vulnerability reproduction by converting fragmented reports into actionable PoC exploits through structured parsing and environment reconstruction.
• Infrastructure-centric methods and LLM-driven multi-agent pipelines enable deterministic, auditable workflows with reported reproduction success rates as high as 82.9% in benchmarks.
• Scalable methodologies significantly reduce manual pentesting efforts by orchestrating environment setup, exploit generation, and iterative refinement for continuous security validation.

Automated web vulnerability reproduction is the process of transforming web vulnerability reports—often in informal, incomplete, or high-level formats—into actionable, executable Proof-of-Concept (PoC) exploits and validating them in reproducible environments. This discipline aims to bridge the gap between detection, analysis, and practical exploitability assessment by systematizing the workflow from vulnerability disclosure to execution and verification, without human intervention. Methodologies in this domain range from infrastructure-centric approaches leveraging containerization and declarative environments, to multi-agent frameworks powered by LLMs and dynamic feedback loops. Recent advances have established a foundation for scalable, auditable, and adaptive workflows that address the growing need for continuous security validation in large-scale and complex web ecosystems.

1. Problem Statement and Motivation

Automated reproduction of web vulnerabilities is essential for software maintainers, penetration testers, and security researchers. Web applications exhibit heterogeneous and complex environments (e.g., varying OS, web server, database, and middleware stacks), rendering manual setup and exploit validation labor-intensive and error-prone (Dashevskyi et al., 2017). This problem is compounded by imprecise or incomplete public vulnerability disclosures: public CVE/NVD entries often provide only fragmented HTTP examples, payload snippets, or code fragments, lacking essential technical details such as parameter names and encoding rules (Yang et al., 1 Apr 2026). Manual pentesting and ad hoc scripting do not scale with the volume and diversity of real-world vulnerabilities.

This motivates research into automating the end-to-end pipeline: (1) understanding and extracting actionable semantics from public vulnerability reports, (2) reconstructing vulnerable environments, (3) generating and executing exploit scripts, and (4) providing machine-checkable evidence of exploitation. Key challenges include adaptation to deployment-specific quirks, authentication barriers, multi-service orchestration, and the need for deterministic, reproducible workflows (Liu et al., 16 Oct 2025).

2. Systems and Framework Architectures

Architectural innovation in automated web vulnerability reproduction spans from infrastructure-level frameworks to multi-agent and LLM-driven orchestration:

Infrastructure-centric & Declarative Systems:

Frameworks such as TestREx (Dashevskyi et al., 2017) and NICE (Nguyen et al., 30 May 2026) encapsulate vulnerable application states, dependencies, and network configurations within container or VM images, often paired with scripted exploit scenarios and machine-checkable assertions. TestREx, for example, supports rapid deployment of diverse configurations via Docker, integrates exploit execution engines using Selenium and Python, and ensures repeatability via CLEAN and REUSE modes. NICE formalizes recipes as NixOS expressions, builds pinned environments, and validates exploitation via Python harnesses and assertion blocks.

LLM-Driven and Multi-Agent Pipelines:

Multi-agent architectures (e.g., AutoEG (Yang et al., 1 Apr 2026), AXE (Sajadi et al., 15 Feb 2026), Cve2PoC (Liu et al., 5 Feb 2026), CVE-Genie (Ullah et al., 1 Sep 2025)) leverage LLMs orchestrated as specialized agents (e.g., planners, executors, critics). A typical design decouples high-level planning (attack vector selection, environment setup strategy) from tactical code refinement and PoC code synthesis, often incorporating dual-loop feedback (strategic vs. tactical) to route failures for effective remediation (Liu et al., 5 Feb 2026).

Vulnerability-Specific and Modular Agents:

AWE (Jaswal et al., 1 Mar 2026) demonstrates the utility of dedicated agents for specific vulnerability classes, integrating persistent memory and browser-backed verification, with an orchestration LLM maintaining exploitation state. This approach restricts the problem space for LLMs, making the overall process more efficient and deterministic, especially for injection vulnerabilities.

3. Methodologies and Algorithms

Approaches to automated reproduction typically comprise a pipeline with the following stages:

1. Parsing and Context Extraction:
   • Structured extraction of CVE fields (category, version, description, references) via prompt-guided LLMs, NLP, or pattern matching (Lotfi et al., 28 Sep 2025).
   • Context enrichment using Retrieval-Augmented Generation (RAG) to fill in technical or environmental gaps with external documentation or CWE database entries (Lotfi et al., 28 Sep 2025, Zhao et al., 11 Oct 2025).
   • Generation of declarative environment specifications or infrastructure-as-code recipes (Dockerfiles, NixOS modules) (Caturano et al., 2022, Nguyen et al., 30 May 2026).
2. Attack Plan Synthesis:
   • Strategic planners analyze vulnerability semantics, code locations, and attack vectors (using information such as CWE, code location pointers) to produce ordered plans for exploit attempts (Sajadi et al., 15 Feb 2026, Liu et al., 5 Feb 2026).
   • Scoring heuristics rank candidate endpoints and payload templates using metrics such as match to CWE, static proximity, and endpoint frequency (Sajadi et al., 15 Feb 2026).
3. Exploit Code Generation and Execution:
   • Templated and progressive exploit instantiation: trigger function abstraction, where extracted parameters are bound to structured HTTP requests (Yang et al., 1 Apr 2026).
   • Synthesis of PoC scripts in the appropriate language/environment, embedding validation markers for success (e.g., response content, DB changes, DOM mutations) (Zhao et al., 11 Oct 2025).
   • Reuse of generic infrastructure to support multiple vulnerability classes via agent-specific pipelines (e.g., XSS, SQLi, SSTI) and context-conscious payload mutation (Jaswal et al., 1 Mar 2026).
4. Dynamic Feedback and Iterative Refinement:
   • Execution traces are parsed and classified (e.g., via oracles, LLM judgment, or assertion blocks) to yield updates (Δ) that guide failure diagnosis (Sajadi et al., 15 Feb 2026, Liu et al., 5 Feb 2026).
   • Dual-loop control (strategic vs. tactical) enables the system to differentiate between code-level bugs and fundamental strategy errors, thereby escaping unproductive debugging loops (Liu et al., 5 Feb 2026).
   • Adaptive feedback integrates runtime signals (e.g., malformed request, filtering, authentication errors) to auto-tune exploit parameters or payloads (Yang et al., 1 Apr 2026).

4. Evaluation Metrics and Benchmarking

Quantitative frameworks systematically measure the fidelity and robustness of automated reproduction systems through metrics tailored to each pipeline stage:

• Attack Success Rate (ASR):

Proportion of exploitation tasks for which an exploit achieves the intended objective (e.g., sensitive info leak, RCE, webshell deployment) (Yang et al., 1 Apr 2026). AutoEG reports an average ASR of 82.41%, exceeding the best baseline (32.88%) (Yang et al., 1 Apr 2026).

• Reproduction Success Rate (RSR):

Fraction of test CVEs where a generated PoC triggers the vulnerability on the vulnerable version but fails on the patched version (Liu et al., 5 Feb 2026). Cve2PoC reported 82.9% on SecBench.js, 54.3% on PatchEval (Liu et al., 5 Feb 2026).

• Environment Setup Success Rate, Execution Success Rate, Trigger Success Rate:

Fine-grained metrics assessing the success of environment initialization, execution of PoC scripts, and confirmation of actual exploit triggering. End-to-end success rates in multi-agent agent evaluations remain below 23% for complex, service-based vulnerabilities using contemporary LLMs (Liu et al., 16 Oct 2025).

• Efficiency and Token Consumption:

Systems are evaluated for mean time-to-solve and LLM API usage. AWE achieves a 4× speedup and >98% token efficiency compared to unconstrained LLM agents (Jaswal et al., 1 Mar 2026); Cve2PoC consumes 10–16× fewer tokens than baselines (Liu et al., 5 Feb 2026).

• Iteration Count and Failure Attribution:

AutoEG reports an average of 2.24 refinement iterations per successful exploit, outperforming baselines (2.5–13+), with most failures attributable to LLM policy blocks, authentication/logic subtleties, or highly chained workflows (Yang et al., 1 Apr 2026).

5. Reproducibility, Validation, and Artifact Generation

A central requirement is not merely exploitation but machine-checkable, reproducible proof of exploitation:

• Declarative Recipes and Pinning:

NICE leverages immutable, version-pinned VM builds and Python test harnesses to ensure that environments, once specified, remain reproducible across time and hosts (Nguyen et al., 30 May 2026). The recipe structure is formalized, linking VMConfigs, test scripts, and assertion blocks.

• PoC Artifact Generation:

Multi-agent systems (e.g., AXE, CVE-Genie) compile PoC.md reports detailing affected files, payload scripts, oracles (e.g., SQL queries, curl+grep output), and complete setup-to-exploit instructions (Sajadi et al., 15 Feb 2026, Ullah et al., 1 Sep 2025).

• Machine-Checkable Assertions and CI Integration:

Validation is enforced by assertive primitives such as check-http-response-contains, log inspection, or DOM snapshot comparisons. These outputs are suitable for CI/CD regression pipelines and upstream vulnerability submission (Nguyen et al., 30 May 2026, Zhao et al., 11 Oct 2025).

• Replay and Auditing:

Advanced agents (e.g., AWE) log all HTTP requests, mutations, and browser events, supporting deterministic replay for auditability and coverage assessment (Jaswal et al., 1 Mar 2026).

6. Failure Modes, Adaptation, and Limitations

Empirical studies report several recurrent failure sources and adaptation strategies:

• Attack-Trigger Gap:

Even when exploit code executes, systems may fail to meet preconditions for actual vulnerability triggering (e.g., authentication tokens, environmental or stateful requirements). In OpenHands+Claude-4, up to 70% of PoC scripts executed, but only 21% triggered exploits, and incomplete authentication information degrades performance by over 33% (Liu et al., 16 Oct 2025).

• Strategy vs. Implementation Errors:

AXE attributes 76% of failures to planning (semantics, endpoint selection, unmet preconditions) rather than code-exploration or execution mistakes (Sajadi et al., 15 Feb 2026).

• Context Deficiency and Reasoning Gaps:

Automated LLM-based tools underperform when vulnerability context is missing (data/sanitization paths, navigation constraints) (Zhao et al., 11 Oct 2025). Function- or file-level context injection into LLM prompts increases success rates by 9–13% (function vs. file) and by 17–20% overall (Zhao et al., 11 Oct 2025).

• Infrastructural and Scalability Constraints:

Some pipelines only handle CLI-driven exploits and do not yet generalize to GUI or browser-based flaws (CVE-Genie) (Ullah et al., 1 Sep 2025). Automation failures may also arise from outdated Docker tags, absent component sources, or incompatibility in legacy web stacks (Caturano et al., 2022).

• Agent-Orchestration Overheads:

Multi-agent approaches may incur nontrivial per-task costs (e.g., $2.00 per CVE for CVE-Genie, due to LLM API usage) and require tuning to balance recall and precision in critic/validator subagents (Ullah et al., 1 Sep 2025).

7. Trends, Impact, and Future Directions

Recent progress demonstrates significant acceleration and scaling of vulnerability reproduction:

• Agent-Oriented Specialization:

Combining broad LLM capabilities with domain-constrained, memory-augmented agents (as in AWE) achieves superior efficiency and determinism, especially for injection-class flaws, relative to both legacy scanners and pure LLM end-to-end agents (Jaswal et al., 1 Mar 2026).

• Integration of Reasoning and Dynamic Context:

Adaptive prompting strategies—chain-of-thought decomposition, in-context learning, feedback validators—raise reproduction rates to 68–72% under optimal conditions (Zhao et al., 11 Oct 2025).

• Declarative Infrastructure and Long-Term Reproducibility:

Systems such as NICE and TestREx show that reproducibility across years requires brittle setup details (package versions, network settings) to be declared and controlled in code, rather than prose or ad hoc scripts (Dashevskyi et al., 2017, Nguyen et al., 30 May 2026).

• Educational and Dataset Utility:

Automated pipelines have been successfully integrated into hands-on laboratory training and used to create large benchmark datasets for security research, accelerating both patch validation and tool evaluation (Dashevskyi et al., 2017, Ullah et al., 1 Sep 2025, Yang et al., 1 Apr 2026).

• Open Challenges:

Full automation on complex, service-based web vulnerabilities remains elusive, with end-to-end success often bounded by environment setup, authentication, and reasoning bottlenecks. Ongoing work targets integration of browser automation (Playwright/Chromium), multimodal agents for image-based PoCs, curriculum learning for orchestration, and reinforcement learning in simulated environments (Liu et al., 16 Oct 2025).

Automated web vulnerability reproduction thus stands at the intersection of software infrastructure, automated reasoning, and empirical security evaluation. Its evolution reflects a shift towards auditable, verifiable, and updatable pipelines, informing both offensive research and defensive validation in high-velocity web development ecosystems.