Proof-of-Vulnerability Test Execution

• Proof-of-vulnerability testing is a process that executes exploit artifacts to conclusively demonstrate software vulnerabilities through differential and formal methods.
• Automated pipelines integrate static/dynamic analysis with LLM-driven synthesis to consistently generate and verify PoV tests across various programming domains.
• Empirical evaluations highlight the importance of reproducibility, cost efficiency, and rigorous benchmarking in advancing vulnerability triage and mitigation.

A proof-of-vulnerability (PoV) test is an executable artifact or procedure whose successful execution demonstrates the presence of a specific vulnerability in a software target. PoV tests serve as definitive evidence for exploitability, ensuring that reported or hypothesized bugs are actionable, reproducible, and not false positives. The execution of PoV tests, both as ground-truth validators and as benchmarks for repair or mitigation measures, has become central to modern vulnerability triage, patch validation, and security evaluation. This article synthesizes the key methodologies, formal frameworks, automation pipelines, and empirical findings that define PoV test execution across vulnerability domains.

1. Formal Foundations and Verification Criteria

The essential goal of PoV test execution is to distinguish between (i) code that is actually vulnerable and (ii) code that is either fixed, patched, or for which candidate mitigations have been proposed. Formal definitions unify diverse approaches:

• Validation Logic: For a given vulnerability instance $v$, with associated PoV test $t_v$, a valid test must (a) fail (i.e., detect exploit effect) on the vulnerable version $V_{\mathrm{vul}}$ and (b) pass (i.e., be neutralized) on an official fixed version $V_{\mathrm{fix}}$ (Garg et al., 28 Nov 2025).
• Differential Execution: Automated frameworks such as VulnRepairEval and SmartPoC encode this as a binary predicate: $s = 1$ if the PoV executes on $V_{\mathrm{vul}}$ ($r^0=1$) and fails on the candidate-patched code ($r^P=0$) (Wang et al., 3 Sep 2025, Garg et al., 28 Nov 2025, Chen et al., 17 Nov 2025).
• Reachability and Triggering Constraints: Tools like PBFuzz formalize PoV-validating inputs $x$ as those satisfying both reachability $C_r(x)$ to a bug site and triggering constraints $C_t(x)$ for the vulnerability predicate (Zeng et al., 4 Dec 2025).

PoV test execution thus enforces a rigorous standard: only concrete, reproducible exploits—distinct from static oracles or theoretical traces—are accepted as proof of bug existence or elimination.

2. Automated PoV Generation Pipelines

Contemporary pipelines operationalize PoV execution through multi-stage workflows integrating static/dynamic analysis, LLM-driven synthesis, and validation oracles.

Typical Pipeline Components

Framework	Target Domain	Key Stages
SmartPoC	Smart contracts	Bug-context extraction, GRE-Engine Gen/Repair, Differential Oracle
PoCGen	npm packages	Info extraction, LLM exploit synth, dynamic runner, type oracles
FaultLine	Multi-language	Data/control flow reasoning, input constraint synth, feedback loop
PBFuzz	C/C++ (Magma)	PLAN/IMPLEMENT/EXECUTE/REFLECT LLM fusion with PBT, dev. detection

• Context Extraction: Funneling only the vulnerable-relevant program regions and NLP findings into code generators or formal analyzers; this may involve dynamic taint tracking (PoCGen (Simsek et al., 5 Jun 2025)), call-graph slicing (SmartPoC (Chen et al., 17 Nov 2025)), or symbolic simulation (PBFuzz (Zeng et al., 4 Dec 2025)).
• Exploit/PoV Synthesis: Use of LLMs (e.g., DeepSeek-R1, GPT-5-mini, GPT-4o-mini) or formal tools (ProVerif (Künnemann et al., 2 Oct 2024)) to synthesize exploit code or parameterize remote API invocation strategies.
• Repair Loops & Instrumentation: Feedback-driven loops (GRE-Engine, FaultLine) in which failures at compile/run/test time are used to refine inputs, suggesting edits, and enforcing structural correctness in test harnesses.
• Validation & Oracle: Embedding “differential verification” directly as assertions (“assert $s_\text{post} \neq s_\text{pre}$”), log checks, or runtime side-effects to serve as the proof signal; success criteria are typically predicate-over-state differences, file modifications, shell outputs, or assertion failures.

3. Execution Environments and Isolation

Correct PoV execution requires controlled and reproducible environments, mandating isolation, deterministic state, and version specificity:

• Containerization: Most recent frameworks (VulnRepairEval, FaultLine, SmartPoC) invoke test execution within Docker or equivalent containers, with explicit image variants for vulnerable and patched code designed to differ only by the tested diff (Wang et al., 3 Sep 2025, Nitin et al., 21 Jul 2025, Chen et al., 17 Nov 2025).
• Environment Bootstrapping: Automated parsing and application of dependency manifests (pyproject.toml, requirements.txt, package.json, Foundry remappings, etc.) (Wang et al., 3 Sep 2025, Chen et al., 17 Nov 2025).
• Instrumentation: Injection of log markers or coverage signals (instrumented function prints, dynamic hook functions) is used to confirm sink/trigger reachability and prevent superficial “unit test” pass/fail from misrepresenting exploitability (Nitin et al., 21 Jul 2025).

These mechanisms eliminate environment drift and cross-contamination, ensuring the only variable under test is the patch or code-under-evaluation.

4. Empirical Evaluation and Effectiveness Metrics

Automated PoV execution pipelines are benchmarked via standardized datasets (e.g., SmartBugs-Vul, Magma, CWE-Bench, Vul4J, SecBench.js) (Chen et al., 17 Nov 2025, Simsek et al., 5 Jun 2025, Garg et al., 28 Nov 2025, Nitin et al., 21 Jul 2025, Zeng et al., 4 Dec 2025). Common metrics include:

• PoV Test Success Rate: The percentage of vulnerabilities for which a validated PoV is generated and confirmed (e.g., SmartPoC: 85.61% on SmartBugs-Vul; PoCGen: 77% on SecBench.js; FaultLine: 16% on CWE-Bench-Java) (Chen et al., 17 Nov 2025, Simsek et al., 5 Jun 2025, Nitin et al., 21 Jul 2025).
• Time-to-Exposure / Efficiency: Median time to first expose a bug via PoV test; PBFuzz demonstrated a median 339s compared to 8680s for AFL++ (25.6x faster) (Zeng et al., 4 Dec 2025).
• Cost per PoV: LLM-powered approaches report API cost per vulnerability ($0.02–$1.83), with lower costs in practice for non-interactive generation workflows (Chen et al., 17 Nov 2025, Simsek et al., 5 Jun 2025, Zeng et al., 4 Dec 2025).
• Validation Precision (PPV/NPV): Precision of validated PoVs under manual or secondary review (e.g., SmartPoC PPV 94.29%, NPV 85.71%) (Chen et al., 17 Nov 2025).
• Reproducibility: The proportion of PoC reports yielding successful exploit reproduction by practitioners; recent studies show PoC completeness is strongly correlated with reproduction success (Pearson $r=0.62$, $p<0.001$) (Dang et al., 21 Oct 2025).

Empirical evidence indicates substantial, though by no means universal, success—limitations arise due to environment complexity, incomplete reports, and irreproducible codebases.

5. Challenges, Failure Modes, and Lessons Learned

Multiple systemic and technical challenges impede universal PoV execution:

• LLM Limitations: About 15% or more of cases fail due to non-convergent code synthesis, environment mismatches, or hallucinated test logic (SmartPoC, PoCGen). Prompt finetuning and retrieval-augmented generation are under investigation (Chen et al., 17 Nov 2025, Simsek et al., 5 Jun 2025).
• Oracle Fidelity: State-query or event-based oracles may not surface all storage or side-channel invariants (e.g., storage-only or proxy contracts in Ethereum) (Chen et al., 17 Nov 2025).
• Report Incompleteness and Reproducibility Gaps: Studies show incident CVEs with higher PoC completeness (all key fields) are strongly more reproducible; direct success rate is 28% but rises to 79% after LLM-supported augmentation. Triggers lacking required environment or “trigger step” information persist as bottlenecks (Dang et al., 21 Oct 2025).
• Syntactic/Semantic Patch Errors: Fuzzy or malformed LLM-generated patches can break test harnesses rather than neutralize exploits, leading to false negatives (Wang et al., 3 Sep 2025, Garg et al., 28 Nov 2025).
• Input Generation Complexity: Naive mutation or generation strategies often fail; methods utilizing symbolic reasoning, property-based testing (PBFuzz), or step-by-step constraint solving yield more consistent PoV input synthesis, especially under boundary-value or compositional input spaces (Zeng et al., 4 Dec 2025).

Recommendations include both procedural (e.g., enforcing PoC completeness templates) and architectural (e.g., structured PLAN/EXECUTE/REFLECT loops, property-based generators, persistent state to avoid LLM “drift”) measures.

6. Methodological Innovations and Future Directions

Key methodological advances and ongoing research trajectories include:

• Integrated Hierarchical Reasoning: Agentic workflows (SMARTPoC, FaultLine, PBFuzz) combine static analysis, LLM-based reasoning, property-based testing, and feedback-driven loops to concretely operationalize and validate test generation (Chen et al., 17 Nov 2025, Zeng et al., 4 Dec 2025, Nitin et al., 21 Jul 2025).
• Differential Verification and Instrumentation: Widespread adoption of differential, state-aware oracles to guard against false positives and coincidental test passing (i.e., asserting on observable controlled state changes) (Chen et al., 17 Nov 2025, Simsek et al., 5 Jun 2025).
• Structured Persistent Memory for Agents: PBFuzz and similar frameworks show that maintaining explicit, phase-locked workflow state (in Markdown/JSON) avoids redundant exploration and “hypothesis drift,” critical for scaling agentic PoV generators (Zeng et al., 4 Dec 2025).
• Formal Methods–Assisted PoV Synthesis: ProVerif + template annotation demonstrates “adaptive” PoV generator schemes for protocol and API targets, auto-translating logic-based attack traces into concrete exploits in diverse programming languages (Künnemann et al., 2 Oct 2024).
• Benchmarking and Completeness Metrics: Empirical studies emphasize the necessity of structured reporting (nine key PoC fields) to maximize real-world PoV usability and reproduction (Dang et al., 21 Oct 2025).

Anticipated future work includes hybridization of symbolic and learning-based components, automated completeness augmentation, cross-language/test oracle engineering, and multi-agent or ensemble test synthesis approaches.

7. Tabular Comparison of Representative Frameworks

System	Domain	Automation Core	Validation Oracle/Method	Empirical Success
SmartPoC (Chen et al., 17 Nov 2025)	Solidity/EVM	LLM + GRE loop	Action-state diff (public ABI queries)	85.61%–86.45%
PoCGen (Simsek et al., 5 Jun 2025)	npm (JS)	LLM + CodeQL + dyn hook	Taint-path coverage + sink-specific check	77%–39%
PBFuzz (Zeng et al., 4 Dec 2025)	Native/C/C++	LLM agent + PBT	Reach/trigger oracle, semantic GDB expr	57/278 (unique 17)
FaultLine (Nitin et al., 21 Jul 2025)	Java/C/C++	LLM agent (3-level)	Instrumentation/coverage + assert	16/100
VulnRepairEval (Wang et al., 3 Sep 2025)	Python (CVE)	Container diff-exec.	PoC run against pre/post-patch	21.7% best

Framework selection is contingent on language, available static/dynamic instrumentation, exploit/deployment affordances, and required evidence quality.

---

Proof-of-Vulnerability test execution, underpinned by fully automated synthesis, rigorous differential verification, and composable feedback mechanisms, is now central to high-assurance security engineering and empirical vulnerability research. The state-of-the-art is characterized by integrated reasoning pipelines, strong isolation semantics, and formal success guarantees, yet remains challenged by environment heterogeneity, incomplete reporting, and the intrinsic complexity of exploit path discovery and generation. Continued innovation at the intersections of static analysis, learning-based synthesis, and formal model extraction is expected to drive further gains in reliability, coverage, and efficiency.