Agentic Vulnerability Validation

• Agentic Vulnerability Validation is a closed-loop process where agents iteratively propose, test, and refine artifacts to confirm security vulnerabilities.
• It applies dynamic oracles and structured evidence by using techniques such as PoC replay, patch synthesis, and runtime checks to boost reliability.
• The approach spans software bug verification, automated patch repair, exploit confirmation, and agent safety audits, emphasizing iterative refinement and evidence integration.

Agentic vulnerability validation is the use of autonomous or semi-autonomous agent loops to convert a security hypothesis into a validated conclusion under executable, observable, and policy-constrained criteria. In the literature represented here, the term spans at least four closely related settings: confirming whether a suspected software flaw is exploitable via PoCs, crashes, sanitizer evidence, or repository-level tests; verifying whether an automatically synthesized patch actually fixes the vulnerability without regressions; validating machine-generated detection artifacts such as web rules before deployment; and auditing deployed agentic systems themselves for structural weaknesses such as unsafe tool execution or memory poisoning (Sajadi et al., 15 Feb 2026, Cao et al., 1 Jul 2026, Garg et al., 2 Apr 2026, Hossain et al., 11 Jun 2026). Across these settings, the common pattern is a staged loop in which an agent proposes an artifact, invokes tools or runtime checks, interprets evidence, and either accepts, rejects, or refines the artifact under explicit gates rather than one-shot confidence alone.

1. Conceptual scope and problem setting

The motivating premise is that static prediction or one-shot generation is insufficient for high-stakes security tasks. RuleForge starts from the operational problem that newly disclosed CVEs far exceed the capacity for manual rule development and validation, so validation must prioritize sensitivity and specificity while preserving precision (Garg et al., 2 Apr 2026). KeaRepair frames validation as the decisive stage of automated vulnerability repair: compilation, PoC replay, and test-suite execution determine whether a synthesized patch is acceptable (Cao et al., 1 Jul 2026). AXE treats validation as exploit-focused confirmation of vulnerability reports from minimal metadata, specifically a CWE classification and a vulnerable code location, with success defined by concrete impact or evidence-rich failure summaries (Sajadi et al., 15 Feb 2026). A2 similarly distinguishes speculative findings from validated vulnerability findings, requiring a working PoC and an independent Task Validator that confirms the expected security-relevant effect (Wang et al., 29 Aug 2025).

A second line of work uses the same term for validating vulnerabilities in agentic systems themselves rather than vulnerabilities in conventional software. "The Containment Gap" studies public-facing agents that call tools, maintain persistent memory, and execute multi-step plans, and asks whether frameworks provide architectural-level structural safety guarantees (Hossain et al., 11 Jun 2026). AgentSeer extends this to agentic-situational evaluation, arguing that model-level testing misses "agentic-only" vulnerabilities that emerge from tool use, memory, and multi-agent coordination (Wicaksono et al., 5 Sep 2025). AgentFixer, in turn, defines agentic vulnerability as any condition in an LLM-based workflow that breaks required invariants across input handling, prompt specification, or output generation, with validation focused on structured incident detection and repair (Mulian et al., 18 Feb 2026).

This suggests that agentic vulnerability validation is best understood as a family of evidence-centered control loops rather than a single benchmark protocol. The unifying requirement is that acceptance depends on externalized evidence: dynamic oracles, structural monitors, production telemetry, or formalized gates.

2. Validation objectives, oracles, and decision criteria

The validation objective depends on the artifact being assessed, but the papers converge on explicit acceptance predicates.

For web detection rules, RuleForge optimizes sensitivity and specificity, with standard definitions

• $TPR = TP / (TP + FN)$
• $TNR = TN / (TN + FP)$
• $FPR = FP / (FP + TN)$
• $\text{Precision} = TP / (TP + FP)$
• $\text{Recall} = TP / (TP + FN)$
• $$F1 = 2 \cdot (\text{Precision} \cdot \text{Recall}) / (\text{Precision} + \text{Recall})$$

and uses AUROC by ranking LLM-judge confidence scores against correctness labels derived from IP validation outcomes (Garg et al., 2 Apr 2026). Its production gate is concrete: synthetic testing uses ten tests per rule, production IP validation applies the rule to up to 5 billion sampled records, and acceptance requires 10–500 matches with more than 70% malicious IP enrichment (Garg et al., 2 Apr 2026).

For patch validation, KeaRepair uses a purely dynamic oracle. A repair is validated only when compilation succeeds, the PoC no longer triggers, and all unit tests pass; the closed loop continues until success or a pre-set iteration cap $N=5$ (Cao et al., 1 Jul 2026). MemRepair formalizes the same idea at repository scale:

• functionality preservation: $$\forall t \in \mathcal{T}: \mathcal{P}(t)=Pass \Rightarrow \mathcal{P}'(t)=Pass$$
• vulnerability mitigation: $\mathcal{P}'(\tau_{vuln})=Pass$

and treats failed validation as a source of refinement rather than a terminal error (Liu et al., 17 May 2026).

For exploit confirmation, AXE defines success by evaluator-defined oracles such as unauthorized file creation, DB modifications, or privilege escalation, and iterates under a bounded attempt budget $MaxA=5$ (Sajadi et al., 15 Feb 2026). A2 uses a stricter split between execution and verification: the Executor may claim success, but only the Task Validator, using read-only tools and dynamically constructed oracles, can declare PASS (Wang et al., 29 Aug 2025). PBFuzz formalizes PoV generation as satisfying both reachability and triggering predicates, $TNR = TN / (TN + FP)$0, with MAGMA_LOG instrumentation distinguishing "Bug X reached" from "Bug X triggered" (Zeng et al., 4 Dec 2025). Verify Before You Fix makes the repair gate explicit as an invariant: no repair is allowed unless there exists an execution trace $TNR = TN / (TN + FP)$1 such that $TNR = TN / (TN + FP)$2 (Gajjar, 12 Apr 2026).

For deployed agents, the oracles are architectural. The Containment Gap defines containment by requiring that executed actions remain in a policy-constrained safe set and memory writes satisfy integrity checks, with invariants such as

• $TNR = TN / (TN + FP)$3
• accept memory write $TNR = TN / (TN + FP)$4 only if $TNR = TN / (TN + FP)$5
• append-only hash chaining $TNR = TN / (TN + FP)$6

(Hossain et al., 11 Jun 2026). AgentSeer instead centers attack success rate, per-channel ASR, tool-level ASR, and tool uplift, all measured from action and component graphs extracted from execution traces (Wicaksono et al., 5 Sep 2025).

3. Architectural patterns and closed-loop workflows

Despite differing artifacts and domains, the systems exhibit a recurring control structure: hypothesis formation, tool-grounded execution, evidence capture, gated decision, and iterative refinement.

RuleForge implements a multi-stage detection-rule loop. For each CVE it generates five parallel candidates, each with up to five refinement attempts, using AWS Fargate and Amazon Bedrock via DSPy. The pipeline performs synthetic tests, invokes an LLM-as-a-judge for false-negative and false-positive risk, optionally runs production IP validation for header-only or bodyless checks, and finally submits candidates for human review (Garg et al., 2 Apr 2026). KeaRepair uses a ReAct-style diagnostic phase before validation, with static-analysis tools such as trace_data_flow, is_path_reachable, and inspect_api_usage, then executes a sequential validation loop of patch correction, compilation, PoC replay, and test execution (Cao et al., 1 Jul 2026).

Repository-level systems make the layered structure explicit. VulnAgent-X organizes validation as "screen, expand, analyze, verify, and fuse." It first computes a lightweight risk score

$TNR = TN / (TN + FP)$7

retains the top-$TNR = TN / (TN + FP)$8 regions with default $TNR = TN / (TN + FP)$9, expands bounded repository context under a default token budget $FPR = FP / (FP + TN)$0, dispatches specialized agents including a Sceptic Agent, triggers dynamic verification only for high-severity low-confidence or conflicting cases, and then fuses static, contextual, agentic, dynamic, and counter-evidence into a final score (Meng et al., 11 Mar 2026). MemRepair follows a Locator–Patcher–Verifier loop with persistent memory and a tri-state controller: Success, Regenerate, or Relocate (Liu et al., 17 May 2026).

Exploit-generation systems instantiate the same loop with different tools. AXE decouples Strategist, Explorer, Exploiter, and PoC Gen Module; the Strategist requests source-level facts from the Explorer, the Exploiter issues HTTP requests and external-tool calls, the evaluator checks impact, and the Strategist replans until success or budget exhaustion (Sajadi et al., 15 Feb 2026). SLYP separates discovery and PoC generation for Windows COM binaries: Stage 1 uses binary exploration, virtual-call resolution, and COM inspection to hypothesize races; Stage 2 uses dynamic debugging, page heap, AppVerifier, and iterative code synthesis until a crash with a call stack reaching the target confirms exploitability (Lee et al., 6 May 2026). PBFuzz makes the phases explicit as PLAN → IMPLEMENT → EXECUTE → REFLECT, with workflow gatekeepers that constrain what state can be modified in each phase (Zeng et al., 4 Dec 2025).

Agentic-system validation reuses the same staged logic, but the artifact is the agent framework or execution trace. The Containment Gap validates memory integrity and policy gates at architectural boundaries rather than within a code-repair or exploit loop (Hossain et al., 11 Jun 2026). AgentFixer implements a validation plane around each LLM call, running ingress checks, prompt analysis, output compliance checks, cross-stage consistency checks, and root-cause analysis over traces and aggregated incidents (Mulian et al., 18 Feb 2026).

4. Evidence sources, observability, and feedback integration

A central property of agentic vulnerability validation is that evidence is not limited to final pass/fail labels. The systems collect structured intermediate signals and feed them back into subsequent reasoning.

RuleForge’s judge is a discriminative evaluator rather than a generator. It asks negatively phrased questions—what is the probability the rule misses malicious requests, and what is the probability it targets a correlated feature rather than the root vulnerability—and computes

• $FPR = FP / (FP + TN)$1
• $FPR = FP / (FP + TN)$2
• $FPR = FP / (FP + TN)$3

If $FPR = FP / (FP + TN)$4 or $FPR = FP / (FP + TN)$5, the judge’s reasoning is appended to the next prompt (Garg et al., 2 Apr 2026). Negative phrasing is reported to improve AUROC and ECE relative to positive or generic confidence elicitation (Garg et al., 2 Apr 2026).

KeaRepair and MemRepair both elevate runtime evidence to a reusable asset. KeaRepair feeds compiler errors, PoC outputs, and test logs back to the agent for patch revision, while its retrieval stage is grounded in verified program facts rather than free-form speculation (Cao et al., 1 Jul 2026). MemRepair stores not only successful fixes and security rationales, but also failure-to-success transitions in Refinement-Trajectory memory, encoded as failed patch, corrective delta, and learned insight (Liu et al., 17 May 2026).

Observability becomes first-class in systems aimed at agentic-system vulnerabilities. AgentSeer constructs an action graph

$FPR = FP / (FP + TN)$6

and a component graph

$FPR = FP / (FP + TN)$7

from MLflow Generative AI traces, annotating actions with components_in_input and components_in_output so attack success can be localized to tools, handoffs, or memories (Wicaksono et al., 5 Sep 2025). AgentFixer combines rule-based checks with LLM-as-a-judge analyses, then aggregates incidents by category and severity, while its trace-comparison mode contrasts failing and successful trajectories to isolate root causes (Mulian et al., 18 Feb 2026).

Code-Augur uses a different evidence model: the agent externalizes its tacit assumptions as in-source security specifications and asks a guided fuzzer to falsify them. Invariant assertions expose otherwise opaque "secure" judgments, and assertion violations are triaged into genuine vulnerabilities or specification refinements (Luo et al., 17 Jun 2026). This suggests a broader interpretation of evidence in agentic validation: not only dynamic failures, but also counterexamples to the agent’s own explicit assumptions.

5. Empirical results across validation settings

The reported gains are heterogeneous because they measure different objects, but several quantitative patterns recur: staged validation reduces false positives, closed-loop refinement materially improves outcomes, and tool-grounded feedback shifts performance more than one-shot generation alone.

RuleForge reports AUROC $FPR = FP / (FP + TN)$8 and ECE $FPR = FP / (FP + TN)$9 on 100 examples for its confidence judge, and a 67% decrease in false positives versus synthetic-test-only validation, together with a 71% decrease in rules matching no IPs on fifty CVEs with a generation budget of five retries (Garg et al., 2 Apr 2026). KeaRepair repairs 46 of 55 reproducible C/C++ vulnerabilities, a repair rate of 83.64%, and the ablation without iteration drops to 26 of 55, which the paper describes as a 76.92% improvement attributable to the closed-loop validation and feedback refinement (Cao et al., 1 Jul 2026).

Exploit-focused systems show similar dependence on validation tooling. AXE achieves a 30% exploitation success rate on CVE-Bench, a 3x improvement over black-box baselines, and a 1.75x gain over a single-agent grey-box baseline (Sajadi et al., 15 Feb 2026). PBFuzz triggers 57 vulnerabilities on Magma, uniquely exposes 17 not triggered by existing fuzzers, and reports median time-to-exposure of 339 seconds versus 8680 seconds for AFL++ with CmpLog, a 25.6x efficiency improvement with an API cost of 1.83 USD per vulnerability (Zeng et al., 4 Dec 2025). SLYP reaches 0.973 F1 on 20 COM objects covering 40 vulnerability cases, and its best PoC-generation configuration verifies 67.5% of cases, whereas production coding agents in their default setup verify essentially no cases when the COM inspection and dynamic debugging tools are absent (Lee et al., 6 May 2026).

Repository-scale and cross-language repair systems show that validation is not a marginal auxiliary stage. Verify Before You Fix reports 69.74% end-to-end resolution and a 12.27% total failure rate, while disabling validation increases unnecessary repairs by 131.7% (Gajjar, 12 Apr 2026). MemRepair achieves 58.0% on SEC-Bench, 58.2% on PatchEval, and 30.58% on the C++ subset of Multi-SWE-bench; its ablation with only the iterative loop and no memory drops to 36.0%, while the full L1+L2+L3 memory stack reaches 58.0% on SEC-Bench (Liu et al., 17 May 2026). VulnAgent-X reports that removing dynamic verification reduces PrimeVul F1 from 0.316 to 0.279, and that the Sceptic Agent lowers false positives by raising precision and lowering FPR (Meng et al., 11 Mar 2026).

For agentic-system vulnerabilities, the empirical signal is that aggregate accuracy alone can conceal targeted harms. The Containment Gap shows that in a simulated government benefits agent, a single poisoned memory write can drive targeted wrongful denial to 88.9%, and under a complex five-factor policy the same attack preserves near-baseline aggregate accuracy while increasing targeted wrongful denials by about 3.5x; the proposed memory validator and policy gate eliminate both attack vectors with sub-millisecond overhead (Hossain et al., 11 Jun 2026). AgentSeer finds that tool-calling increases ASR by 24% relative uplift for GPT-OSS-20B and 60% for Gemini-2.0-flash, and that context-aware iterative agentic attacks succeed on objectives that failed at model level (Wicaksono et al., 5 Sep 2025).

6. Limitations, controversies, and open problems

The literature is explicit that validation remains bounded by the quality of its oracles, tools, and abstractions. KeaRepair acknowledges that passing available PoCs and unit tests does not guarantee the absence of regressions not covered by the suite (Cao et al., 1 Jul 2026). A2 reports that some vulnerabilities are inherently unsuitable for its emulator-only validation setup, including network MITM cases, and that Validator filtering reduces but does not eliminate Executor hallucinations (Wang et al., 29 Aug 2025). Patch2Vuln shows an even sharper boundary: bounded old/new validation produced only two minimized behavioral differentials on 25 package pairs, with no crash, timeout, sanitizer finding, or memory-corruption proof, and its authors attribute much of the failure to binary-diff coverage and missing reachability tracing (David et al., 7 May 2026).

Several works identify model-coupling and judge bias as structural risks. RuleForge notes that its evaluator uses the same model family as generation, which may introduce self-preference bias, and that generic confidence elicitation is often overconfident and poorly discriminative (Garg et al., 2 Apr 2026). "Helpful Agent Meets Deceptive Judge" broadens this concern, showing that feedback-based workflows are vulnerable to persuasive but flawed critiques, including grounded adversarial feedback, and formalizes metrics such as $\text{Precision} = TP / (TP + FP)$0, recovery score, flip probability, and hazard of switching per round (Ming et al., 3 Jun 2025). This is directly relevant to agentic validation pipelines that rely on judges to critique generators.

There is also a foundational limit to what "validation" can claim. "Limits to Verification and Validation of Agentic Behavior" argues that for non-trivial, viable behavioral standards, deciding whether a policy always preserves acceptable behavior is not computable, and that validation of physical outcomes in open-world environments cannot provide certainty (Jilk, 2016). In the context of agentic vulnerability validation, this does not negate the value of staged testing, containment, or dynamic oracles, but it does rule out certainty claims once the environment, attacker behavior, or consequences exceed the bounded formalism of the test harness.

A plausible implication is that future progress will continue along two axes already visible in the surveyed work. One axis is stronger evidence: richer runtime instrumentation, better PoCs, fuzzing, symbolic execution, and provenance-aware observability. The other is stronger control structure: tighter gates, persistent memory of failures, explicit specification externalization, and validation-aware agent orchestration. The field’s current trajectory favors systems that treat validation not as post hoc scoring, but as the organizing principle of the entire agent loop.