VCAO: Verifier-Centered Agentic Orchestration for Strategic OS Vulnerability Discovery

Abstract: We formulate operating-system vulnerability discovery as a \emph{repeated Bayesian Stackelberg search game} in which a Large Reasoning Model (LRM) orchestrator allocates analysis budget across kernel files, functions, and attack paths while external verifiers -- static analyzers, fuzzers, and sanitizers -- provide evidence. At each round, the orchestrator selects a target component, an analysis method, and a time budget; observes tool outputs; updates Bayesian beliefs over latent vulnerability states; and re-solves the game to minimize the strategic attacker's expected payoff. We introduce \textsc{VCAO} (\textbf{V}erifier-\textbf{C}entered \textbf{A}gentic \textbf{O}rchestration), a six-layer architecture comprising surface mapping, intra-kernel attack-graph construction, game-theoretic file/function ranking, parallel executor agents, cascaded verification, and a safety governor. Our DOBSS-derived MILP allocates budget optimally across heterogeneous analysis tools under resource constraints, with formal $\tilde{O}(\sqrt{T})$ regret bounds from online Stackelberg learning. Experiments on five Linux kernel subsystems -- replaying 847 historical CVEs and running live discovery on upstream snapshots -- show that \textsc{VCAO} discovers $2.7\times$ more validated vulnerabilities per unit budget than coverage-only fuzzing, $1.9\times$ more than static-analysis-only baselines, and $1.4\times$ more than non-game-theoretic multi-agent pipelines, while reducing false-positive rates reaching human reviewers by 68\%. We release our simulation framework, synthetic attack-graph generator, and evaluation harness as open-source artifacts.

• The paper introduces a game-theoretic framework formalizing OS vulnerability discovery as a repeated Bayesian Stackelberg search game.
• It integrates heterogeneous analysis tools into an agentic orchestration system that optimally allocates resources and reduces false positives by 68%.
• Empirical results show up to 2.7× improvement in severity-weighted findings and enhanced sibling vulnerability discovery across diverse Linux subsystems.

Verifier-Centered Agentic Orchestration for Strategic OS Vulnerability Discovery

Problem Formulation and Theoretical Foundation

The paper introduces a formalization of operating-system (OS) vulnerability discovery as a repeated Bayesian Stackelberg search game, with a Large Reasoning Model (LRM) orchestrator as the defender strategically allocating analysis resources. The attacker is modeled as a rational agent navigating an intra-kernel attack graph to maximize damage, while the defender leverages sequential Bayesian updates and re-solves to minimize the attacker's expected payoff. The intra-kernel attack graph delineates possible privilege escalation and data exfiltration paths, annotating nodes and edges with class-conditional prior probabilities and exploitability metrics. This model enables the explicit embedding of domain knowledge such as CVSS-severity-based vulnerability priors and historical defect density.

The core optimization involves solving a defender-orchestrated resource allocation problem, subject to budget constraints, where heterogeneous analysis tools (e.g., static analyzers, fuzzers, and sanitizers) each have cost, coverage, and false-positive parameters. The defender's action space encompasses the selection of kernel file/function targets, analysis methods, and time budgets, tightly integrating the agentic activation of various tools. The Stackelberg game structure is leveraged to capitalize on defender commitment power, with optimal mixed strategies derived via DOBSS-based mixed-integer linear program (MILP), incorporating attacker type distributions and uncertain latent vulnerability states. The paper also demonstrates a formal sublinear regret guarantee ($O(\sqrt{T})$) for the orchestrator under online learning from noisy tool observations.

VCAO Architecture

The Verifier-Centered Agentic Orchestration (VCAO) architecture operationalizes the theoretical framework with six distinct yet interdependent layers:

1. Surface Mapper (L1): Extracts security-relevant OS entry points (syscalls, ioctls, parsers) and establishes the foundation for subsequent reachability analysis.
2. Intra-Kernel Attack Graph Builder (L2): Constructs the attack graph, with privilege transitions and data-flow dependencies marked as nodes/edges, leveraging CVSS-based probability estimation.
3. Game-Theoretic Ranker (L3): Performs real-time solution of the DOBSS-VD MILP, yielding optimal budget allocation vectors based on current beliefs and attacker posteriors.
4. Parallel Executor Agents (L4): Specialized agents (e.g., Patch-Diff Miner, CodeQL, Fuzzing, KASAN, KCSAN) receive prioritized instructions to probe distinct vulnerability classes.
5. Cascaded Verifier/Critic (L5): Implements a multi-stage validation pipeline (reproducibility, severity, deduplication) prior to human review, dramatically decreasing false-positive rates.
6. Safety Governor (L6): Enforces security constraints including full isolation, audit logs, and mandatory human triage prior to any vulnerability disclosure.

This agentic system exhibits a feedback-driven orchestration loop: tool outputs are processed into updated Bayesian beliefs, which directly inform subsequent game-theoretic reallocations.

Algorithms and Evaluation Methodology

The orchestration loop is well-defined: in each round, the orchestrator solves the DOBSS-VD MILP for budget allocation, dispatches analysis actions to the executor agents, collects tool outputs, updates Bayesian beliefs, implements sibling-bug-triggered searches, and applies cascaded verification. The attack path enumeration employs a belief-weighted priority queue to control combinatorial explosion. The system explicitly models and allocates for the discovery of vulnerabilities structurally related (“sibling patterns”) to previously validated findings.

Experiments were performed on five diverse Linux kernel subsystems across two modes: (1) replaying 847 historical CVEs for ground-truth convergence metrics (e.g., time-to-first-discovery, severity-weighted findings per budget), and (2) live discovery on mainline kernel snapshots, with findings validated via manual reproduction.

Baselines included uniform budget allocation, churn-based heuristics, traditional Syzkaller and CodeQL deployments, and a non-game-theoretic multi-agent pipeline. Major metrics were validated finding yield (SVUB), false positive rate at human review, sibling vulnerability discovery, and modeled attacker payoff reduction.

Empirical Results

Empirical analysis reveals that VCAO achieves substantial improvements:

• SVUB: $2.7\times$ over fuzz-only, $1.9\times$ over static-only, and $1.4\times$ over multi-agent (no game theory) on validated, severity-weighted vulnerability findings per unit budget.
• False Positive Rate: A decrease from 31–47% under single-tool deployments to 15.1% for VCAO, representing a 68% reduction.
• Sibling-Bug Discovery: VCAO augmented with sibling search discovers previously undetected clusters, outperforming its ablated variant by +9.7% SVUB.
• Budget Efficiency: The discovery yield advantage widens as analysis budgets increase, due to optimal reallocation that avoids diminishing returns.
• Ablation Studies: Bayesian updates, explicit Stackelberg optimization, and attack-graph-based structure are the largest contributors to VCAO's performance gap.

Notably, these improvements are consistent across subsystems, with the largest gains in areas such as networking and namespace/capability logic, where the attack surfaces are most complex and poorly served by naive fuzzing.

Theoretical and Practical Implications

The adoption of Bayesian Stackelberg games for OS vulnerability discovery reframes automated security from a myopic coverage race to a fine-grained, adversary-aware resource allocation problem. By making optimal use of tool diversity and available analysis budgets, VCAO substantially improves defender outcomes while minimizing wasted review effort. The architecture is modular, enabling adaptation to new tool capabilities and integration with future LRM scales.

The formal regret bounds assure that continuous deployment over time maintains performance close to the (unavailable) oracle allocation. The separation of verification layers provides a pragmatic reduction in the human triage burden—a crucial practical consideration for both open source and enterprise kernel security teams.

This work articulates a defensively aligned, safety-conscious protocol for agentic vulnerability discovery—a salient consideration as LLM- and LRM-based agents become increasingly capable and accessible.

Limitations and Future Directions

VCAO’s Stackelberg formulation presumes rational attackers and requires well-calibrated tool observation models; generalization to attackers with arbitrary or evolving objectives remains to be fully explored. The intra-kernel attack graph construction, while principled, demands further automation to robustly track privilege transitions in rapidly evolving kernels. Tool-specific efficacy is version-dependent and may require frequent recalibration.

Potential future developments include: real-time integration with threat intelligence for attacker type adaptation, automated privilege boundary inference with formal verification, and extension to distributed multi-OS attack graphs. Scaling to massive monolithic codebases and incorporating exploitability prediction with next-generation LRMs are likely to further amplify defender advantage.

Conclusion

This paper presents VCAO, a comprehensive, game-theoretically grounded, verifier-centered orchestration system for OS vulnerability discovery. By unifying Stackelberg security games, intra-kernel attack-graph modeling, Bayesian online learning, and cascaded agentic tool application, VCAO establishes new standards for validated vulnerability yield, resource efficiency, and operational safety. The open-sourcing of the simulation and evaluation suite offers a valuable resource for reproducible research and continued progress in automated kernel security.