MAVUL: Multi-Agent Vulnerability Detection

• MAVUL is an LLM-based multi-agent system for vulnerability detection using cross-procedural contextual reasoning and interactive refinement.
• By integrating a vulnerability analyst, security architect, and evaluation judge agent, MAVUL iteratively refines its detection accuracy through feedback loops.
• Evaluations on pairwise datasets show MAVUL achieves over 62% higher pairwise accuracy and 600% improved performance compared to single-agent systems.

MAVUL, short for Multi-Agent Vulnerability Detection via Contextual Reasoning and Interactive Refinement, is an LLM-based system for vulnerability detection in open-source software that is designed around cross-procedural code understanding, multi-round agent interaction, and fine-grained evaluation. It targets a setting in which binary function-level classification is insufficient because real vulnerabilities often depend on callers, callees, repository context, and patch semantics. The framework combines a vulnerability analyst agent, a security architect agent, and an evaluation judge agent, and was reported to achieve over 62% higher pairwise accuracy than existing multi-agent systems and over 600% higher average performance than single-agent systems on a pairwise vulnerability dataset (Li et al., 30 Sep 2025).

1. Problem scope and formal setting

MAVUL addresses vulnerability detection in real-world open-source software, with emphasis on cross-procedural, repository-level vulnerabilities, contextual reasoning over code relationships, and evaluation that goes beyond a vulnerable/non-vulnerable label. The motivating claim is that many prior methods operate on isolated functions, while actual vulnerabilities often require understanding how data flows through callers and callees, how patches change behavior, and whether a diagnosis matches the ground-truth vulnerability type rather than merely the ground-truth binary label.

The paper situates this problem against a conventional formulation of vulnerability detection as binary classification over functions. Given a dataset $\mathcal{D}$ of pairs $(\chi,\psi)$, where $\chi$ is the target function and $\psi \in \{0,1\}$ is the vulnerability label, traditional methods optimize

$$\min_{\theta_\mathcal{E}, \theta_\mathcal{C}} \sum_{(\chi, \psi) \in \mathcal{D}} L_{CE}\big(\mathcal{C}(\mathcal{E}(\chi)), \psi\big).$$

This formulation is presented as background rather than as MAVUL’s own training objective; MAVUL itself is not trained end-to-end in this way, but instead uses prompted GPT-4o agents at inference time.

The evaluation setting is pairwise. Each pair contains a pre-patched vulnerable version of a function and a post-patched fixed version. The pairwise design is central because it tests whether a system can distinguish the vulnerable pre-patch from the safe post-patch while also aligning its reasoning with richer ground truth such as CWE type, CVE description, commit patch, commit message, callers, and callees. A central implication is that binary correctness alone can be misleading: a system may flag the vulnerable version for the wrong reason and still appear correct under coarse evaluation.

2. Agent architecture and control flow

MAVUL is organized around three GPT-4o agents with distinct roles: a vulnerability analyst agent, a security architect agent, and an evaluation judge agent. The analyst is the primary detector. It inspects the target function, reasons about possible flaws, invokes tools to retrieve contextual code, and emits a structured report. The architect serves as a reviewer that examines the analyst’s entire reasoning trajectory and final report, then either agrees or issues a critique. The judge is not part of the detection loop; it compares the final analyst report against multi-dimensional ground truth and returns a verdict of MATCH or MISMATCH.

The analyst’s output is constrained to a JSON schema: $(\chi,\psi)$1 The architect likewise returns a strict JSON object: $(\chi,\psi)$2 The judge returns: $(\chi,\psi)$3

The control flow is iterative. The analyst first analyzes the function and may perform multiple internal Reason–Act–Observe steps. Its output, along with the full Thought–Action–Observation trajectory, is sent to the architect. If the architect sets agreement = false, the analyst receives the critique as “History of Previous Attempts and Critiques” and must begin a new analysis that explicitly addresses the feedback. This continues for at most 3 rounds or until agreement is reached. Only then is the analyst’s final report evaluated by the judge.

A common misunderstanding addressed by the design is that “multi-agent” merely means multiple parallel answers. In MAVUL, the key distinction is cross-role interaction with iterative feedback, not just multiple decoders. The architect reviews the analyst’s intermediate reasoning, not merely the final label, and this feedback loop is treated as a primary source of error correction.

3. Contextual reasoning and cross-procedural code understanding

The paper uses the term contextual reasoning to denote reasoning that goes beyond the local body of the target function and explicitly incorporates surrounding repository context. In MAVUL, this is implemented through a ReAct-style loop in which the analyst alternates between reasoning steps and tool calls. The analyst can retrieve callers, callees, and function bodies, then revise its interpretation of the target function in light of those observations.

The available tools are exactly three:

• get_function_body(name): returns the full source code of the named function.
• get_callers(name): returns all functions that directly call the named function.
• get_callees(name): returns all functions directly invoked by the named function.

These tools expose pre-extracted information generated by static analysis, with Clang used to construct caller/callee mappings and function bodies. The analyst decides when to use them. This means that cross-procedural understanding is not encoded as an explicit symbolic graph algorithm inside MAVUL; instead, the LLM reconstructs relevant slices of program behavior from retrieved function-level context.

This structure is intended to support tasks such as tracing whether a size parameter passed into a sink is validated in the caller, determining whether allocation and use occur in different procedures, or checking whether a callee imposes constraints that the target function relies on. The paper’s case study illustrates this with cdf_read_short_sector, where the analyst initially over-trusted local boundary checks, then—after critique—revisited the function in relation to its caller and identified missing validation of buf size and unsafe reliance on assert.

A useful way to interpret this design is as a repository-aware alternative to function-only LLM prompting. That interpretation is not formalized mathematically in the paper, but the reported ablations suggest that caller/callee retrieval materially changes the error profile by reducing missed vulnerabilities.

4. Interactive refinement and fine-grained evaluation

Interactive refinement is the core mechanism through which MAVUL attempts to improve reliability. The analyst’s first-pass reasoning is not treated as authoritative. Instead, the architect inspects the original function, the entire analyst trajectory, and the final JSON assessment, then either accepts the analysis or issues a targeted critique. When disagreement occurs, the analyst is required to start a new analysis that directly addresses the critique rather than merely appending a justification to its previous answer.

The paper’s detailed case study shows this process concretely. In the first round, the analyst concluded that cdf_read_short_sector was not vulnerable because it observed boundary checks on pos. The architect then objected on two grounds: reliance on assert(ss == len) suggested CWE-617 (Reachable Assertion), and the absence of validation that buf was large enough for len before memcpy suggested CWE-120 (Buffer Overflow). In the second round, the analyst re-examined the same function and revised its report to vulnerable, with a buffer-overflow diagnosis and an explanation that explicitly referenced missing buffer-size checks and reliance on assert. The architect then agreed.

The evaluation judge is the second major innovation. Rather than marking a prediction correct whenever the binary label matches, the judge compares the analyst’s report with multi-dimensional ground truth information: binary label, CWE type, CVE description, commit patch, and commit message. For pre-patched code, the judge returns MATCH if the analyst identifies the ground-truth vulnerability in that code, even if other vulnerabilities are also mentioned. For post-patched code, the judge returns MATCH if the analyst correctly concludes non-vulnerable and does not incorrectly claim that the original ground-truth vulnerability is still present.

This evaluation logic is paired with pairwise outcome categories. Over $N$ pairs, with $C$, $V$, $B$, and $R$ denoting counts of pair-correct, pair-vulnerable, pair-benign, and pair-reversed outcomes, MAVUL reports

$(\chi,\psi)$0

The paper treats P-C as the main pairwise accuracy indicator.

The judge is explicitly framed as an “unbiased judge” because it prevents misleading binary comparisons. Manual validation on approximately 50 random samples reportedly showed over 95% accuracy relative to human assessment (Li et al., 30 Sep 2025).

5. Dataset, baselines, and empirical performance

MAVUL is evaluated on a pairwise vulnerability dataset derived from JitVul, itself built on PrimeVul. The construction pipeline begins from 879 pairs, removes redundant and incomplete cases to yield approximately 600 pairs, and then randomly samples 200 pairs for evaluation because multi-agent, multi-round analysis with tool use is token-intensive. The resulting evaluated set spans 90 projects and remains representative of the original CWE distribution.

The full set statistics reported in the paper are substantial: 211 projects, target-function line counts from 10 to 4688 with average 354, context lines from 0 to 41722 with average 1540, 0–203 callees with average 19, and 0–32 callers with average 2. In the evaluated subset, target-function lines range from 14 to 3644 with average 335, context lines from 27 to 41722 with average 1938, callees from 1 to 203 with average 21, and callers from 1 to 32 with average 2. All code is C/C++ (Li et al., 30 Sep 2025).

The baselines comprise two single-agent systems—CoT and JitVul—and two multi-agent systems—GPTLens and VulTrial. CoT is implemented as analyst-style prompting without tools and without the architect. JitVul is a single-agent, tool-using contextual system without multi-round architect feedback. GPTLens uses multiple auditor agents and a critic to rank vulnerability candidates. VulTrial uses a mock-court format with a security researcher, code author, moderator, and review board, but does not use the contextual tools employed by MAVUL.

The main results are summarized below.

Method	P-C	P-V	P-B	P-R
CoT	1.5	7.5	24.0	67.0
JitVul	3.5	0.0	81.0	15.5
GPTLens	13.5	22.0	43.0	21.5
VulTrial	8.0	9.0	65.5	17.5
MAVUL	17.5	5.5	43.5	33.5

These numbers support several specific conclusions. First, MAVUL achieves the highest P-C, which the paper interprets as pairwise accuracy. Second, its P-V is markedly lower than GPTLens, indicating fewer cases where the vulnerable pre-patch is correctly flagged but the safe post-patch is still over-flagged. Third, P-B is dramatically lower than JitVul, indicating fewer missed vulnerabilities among pre-patched functions. The paper quantifies the improvement as about 62.8% higher than the average P-C of GPTLens and VulTrial and 7× the average P-C of CoT and JitVul, corresponding to the “over 600% higher average performance” claim (Li et al., 30 Sep 2025).

Round-wise results further attribute these gains to iterative analyst–architect communication. Reported P-C values rise from 3.5 in Round 1 to 10.0 in Round 2 and 17.5 in Round 3, while P-B drops from 81.0 to 44.0 and then 43.5. The paper states that the system’s effectiveness is markedly improved with increased communication rounds, especially through reduction of false negatives.

Ablation results separate the effects of contextual reasoning and architect feedback. A CoT-like configuration without tools and without architect reaches P-C = 1.5%. A JitVul-like configuration with tools but no architect reaches 3.5%. An analyst-plus-architect configuration without tools reaches 9.0%. The full MAVUL configuration reaches 17.5%. The paper explicitly concludes that both contextual reasoning and the architect are essential, and notes that removing the architect causes P-C to drop from 17.5% to 3.5%, an 80% decrease.

6. Evaluation controversy, limitations, and significance

A major theme of MAVUL is that coarse-grained evaluation is itself a source of error. The paper defines an Error Rate measuring how often simple binary evaluation disagrees with the judge-based multi-dimensional evaluation. This comparison is especially unfavorable to prior multi-agent baselines. Reported Error Rates are 91.7% for GPTLens, 88.6% for VulTrial, and 60.4% for MAVUL. In the judge-free setting, VulTrial appears to achieve P-C = 22.0%, but this falls to 8.0% once the evaluation judge checks type and reasoning. MAVUL also drops—from 24.0% to 17.5%—but much less severely. The intended conclusion is not simply that MAVUL performs better, but that a substantial portion of previously reported success can be an artifact of binary-only scoring (Li et al., 30 Sep 2025).

The framework’s limitations are explicit. It is token-expensive and cost-intensive, which is why evaluation is limited to 200 pairs and at most 3 interaction rounds. It depends heavily on GPT-4o, with the authors noting that weaker models may use tools poorly; DeepSeek-R1 is mentioned as having low tool-use “hit rates.” The experiments cover C/C++ OSS projects only, so transfer to other programming languages would require prompt and tooling adaptation. Tooling quality is also a dependency: incorrect caller/callee extraction could distort the analyst’s reasoning. Finally, despite the relative gains, the reported P-C of 17.5% indicates that the setting remains difficult and that the system is not presented as production-level vulnerability detection.

Its broader significance lies in the combination of three ideas that the paper treats as jointly necessary for realistic LLM-based vulnerability detection: cross-procedural contextual reasoning, multi-round multi-agent interaction, and LLM-as-a-judge evaluation with multi-dimensional ground truth. This suggests a shift away from function-only, one-shot prompting and away from purely binary benchmarking. A plausible implication is that future repository-level vulnerability detection systems will need to treat reasoning traces, code retrieval, and evaluation semantics as first-class components rather than auxiliary features.