Refute-or-Promote: An Adversarial Stage-Gated Multi-Agent Review Methodology for High-Precision LLM-Assisted Defect Discovery

Abstract: LLM-assisted defect discovery has a precision crisis: plausible-but-wrong reports overwhelm maintainers and degrade credibility for real findings. We present Refute-or-Promote, an inference-time reliability pattern combining Stratified Context Hunting (SCH) for candidate generation, adversarial kill mandates, context asymmetry, and a Cross-Model Critic (CMC). Adversarial agents attempt to disprove candidates at each promotion gate; cold-start reviewers are intended to reduce anchoring cascades; cross-family review can catch correlated blind spots that same-family review misses. Over a 31-day campaign across 7 targets (security libraries, the ISO C++ standard, major compilers), the pipeline killed roughly 79% of 171 candidates before advancing to disclosure (retrospective aggregate); on a consolidated-protocol subset (lcms2, wolfSSL; n=30), the prospective kill rate was 83%. Outcomes: 4 CVEs (3 public, 1 embargoed); LWG 4549 accepted to the C++ working paper; 5 merged C++ editorial PRs; 3 compiler conformance bugs; 8 merged security-related fixes without CVE; an RFC 9000 errata filed under committee review; and 1+ FIPS 140-3 normative compliance issues under coordinated disclosure -- all evaluated by external acceptance, not benchmarks. The most instructive failure: ten dedicated reviewers unanimously endorsed a non-existent Bleichenbacher padding oracle in OpenSSL's CMS module; it was killed only by a single empirical test, motivating the mandatory empirical gate. No vulnerability was discovered autonomously; the contribution is external structure that filters LLM agents' persistent false positives. As a preliminary transfer test beyond defect discovery, a simplified cross-family critique variant also solved five previously unsolved SymPy instances on SWE-bench Verified and one SWE-rebench hard task.

• The paper presents a stage-gated, adversarial multi-agent review that cuts false positives by achieving a 79% pre-disclosure rejection and an 83% kill rate in refined protocols.
• The methodology employs mechanisms like Stratified Context Hunting, Adversarial Kill Mandates, and Cross-Model Critic to address shared biases and enhance review precision.
• Empirical validation over a 31-day campaign in security libraries and standards domains demonstrates the method’s practical impact and highlights trade-offs between precision and recall.

Refute-or-Promote: Stage-Gated Adversarial Multi-Agent Review for LLM-Assisted High-Precision Defect Discovery

Motivation and Problem Scope

Refute-or-Promote addresses the acute precision deficit in LLM-assisted defect discovery for real-world codebases. The uncontrolled proliferation of plausible-but-incorrect defect reports originating from LLMs frequently overwhelms maintainers and degrades trust in AI-generated findings. Empirical evidence from large-scale triage crises—such as the permanent closure of curl’s bug bounty and the pausing of HackerOne's Internet Bug Bounty—demonstrates that high-volume, low-precision LLM reporting renders conventional triage economically and operationally unsustainable.

The core premise is that LLMs, including ensembles of powerful proprietary and open-source models, exhibit strong plausibility bias and correlated error patterns: they commonly achieve surface-level agreement even when the underlying analysis is erroneous. Standard multi-agent debate and ensemble aggregation mechanisms—without explicit adversarial incentives, context independence, and empirical grounding—do not provide sufficient precision for responsible vulnerability disclosure.

Methodology: Adversarial, Stage-Gated, Multi-Agent Review

The proposed methodology forms a rigorously engineered adversarial pipeline with four primary components:

• Stratified Context Hunting (SCH): Parallel candidate generation stratified across source context, code scope, and iterative waves, with per-candidate self-critique.
• Adversarial Kill Mandates: Explicit track assignment to adversarial agents with a destructive, not merely evaluative, role—agents must find grounds to disprove, not refine or converge.
• Context Asymmetry: Systematic context isolation and cold-start reviewer assignment to mitigate anchoring bias and correlated error cascades.
• Cross-Model Critic (CMC): Final-stage critique using agents from distinct model families, targeting error modes unobservable in homogeneous ensembles.
• Mandatory Empirical Validation: No candidate advances to disclosure absent empirical demonstration (e.g., PoC execution, test suite confirmation).

The pipeline enforces staged escalation—across creative and adversarial agents, increasing review diversity, and empirical gates—before human operators consider final findings. Notably, the methodology is retrospectively codified from extensive field experience, not ex ante protocol specification.

Empirical Evaluation and Results

Over a 31-day campaign spanning seven targets (security libraries, the ISO C++ standard, production compilers), the Refute-or-Promote pipeline processed approximately 171 candidate findings. Of these, approximately 79% were rejected pre-disclosure by adversarial review, with the consolidated-protocol subset achieving an 83% kill rate.

Externally validated outcomes comprise:

• 4 CVEs (3 public, 1 embargoed): affecting libfuse, lcms2, and OpenSSL
• Acceptance of ISO C++ standard defects and editorial changes: including LWG 4549 and five merged editorial PRs
• Three confirmed compiler conformance bugs
• Eight merged security-related fixes (with/without CVE)
• Notable campaign-level failures, such as the unanimous (80+ agent) false-positive endorsement of a Bleichenbacher oracle in OpenSSL, later falsified by empirical testing—a critical justification for the empirical validation gate

In cross-model analysis, approximately 16% of fixes approved by same-family agents in libfuse were flagged for correctness errors by CMC reviewers from different model families. Unanimity among LLM agents, even across substantial agent populations, was shown to be non-correlative to correctness and often a marker for shared bias.

The pipeline exhibited robust transfer to non-security domains (e.g., ISO C++ specification) where no executable oracle exists, but highlighted cases of negative transfer when methodology design did not adapt to the new validation oracles or empirical regimes.

Implications for LLM Reliability, Precision, and Review Architecture

The study exposes the severe limitations of consensus-based LLM evaluation and cooperative debate frameworks for high-stakes, high-precision tasks such as vulnerability discovery. The systematic implementation of adversarial mandates, context asymmetry, cross-model review, and empirical validation addresses key classes of false positives:

• Reasoning Errors: Mitigated by adversarial kill agents and creative/adversarial review
• Anchoring and Cascading Bias: Reduced through context isolation and cold-start agent deployment
• Correlated Model Errors: Addressed by the CMC cross-family critique mechanism

Furthermore, empirical validation is non-negotiable; the inability of even large, diverse agent populations to consistently disprove high-plausibility but incorrect findings, as shown by the OpenSSL Bleichenbacher episode, demonstrates the necessity of runtime gating.

The pipeline’s unidirectional, precision-optimized nature makes recall dependent on explicit human or architect-driven resurrection of improperly killed true positives—highlighting a central tradeoff between false positive suppression and the risk of false negatives. The methodology does not automate the override or rule-modification process; substantive human orchestration remains integral for high-signal findings, rule refinement, and escalation path construction.

Limitations and Prospective Directions

The evaluation lacks ablation studies isolating individual pipeline components, and methodology evolution during campaigns introduces confounding variables. Results are strongest in domains with mature validation oracles (C/C++ security, standards conformance), with less evidence for transferability to structurally dissimilar domains. Operator expertise and manual prompt engineering also factor meaningfully into effectiveness.

Potential AI research directions include:

• Exploration of decentralized, debate-based adversarial gating mechanisms
• Automation of override/resurrection agents to reduce dependence on human orchestrators
• Extension of this architecture to autonomous agents, proof-assistant guidance, or formal methods integration
• Large-scale, controlled evaluation across diverse model families and domains, with rigorous component ablation

Conclusion

Refute-or-Promote introduces and empirically validates an adversarial, stage-gated, multi-agent pipeline for LLM-assisted defect discovery in production-relevant code and specification bases. The central finding is that model consensus is uninformative in the presence of shared biases and that reliability emerges only from adversarial, context-isolated, and cross-model challenge—culminating in empirical runtime validation. The architecture and supporting protocol offer a reproducible template for reducing persistent LLM false positives when independent ground truth can be established, with potential applicability far beyond software security defect discovery.

This contribution provides a formal methodology, open-source orchestration artifacts, and strong evidence for the necessity of adversarial decomposition and empirical grounding in trustworthy LLM system deployment (2604.19049).