Archer: Towards Agentic Review for Compiler Optimizations

Abstract: Modern compilers are frequently updated, but expert review capacity is highly limited, leading to delayed integration and, in some cases, subtle semantic bugs entering the compiler codebase. Automating the code review process with modern general code review agents may be feasible, but it faces critical challenges due to compiler complexity. In this paper, we use LLVM as our target compiler and present Archer, the first automated agentic code review tool for compiler optimizations. Archer constrains the agentic review process from both ends by using obligations to guide analysis and a deterministic validation guard to admit only findings backed by executable evidence. We evaluated Archer on 70 open PRs and 328 closed PRs in LLVM from the last two months. The review results are shocking and concerning: Archer discovers that 21% of open PRs and 11% of closed PRs are buggy, i.e, introducing semantic bugs such as miscompilations in LLVM. Our findings expose a critical gap in the capacity for critical review in large compiler projects and demonstrate the practical value of Archer as an additional reviewer.

• The paper introduces an LLM-driven framework that automates the semantic review of LLVM compiler optimization patches by leveraging historical bug-fix patterns and dynamic obligation construction.
• It employs a multi-stage pipeline that constructs pass-specific semantic obligations and enforces deterministic, executable validation using tools like Alive2.
• Evaluation on real-world LLVM PRs shows Archer's superior bug detection accuracy with minimal false positives compared to general-purpose LLM review agents.

Archer: Towards Agentic Review for Compiler Optimizations

Introduction and Motivation

Archer introduces an LLM-driven agentic framework specifically targeting semantic review of compiler optimization patches, with a particular focus on the LLVM compiler infrastructure. The paper situates its contribution in the context of substantive review bottlenecks in major compiler projects (e.g., LLVM), where expert bandwidth is consistently outpaced by the volume and complexity of optimization patches. Unlike general-purpose code review agents, code review for compiler optimizations is uniquely challenging: it is not only patch-specific and highly sensitive to program semantics, but also requires actionable, evidence-based validation of the preservation of correctness properties.

A key insight underlying Archer is that effective automated review must (1) leverage a repository of historical correctness fixes to distill and guide the detection of fragile semantic interactions (i.e., using "obligations"), and (2) admit only findings that can be validated via deterministic, executable evidence, thus avoiding the well-documented pitfalls of hallucinated or unfounded LLM rationales.

Figure 1: Archer conducting automated, expert-like review of an LLVM optimization PR, demonstrating obligation guidance and executable validation.

System Design

Archer’s architecture explicitly constrains the agentic review workflow on both the input and output sides. On the input end, dynamic obligation construction abstracts semantic patterns from historical bug-fix patches, producing reusable IR-level semantic obligations at the granularity of compiler passes. On the output end, a deterministic validation guard ensures that any review finding must be backed by executable, patch-specific evidence that demonstrates a semantic violation under the candidate optimization.

Figure 2: High-level workflow of Archer, integrating dynamic obligation construction and deterministic validation into the agentic review process.

Dynamic Obligation Construction

Obligations are constructed in a multi-stage pipeline leveraging several LLMs:

1. Bug-fix Patch Grouping: Historical fixes are grouped by transformation pass.
2. Obligation Abstraction: LLM-A abstracts the semantic rationale underlying each fix into a candidate obligation, discarding low-level or outdated details.
3. Executable Reproducer Generation: LLM-B concretizes the candidate obligation into an LLVM IR pair that demonstrates the original bug.
4. Validation: The obligation is retained only if translation validation (typically using Alive2) confirms a semantic discrepancy in the IR pair.
5. Summarization: LLM-C summarizes all validated obligations for each pass, yielding a compact, LLM-usable set of pass-level obligations.

   Figure 3: Example of Archer's automated construction of pass-level obligations for the InstCombine optimization pass.

Guided Agentic Analysis

At review time, constraints imposed by per-pass obligations direct the agent's search and analysis actions. The agent explores the codebase, retrieves relevant context, inspects IR semantics, and develops validation strategies—mutations with explicit semantic rationales and expected validation outcomes.

Figure 4: Illustration of Archer's semantic rationale encoding in its validation strategies.

Deterministic Validation Guard

Rather than allowing LLMs to report findings based on plausibility or natural language reasoning alone, Archer’s validation guard enforces the following:

• Patch Triggering: Evidence must exercise the new or modified optimization path, distinguishing semantic discrepancies introduced by the patch from unrelated behaviors.
• Oracle-backed Validation: Candidate evidence is verified through tools such as Alive2 and LLUBI, combining bounded translation validation and UB-aware IR interpretation.
• Executable Evidence Reporting: Only findings with tested, patch-relevant semantic errors are output; unsupported or inconclusive cases are discarded.

Implementation

Archer is instantiated atop the mini-SWE-agent framework, using custom tool interfaces that mediate context retrieval, IR transformations, translation validation, and UB-aware differential testing. All review operations are confined to the local patch environment, with the agent denied internet access to prevent context drift or version mismatches.

Evaluation

Archer is benchmarked on extensive real-world and regression LLVM PR datasets. Notable results include:

• Real-World PRs (398 instances): Archer identified semantic bugs in 21% of open and 11% of closed recent LLVM optimization PRs, reporting 51 semantic bugs, of which 69% were already fixed and 23% were confirmed by LLVM developers. Only 6% of reports were false positives, and most false positives were explainable by external context or deprecation mismatches.
• Bug Types: The majority of bugs correspond to miscompilations, with a significant minority involving compiler crashes. Archer uncovers defects spanning a wide range of passes, including peephole and vectorization optimizations.

Figure 5: Tool call distributions during Archer’s agentic review process, emphasizing validation tool usage.

• Head-to-Head Comparisons: On a regression dataset of bisected LLVM PRs:
  • General LLM and agentic baselines (including mini-SWE-agent, Copilot, Codex, CodeRabbit, and Greptile) fail to surface semantic bugs at meaningful rates, typically generating verbose but irrelevant or unsupported comments.
  • Archer, configured with several foundation models (Gemini-3.1-Pro, Qwen3.5-Plus, DeepSeek-V3.2), consistently outperformed all other approaches; Gemini-3.1-Pro showed the best recall but even open models identified a substantial subset of bugs.
  • 

Figure 6: Overlap of semantic bugs found by Archer instantiated with different LLMs.

• Ablation Studies: Removing either obligations or deterministic validation severely degrades performance, with obligations proving critical for surfacing semantically non-local transformation bugs, and validation guards essential for filtering hallucinated or overfit findings.
• Obligation Construction Variants: Providing obligations as distilled, structured, and pass-specific (Archer’s default) outperforms both RAG-style patch retrieval and providing unprocessed bug-fix histories, offering higher bug-finding rates and much smaller, more efficient context footprints.

Practical and Theoretical Implications

Practically, Archer demonstrates the necessity of agentic workflows augmented with domain-specialized, dynamically constructed semantic guidance and strict, executable validation in compiler settings. The high bug rate in recently closed PRs underscores the real limitations of manual review, even in mature communities.

Theoretically, the Archer paradigm suggests a scalable path for LLM-based review agents in safety- and correctness-critical system domains beyond compilers. Constructing obligations from historical fixes and enforcing evidence-based reporting may generalize to other domains characterized by semantic complexity and review scarcity. The need for hybrid, multi-oracle validation (combining translation validation and UB-aware interpretation) is shown to be essential for semantic bug discovery.

Potential directions for future work include adaptive refinement of obligation databases (bootstrapped from failed or uncertain reviews) and optimizing agentic budget allocation driven by observed review trajectories.

Conclusion

Archer systematizes LLM-driven agentic review by combining pass-level semantic obligation guidance and deterministic validation, yielding strong, actionable results in the context of LLVM compiler optimization PRs. Its engineered workflow exposes critical review gaps left by manual inspection and general AI agents, and highlights necessary ingredients for deploying LLM systems in correctness-critical infrastructure code review. The approach’s extensibility suggests broad applicability for automated semantic review in complex software systems.