RevMate: LLM Code Review Assistant

• RevMate is an LLM-based assistant for modern code reviews that integrates automated, context-aware comment suggestions using RAG and LLM filtering.
• It utilizes a dual-mode pipeline with historical code chunk retrieval and contextual prompt engineering to generate and refine review comments.
• Empirical studies in open- and closed-source environments indicate that RevMate’s suggestions achieve comparable development impact to human reviews.

RevMate is an LLM-based assistant for modern code review environments, designed to generate, filter, and assist in the integration of automated review comments within established workflows. Leveraging Retrieval-Augmented Generation (RAG) and an “LLM-as-a-Judge” paradigm, RevMate grounds its comment suggestions in project-specific history and automates the rejection of irrelevant outputs. Its effectiveness was assessed through a large-scale empirical user study in both open-source (Mozilla) and closed-source (Ubisoft) organizations, offering the first comprehensive measurement of LLM-generated review comment acceptance and influence on subsequent development activity (Olewicki et al., 2024).

1. Architectural Design

RevMate's architecture is predicated upon RAG techniques and an LLM-based post-generation filter:

• Retrieval-Augmented Generation (RAG): RevMate constructs a vector database (Qdrant) of “chunk ⇔ comment” pairs, extracted from historical review data. Each code chunk and its corresponding human-written comment are embedded using OpenAI's text-embedding-3-large model ($d=1024$) at a low temperature setting of 0.2 for stability. During each review, the model retrieves the top-10 most similar historical examples for the changed code chunk via cosine similarity. There are two RAG operating modes:
  • Example Variant: Retrieved examples are introduced as few-shot prompts for the base LLM.
  • Code Variant: The LLM is queried regarding which code functions or additional line contexts are needed, with these contexts fetched from the repository (via rust-code-analysis).
• Context Window Construction: RevMate maintains a chain-of-thought memory buffer with the following structure:
  1. Reformatted patch (file-by-file diffs, explicit line numbers).
  2. Summary of the added lines (generated by GPT4o).
  3. Retrieval of additional function/line contexts (as needed, via “askLLMNeedsFuncs” and “askLLMNeedsLines”).
  4. Aggregated context including persona (expert reviewer), summary, retrieved code, and/or few-shot examples.
  5. Final review comment generation, with the LLM instructed via “askLLMReview” to produce comments in JSON format.

• LLM-as-a-Judge: A secondary GPT4o invocation, “askLLMFilter,” acts as a filter over the generated suggestions. Provided with the patch, candidate comments, and a curated catalog of “undesired” comment types (e.g., inconsistent, non-actionable, or descriptive), the LLM discards suggestions failing actionable or contextual relevance. This filtering process is entirely prompt-based, with no supplemental numeric thresholds or explicit reranking applied.

2. Implementation and Workflow Integration

RevMate is integrated into mainstream code review platforms, with implementation particulars as follows:

• Base Models and Embedding: Both generation and filtering utilize GPT4o (temperature 0.2) and embeddings are produced by text-embedding-3-large.
• Token Management: OpenAI's internal tokenizer ensures that concatenated patches and contexts do not exceed GPT4o’s context window (~128k tokens).
• Caching & UI: Generated suggestions are cached per patch for UI consistency. Comments are parsed from JSON and presented within respective review platforms.
• Workflow Customization:
  • Mozilla (Phabricator): RevMate acts as a reviewer, directly proposing generated comments under its own identity. Human reviewers must explicitly assess each suggestion.
  • Ubisoft (Swarm): Markers (“★”) are inserted alongside code lines, only revealed upon reviewer interaction. Accepted comments are attributed to the accepting human reviewer.
• Post-Processing: No reranking beyond LLM-filtering. System state is synchronized across reloads to maintain reviewer context.

3. Empirical Evaluation Protocol

The effectiveness of RevMate was validated through a controlled, mixed open-/closed-source study:

• Participants: 59 expert reviewers (Mozilla: 28; Ubisoft: 31), average tenure 7–13 years.
• Experimental Design: 587 patch reviews (Mozilla: 165; Ubisoft: 422) over 5–6 weeks, with participants randomly assigned to either “Code” or “Example” RAG variants. Each reviewer assessed every suggestion provided within their workflow environment.
• Collected Metrics:
  • Acceptance Rate: $\displaystyle \frac{\#Accepted}{\#Evaluated}$
  • Appreciation Rate: $$\displaystyle \frac{\#Accepted+\#ValuableTip}{\#Evaluated}$$
  • Time Overhead: Measured time per suggestion and per patch.
  • Revision Impact: Analyzed for each accepted comment as to whether it prompted a line-level or chunk-level change, or initiated a discussion thread.
  • Automated Comment Categorization: Generated comments embedded, clustered (k-means, $k=400$), and categorized (LLM labeling) with cross-referencing to Turzó & Bosu’s taxonomy (Functional, Refactoring, Documentation, Discussion). Cohen’s $\kappa = 0.45$ indicated moderate annotation agreement.

4. Quantitative Results

The study produced a detailed breakdown of RevMate’s performance:

Metric	Mozilla	Ubisoft
Acceptance Rate	8.1%	7.2%
Appreciation Rate (“valuable tips”)	23.0%	28.3%

• RAG Variant Performance (Ubisoft): “Example” outperformed “Code” in acceptance (8.9% vs 5.3%, $p=.011$, small Cohen’s $d$). Conversely, “Code” improved appreciation (Mozilla: 29.9% vs 19.9%, $p = .022$; Ubisoft: 32.2% vs 24.5%, $p = .002$).
• Comment Type Acceptance: Functional comments constituted ~80% of suggestions (acceptance ≈ 5%), whereas Refactoring comments comprised ~15% (acceptance ≈ 18%). Acceptance rate difference was highly significant ($p < .002$, large effect).
• Time Overhead (Ubisoft): Median evaluation time was 27.6s for accepted and 16.0s for ignored suggestions; median triage time per patch was 43s (95% CI: 5s to 5m12s).
• Impact on Code Revisions: Accepted LLM comments triggered line-level changes in 62.3% of cases (vs. 64.3% for human), chunk-level in 73.9% (vs. 73.2%), with minimal statistical difference. However, follow-up discussion threads occurred less often with LLM suggestions (23.2%) than with human comments (33.9%, $\displaystyle \frac{\#Accepted}{\#Evaluated}$0, medium effect).

5. Comparative Analysis and Key Findings

• RevMate’s LLM-generated comments, despite modest acceptance (~8%), were valued as review or development tips considerably more often (23–28%).
• Refactoring-focused suggestions were 3–4x more likely to be accepted than Functional comments.
• Reviewers invested a median of 43s per patch on RevMate outputs, viewed as a reasonable overhead.
• LLM-generated comments led to follow-up code revisions at rates similar to those produced by human reviewers (≈ 74% chunk-level change), indicating equivalent actionable impact.
• Fewer off-topic discussion threads were associated with LLM-generated comments, consistent with explicit filtering of non-actionable or descriptive outputs.

6. Current Limitations and Prospective Directions

• RevMate currently supports only distinct “Code” and “Example” RAG modes; integrating both retrieval paradigms could increase precision.
• Filtering relies exclusively on prompt-driven LLM judgment, with no hard scoring functions; future work may explore adding numeric heuristics or fine-tuning judge models.
• External validation on other LLMs (e.g., StarCoder, LLaMA 2) or on-premise/self-hosted deployments is pending.
• Feedback suggests the utility of distinguishing “developer tips” (pre-review), “reviewer tips,” and “publishable comments” via multi-stage prompt classification.
• Broader field evaluations and cost-benefit analysis between fine-tuning and retrieval-based prompting remain open research trajectories (Olewicki et al., 2024).