Understanding the Limits of Automated Evaluation for Code Review Bots in Practice

Abstract: Automated code review (ACR) bots are increasingly used in industrial software development to assist developers during pull request (PR) review. As adoption grows, a key challenge is how to evaluate the usefulness of bot-generated comments reliably and at scale. In practice, such evaluation often relies on developer actions and annotations that are shaped by contextual and organizational factors, complicating their use as objective ground truth. We examine the feasibility and limitations of automating the evaluation of LLM-powered ACR bots in an industrial setting. We analyze an industrial dataset from Beko comprising 2,604 bot-generated PR comments, each labeled by software engineers as fixed/wontFix. Two automated evaluation approaches, G-Eval and an LLM-as-a-Judge pipeline, are applied using both binary decisions and a 0-4 Likert-scale formulation, enabling a controlled comparison against developer-provided labels. Across Gemini-2.5-pro, GPT-4.1-mini, and GPT-5.2, both evaluation strategies achieve only moderate alignment with human labels. Agreement ratios range from approximately 0.44 to 0.62, with noticeable variation across models and between binary and Likert-scale formulations, indicating sensitivity to both model choice and evaluation design. Our findings highlight practical limitations in fully automating the evaluation of ACR bot comments in industrial contexts. Developer actions such as resolving or ignoring comments reflect not only comment quality, but also contextual constraints, prioritization decisions, and workflow dynamics that are difficult to capture through static artifacts. Insights from a follow-up interview with a software engineering director further corroborate that developer labeling behavior is strongly influenced by workflow pressures and organizational constraints, reinforcing the challenges of treating such signals as objective ground truth.

• The paper investigates the efficacy of rubric-guided and LLM-as-a-Judge methods in aligning automated evaluations with human labeling in industrial code reviews.
• It demonstrates that Likert scale evaluations, notably using GPT-4.1-mini, achieve moderate agreement with human labels while being sensitive to prompt and workflow factors.
• The study highlights that automated scores, though useful for heuristic triage, must be interpreted cautiously due to operational context and inherent subjectivity.

Automated Evaluation Limits for Industrial Code Review Bots

Motivation and Context

Automated code review (ACR) bots have become integral to modern software engineering workflows, particularly in industrial settings where large volumes of pull requests (PRs) must be reviewed quickly and effectively. Despite advancements in ML and LLM-driven comment generation, there remains a critical challenge: how to evaluate the practical usefulness of bot-generated review comments systematically and reliably. This paper investigates whether automated evaluation strategies—specifically rubric-guided and LLM-as-a-Judge methods—align with human-labeled "fixed"/"wontFix" outcomes in production settings, using an industrial dataset from Beko comprising 2,604 bot-generated PR comments (2604.24525).

Related Work and Evaluation Paradigms

Prior work largely focused on code review comment generation and operational impact, rarely addressing systematic evaluation at scale. Early ML models used reference metrics, but these proved insufficient for open-ended, context-sensitive tasks. Recent studies on LLM evaluation—such as G-Eval [liu-etal-2023-g], MT-Bench [zheng2023judging], and surveys [gao-etal-2025-LLM, he-et-al-LLM-as-a-Judge]—motivate reference-free, rubric-driven protocols. However, existing frameworks are seldom tailored for SE tasks, and industrial evaluation remains constrained by the subjectivity and organizational context of human labels.

Experimental Setup

The study employs two automated evaluators:

• G-Eval: Rubric-driven chain-of-thought LLM evaluation, scoring comments via a 0–4 Likert scale and binary rubric.
• LLM-as-a-Judge: Direct evaluator prompting, using the same inputs and rubric criteria.

All evaluations use identical PR context (title, description, diff, and bot comment) as input and compare predictions to developer-provided fixed/wontFix labels. Evaluations are conducted across three LLM backends (Gemini-2.5-pro, GPT-4.1-mini, GPT-5.2) under both binary and Likert mappings.

Figure 1: Illustration of multiple automated evaluators independently assessing the usefulness of ACR bot comments against human-labeled ground truth across PRs.

Quantitative Results

Automated evaluators achieved only moderate agreement with human labels—ratios ranging from 0.44 to 0.62—depending on model and rubric variant. Likert scale rubric consistently outperformed binary, yielding higher alignment via gradual thresholds. The best configuration (G-Eval Likert with GPT-4.1-mini) achieved a 0.76 F1, 0.66 precision, 0.90 recall, and -0.0590 MCC, indicating strong recall but weak class correlation under imbalance.

Performance varied across models and rubric design, highlighting sensitivity to both prompt style and backend. Notably, agreement ratios are not sufficient as a sole signal, given the confounding effects of organizational workflow, prioritization, and business constraints captured in human labeling.

Figure 2: Evaluation setup contrasting G-Eval and LLM-as-a-Judge methodologies under binary and Likert scales, and their agreement with human-labeled outcomes.

Practical and Theoretical Implications

Automated evaluation methods provide only partial, highly approximate proxies for human judgment in industrial code review. The study exposes core limitations:

• Subjectivity of Usefulness: Developer actions (fix/ignore) reflect not only comment quality, but workflow priorities and contextual pressures. Consequently, human-labeled outcomes cannot be reliably treated as objective ground truth.
• Rubric and Model Sensitivity: Both rubric design and LLM backend influence agreement, reinforcing the need for careful diagnostic validation and periodic reassessment in industrial deployment.
• False Positive Risk: Over-reliance on automated scores risks propagating low-quality or contextually irrelevant feedback, potentially eroding developer trust and increasing alert fatigue.
• Organizational Impact: Labels such as "wontFix" may denote operational outcomes (e.g., deadline-driven triage) rather than comment semantics, limiting the ceiling for agreement and utility in automation.

Practically, teams should treat automated scores as weak heuristics for triage and prioritization, not definitive signals for acceptance or dismissal. For researchers, results underscore the necessity for task-specific benchmarks and caution against direct mapping of rubric-weighted automated evaluation to human standards.

Industrial Interview Insights

A semi-structured interview with a software engineering director at Beko revealed key themes:

• Labels are operational artifacts, not pure quality judgments.
• Diff-centered evaluation limits the ability to capture developer reasoning, which often spans architecture and broader system constraints.
• Effective industrial evaluation is multi-dimensional, balancing risk, alert fatigue, and SDLC integration.
• Despite desire for continuous monitoring, systematic automated evaluation is rarely sustained without better automation.

These qualitative findings reinforce the quantitative diagnosis: workflow context and business process systematically shape evaluation signals in ways that automated methods cannot fully capture.

Limitations

Internal validity is threatened by contextual cues embedded in PR titles, subjectivity and ambiguity in developer ground truth, and rubric/prompt sensitivity effects. External validity is limited by single-organization dataset scope, particular LLM backends, and potential coupling effects when evaluator and generator share model family.

Future Directions

• Development of SE-specific, context-aware evaluation benchmarks that explicitly integrate workflow and operational constraints.
• Investigation of fine-tuned evaluators resistant to workflow label noise.
• Empirical studies on generalization across organizations, languages, and toolchains.
• Robustness analyses under partial PR diffs and missing context.

Conclusion

Automated evaluation approaches for ACR bot comments, whether rubric-driven or direct judge paradigms, demonstrate only moderate agreement with human-labeled outcomes in industrial settings. This underscores that workflow context, prioritization, and organizational dynamics inject confounding into human evaluation signals, inherently limiting the reliability of fully automated methods. Automated metrics remain best-suited for heuristic triage and large-scale monitoring, but must be interpreted with caution and supplemented by periodic human validation. Theoretical development of task-specific evaluation methods and empirical characterization of generalization boundaries are critical for future progress in AI-assisted SE evaluation (2604.24525).