Automated Classification of Human Code Review Comments with Large Language Models

Abstract: Context: Code reviews are essential for maintaining software quality, yet many human review comments suffer from issues such as redundancy, vagueness, or lack of constructiveness. These types of comments may slow down feedback and obscure important insights. Prior work on code review comments mostly explore the detection and categorization of useful comments, while fine-grained categorization of comment issues remains underexplored. Objective: This work aims to design and evaluate an automated system for classifying code review comments according to specific categories of issues. Methodology: We introduced a nine-label taxonomy for code review comments, covering six review comment smells and three common useful intents, and manually labeled 448 comments from a publicly available dataset. We benchmarked zero-shot and one-shot single-label classification over each comment and its associated unified diff hunk, comparing GPT-5-mini, LLaMA-3.3, and DeepSeek-R1. We reported macro-F1 as the primary metric. Results: Zero-shot performance was moderate under class imbalance (macro-F1 0.360 to 0.374). One-shot exemplar conditioning had model-dependent effects: GPT-5-mini and DeepSeek-R1 macro-F1 scores improved, however LLaMA-3.3 suffered a slight decrease. Exemplars most consistently helped intent-boundary labels, whereas classification of evidence-sensitive labels remain challenging. Conclusion: Our results indicate that comment--diff evidence is sufficient for some labels but limited for evidence-sensitive smells. Future work includes adding thread context, improving intent-preserving rewrites, and validating robustness across platforms.

• The paper introduces a novel nine-label taxonomy to differentiate high-value review comments from problematic (smell) ones.
• It evaluates LLMs (GPT-5-mini, LLaMA-3.3-70B-Instruct, DeepSeek-R1) under zero-shot and one-shot prompting, using metrics like macro-F1 and accuracy.
• Empirical results highlight strong performance for clear, surface-level cues, while revealing challenges in detecting evidence-sensitive and context-dependent smells.

Automated Classification of Human Code Review Comments Using LLMs

Overview and Motivation

This paper investigates automated classification of human code review comments using LLMs, introducing a fine-grained taxonomy for distinguishing "smell" (i.e., problematic or low-value) comments from useful subclasses. The authors address two central challenges: the lack of systematic taxonomies for comment-level review smells and the absence of curated datasets labeled under such taxonomies. They operationalize a nine-label taxonomy encompassing six distinct review comment smells—Incorrect, Toxic, Unrelated, Vague, Redundant, and Praise—alongside three useful subclasses encapsulating high-value review intents. The system is evaluated on 448 manually labeled comment–diff pairs extracted from a prior OpenDev Nova corpus.

LLM-based classification is benchmarked in both zero-shot and one-shot settings, employing GPT-5-mini, LLaMA-3.3-70B-Instruct, and DeepSeek-R1. Macro-F1, accuracy, and Matthews Correlation Coefficient serve as primary evaluation metrics to account for significant label imbalance and to expose per-category model behavior.

Figure 1: High-level depiction of the end-to-end methodology: taxonomy definition, dataset curation, manual annotation, LLM evaluation, and result analysis.

Taxonomy and Dataset Construction

The taxonomy is derived via a focused review of the literature on code review comment usefulness, reviewer intent, and communication patterns. Smell categories are justified through prior studies and operationalized to be mutually exclusive and annotation-practical. While the majority of past work coarsely dichotomizes comments (useful vs. non-useful), this taxonomy decomposes quality issues into explicit types, aiming to facilitate systematic detection and eventual automated repair.

The dataset construction leverages the OpenDev Nova review corpus, selecting all candidate comments labeled as False Positive or Praise (smells), and balancing with a random sample of non-smell comments to enforce equal representation. Manual labeling is performed in two stages with calibration, independent double annotation, and adjudicated conflict resolution. The final set comprises 448 comment–diff pairs, each assigned a single label.

Figure 2: Example of an Incorrect review comment where the reviewer mistakenly identifies an error, illustrating the semantic complexity involved in evidence-sensitive smell detection.

Evaluation Setup

Three LLMs are selected to cover both proprietary and open-weight instruction-tuned architectures: GPT-5-mini (API model optimized for instruction following), LLaMA-3.3-70B-Instruct (publicly available, strong on code and natural language), and DeepSeek-R1 (emphasizing semantic reasoning). Classification is performed by presenting the model with the entire comment and its associated diff hunk, followed by schema-constrained single-label output. Two prompting configurations are evaluated: zero-shot (no exemplars, only label definitions) and one-shot (taxonomy plus a representative exemplar per category).

Prompts are specifically designed to minimize ambiguity, maximize output regularity, and facilitate direct comparison between models. Consistent temperature settings are used wherever feasible.

Empirical Results

Aggregate Metrics

Under zero-shot prompting, macro-F1 ranges from 0.360 to 0.374, with overall accuracy between 0.597 and 0.626 across models. One-shot prompting induces model-dependent effects: GPT-5-mini and DeepSeek-R1 gain in macro-F1 (e.g., GPT-5-mini increases to 0.409), while LLaMA-3.3-70B-Instruct experiences a slight degradation (to 0.344).

High-frequency literal labels (Actionable, Praise) are classified with strong F1 (e.g., Praise F1 up to 0.882 zero-shot); in contrast, evidence-sensitive or context-dependent smells—Incorrect, Redundant, Vague—contribute to poor recall and F1, revealing the limitations of local context. Incorrect, in particular, is almost never detected correctly in zero-shot.

Per-Category Analysis

• Actionable and Praise: Reliably separated (F1 0.69+), reflecting surface-level linguistic cues and code-anchoring.
• Clarification and Question: Moderately distinguishable (F1 0.51–0.67), with clear improvement under one-shot for some models.
• Evidence-sensitive smells (Incorrect, Redundant, Vague): Persistently low F1, especially for Incorrect (near-zero detection). Models rarely reject reviewer claims based on code semantics alone, indicating fundamental limits of comment–diff evidence.

Figure 3: Example of a Clarification comment where the review adds context pertinent to deleted code—an archetype highlighting the distinction between codebase documentation and review-thread rationale.

Discussion and Implications

Theoretical and Methodological Insights

• Evidence-sensitive classification remains an open problem: For smells such as Incorrect, the lack of broader context (thread, project history, bidirectional reviewer–author interactions) fundamentally constrains model verification. Even state-of-the-art LLMs default to interpreting reviewer assertions as valid unless contradicted.
• Exemplar value is model- and label-specific: While one-shot prompting boosts discriminative performance for some boundaries (e.g., Question vs. Actionable), it can degrade it for others, suggesting non-trivial interactions between prior knowledge, prompt design, and label semantics.
• Binary triage shows moderate utility: At the "smell vs. not smell" level, binary F1 reaches 0.623 for the best configuration, with relatively low false-positive rates. This supports triage applications where detection sensitivity can be traded for precision.

Practical Implications and Future Directions

• Training policies should treat Incorrect comments explicitly as negative exemplars: Automated systems must avoid replicating reviewer misconceptions.
• Detection systems can drive reviewer education: Taxonomy-aligned detectors can provide private, actionable feedback to junior reviewers or serve as formative metrics for onboarding.
• Repair and rewriting require additional context: Future LLM-based repair pipelines must incorporate thread-level scope, trace cross-artifact links (e.g., code comments answered by review thread), and distinguish between in-code clarifications vs. review-only rationale.
• Evaluation protocols should explicitly model context drift, author role, and intent evolution: This is inevitable in real-world review settings where comment authorship (bot/human) and iterative edits alter effective communicative function.

Limitations

Dataset size, singular project focus (OpenDev Nova, Python), and construction via re-labeling may limit generalizability. Annotator subjectivity is mitigated by calibration and adjudication but cannot be eliminated. API sampling stochasticity may marginally affect reproducibility, as temperature control is unavailable for one model.

Conclusion

This work provides a critical empirical foundation for comment-level code review smell detection, demonstrating the strengths and limits of current LLMs under shallow-to-moderate prompting. Comment–diff context is sufficient for detecting literal or high-frequency review intents but inadequate for evidence-sensitive or referential smells without thread-level augmentation. The curated taxonomy and dataset constitute a reusable benchmark for future work on fine-grained review comment analysis and repair, and highlight open challenges at the intersection of automated analysis, reviewer intent recovery, and tool-supported code review process optimization.

For further technical detail and follow-up studies leveraging this taxonomy and dataset, see "Automated Classification of Human Code Review Comments with LLMs" (2604.23667).