CodeFuse-CR-Bench: Comprehensive Python CR Benchmark

• CodeFuse-CR-Bench is a comprehensive, repository-level benchmark for evaluating automated code review on real-world Python projects, emphasizing both semantic and formal review quality.
• It leverages a hybrid evaluation framework that combines rule-based checks with model-based judgments to assess review correctness, location accuracy, and semantic value.
• The benchmark comprises 601 curated instances from 70 Python projects across nine problem domains, bridging the gap between isolated tasks and holistic review analysis.

Searching arXiv for the benchmark and closely related work to ground the article. CodeFuse-CR-Bench is an end-to-end, comprehensiveness-aware benchmark for automated code review in real-world Python projects. It was introduced as the first comprehensiveness-aware benchmark for repository-level CR evaluation, and comprises 601 high-quality instances from 70 Python projects covering nine Pull-Request problem domains. Each instance provides rich, multi-faceted context, including the associated issue, PR details, and repository state, so that models can be evaluated on holistic, context-rich review behavior rather than on isolated sub-tasks or context-poor snapshots (Guo et al., 18 Sep 2025).

1. Design objective and problem setting

CodeFuse-CR-Bench was designed to bridge what its authors call the “reality gap” between existing code review evaluations and the holistic reasoning developers perform daily. The motivating claim is that automated code review is not adequately represented by narrow tasks such as comment generation in isolation, because real review requires understanding issue intent, PR rationale, repository context, defect location, and the quality of the proposed feedback.

The benchmark therefore targets repository-level code review rather than simplified defect spotting alone. Its task framing assumes that useful review output must satisfy at least two distinct criteria: it must be semantically valuable as review feedback, and it must also be correctly grounded in the file and line context of the PR. This is why the evaluation framework combines white-box rule-based checks with black-box model-based judgments rather than relying on text overlap alone (Guo et al., 18 Sep 2025).

A common misconception in this area is that fluent review comments are sufficient evidence of code review competence. CodeFuse-CR-Bench rejects that premise structurally: the benchmark is built so that a review can read plausibly while still failing location or formal-matching checks, and conversely can be well-positioned while remaining semantically weak. This dual perspective is central to its notion of “comprehensiveness-aware” evaluation.

2. Corpus construction and filtering pipeline

CodeFuse-CR-Bench consists of 601 high-quality CR task instances drawn from 70 actively maintained, popular Python repositories. Repository selection began from the top 1,000 starred Python repos on GitHub. To ensure maturity and standardized CR practices, the retained projects were required to have at least 1,500 merged PRs and at least one commit or PR in the past year, specifically from 2024-08-15 to 2025-08-15. Forks were excluded by requiring active PR counts, yielding 230 candidate repositories with an average of 21 k stars (Guo et al., 18 Sep 2025).

PR acquisition was performed by crawling all PRs merged into the default branch via the GitHub GraphQL API. Only PRs that reference at least one closing issue were retained. The rationale given is that merged status plus issue linkage strongly correlates with high-quality, deliberate CR and clear development intent.

Each PR may contain many commits and many review comments, so the benchmark distills these to a single representative CR task instance per PR by weighted heuristic scoring of commits. High-impact rules with weight $3.0$ are has_resolved_review_comments and has_referenced_line_changed_comments. Medium-impact rules with weight $2.0$ are clear_commit_message, conventional_commit format, reasonable_commit_size, and has_associated_review_comments. Low-impact rules with weight $1.0$ are issue_reference, semantic_commit_message, focused_file_changes, and descriptive_commit_content. The top-scoring commit per PR is selected, and only those review comments whose referenced lines were actually changed or whose threads were explicitly marked “resolved” are extracted.

Noise filtering then combines automatic and manual stages. Two authors with at least 5 years of Python development experience manually screened 40 124 auto-filtered instances. They removed reviews that are purely “LGTM,” “Thanks,” or trivial formatting only; comments asking only for documentation/tests without code logic changes; instances with external hyperlinks, images, or references to other PRs/SHAs; and problem statements shorter than 40 words. Remaining instances were assessed via a questionnaire on alignment, coverage, defect identification, difficulty, risk, and annotator confidence. The final 601 instances cover all nine problem domains and a balanced distribution of difficulty and effort (Guo et al., 18 Sep 2025).

3. Instance schema, context fields, and annotation dimensions

Each CodeFuse-CR-Bench instance is a self-contained snapshot of a real PR and its review cycle. The benchmark provides 22 structured fields organized into four categories: Basic Info, PR-Related, CR-Related, and Repository-Level Context. Basic Info includes Instance ID, Owner/Repo, and Language. PR-Related fields include Pull No., Title, Created at, Base Commit, Body, Issue Problem Statement, Hint Text, Resolved Issue No., Commit Patch to Review, Head Commit, Head Commit Message, Problem Domain, and Difficulty. CR-Related fields include Review Comment Text, Diff Hunk, Review Path, and Review Effort. Repository-Level Context includes Merge Commit and Merge Commit Patch (Guo et al., 18 Sep 2025).

The benchmark’s problem-domain labeling spans nine categories: Bug Fixes, New Feature Additions, Code Refactoring/Architectural Improvement, Documentation Update, Test Suite/CI Enhancements, Performance Optimizations, Security Patches/Vulnerability Fixes, Dependency Updates/Environment Compatibility, and Code Style, Linting, Formatting Fixes. Difficulty is recorded as low, medium, or high, and Review Effort on a 1–5 scale. Where repositories already provided these annotations, they were used directly; otherwise an LLM-as-a-judge, Qwen3-235B-A22B, was employed in a few-shot classification prompt over PR title, body, patch, and commit message.

The significance of this schema lies in the benchmark’s insistence that real code review requires more than diff-local reasoning. The benchmark explicitly argues that review quality depends on understanding the issue and PR intent, locating defects within diffs, reasoning about cross-file and project-wide state, and writing actionable, correctly positioned suggestions. This suggests that CodeFuse-CR-Bench is not merely a dataset of review comments, but a structured testbed for repository-grounded reasoning over the full review context.

4. Hybrid evaluation framework

CodeFuse-CR-Bench uses a hybrid evaluation framework composed of model-based judgments and rule-based checks. The model-based component has two parts. The first is a reward model built on Qwen3-8B with a LoRA head of rank $32$ and $\alpha = 16$, fine-tuned on 174 661 positive and 114 458 negative CR examples from external repositories with no data leakage with test repos. Its training loss is

$$\mathcal{L}_{\mathrm{BPR}} = -\log \sigma\!\left(r_\theta(x,y^+) - r_\theta(x,y^-)\right),$$

$$\mathcal{L}_{\mathrm{BCE}} = -\sum \left[l_i \log p_i + (1-l_i)\log(1-p_i)\right], \quad p_i=\sigma(r_\theta(x_i,y_i)),$$

and

$$\mathcal{L}_{\mathrm{total}} = \mathcal{L}_{\mathrm{BPR}} + \lambda \mathcal{L}_{\mathrm{BCE}}, \quad \lambda=1.$$

At evaluation time, $r_\theta(\text{query}, \text{review}) \in (0,1)$ is used as the reward score. The second model-based component is an LLM-as-a-judge using OpenAI o3-2025-04-16. It evaluates review usefulness from four perspectives—Functionality, Quality, Style, and Documentation—and rates Correctness, Relevance, Clarity, Consistency, and Language on a 1–5 scale. The model-based score is the average of the reward model score and the LLM judge score (Guo et al., 18 Sep 2025).

The rule-based component is intended to measure formal correctness of defect location and the surface similarity of comments. Location similarity is computed from File Path Match, Line Number Accuracy, and Diff-Hunk Similarity. The weighted composite is

$$\mathrm{LocationScore} = 0.70 \cdot \mathrm{PathMatch} + 0.15 \cdot \mathrm{Acc\_LN} + 0.15 \cdot \mathrm{HunkSim}.$$

For line number accuracy,

$2.0$0

Semantics similarity is measured by BLEU-4,

$2.0$1

with

$2.0$2

Defect-Match then computes a sub-match score as the average of LocationScore and BLEU for each predicted defect, and forms a DefectScore from the average of $2.0$3 and the mean sub-match score. The rule-based score is the average of LocationScore, BLEU, and DefectScore.

The final comprehensive score is

$2.0$4

with $2.0$5 in the reported experiments. The benchmark therefore operationalizes code review quality as a balanced combination of semantic utility and formal grounding rather than as a single scalar notion of textual similarity.

5. Baselines and empirical findings

The baseline study evaluates seven LLMs under an oracle-based context consisting of the full diff together with merge commit files, and also compares retrieval strategies based on BM25 top-$2.0$6. The reported comprehensive scores establish an initial performance landscape rather than a solved leaderboard (Guo et al., 18 Sep 2025).

LLM	Model-Based	Rule-Based	Comprehensive
DeepSeek-v3.1	58.69	28.34	42.51
Kimi-K2-0905	62.11	20.81	46.77
Qwen3-235B	58.10	24.30	40.45
Claude-Sonnet-4	60.67	33.31	47.46
Gemini 2.5 Pro	63.65	29.47	52.37
GPT-4o	54.57	8.10	35.47
GPT-5	64.80	18.30	41.96

Gemini 2.5 Pro achieves the highest comprehensive score, $2.0$7, balancing strong model-based quality with solid rule-based performance. GPT-5 attains the highest model-based score, $2.0$8, but under-performs on rule-based matching with $2.0$9. Claude-Sonnet-4 has the strongest rule-based result, $1.0$0, while GPT-4o’s rule-based score of $1.0$1 indicates substantial problems with comment placement or formal correctness.

Performance by problem domain reinforces this picture. For Gemini 2.5 Pro, the comprehensive scores are $1.0$2 for Bug Fixes, $1.0$3 for New Feature Additions, $1.0$4 for Code Refactoring/Architectural Improvement, $1.0$5 for Documentation Update, $1.0$6 for Test Suite/CI Enhancements, $1.0$7 for Performance Optimizations, $1.0$8 for Security Patches/Vulnerability Fixes, $1.0$9 for Dependency Updates/Environment Compatibility, and $32$0 for Code Style, Linting, Formatting Fixes. The benchmark reports that Gemini leads by 5–28 points over the next best in every domain.

Context-retrieval sensitivity is comparatively mild for Gemini 2.5 Pro. Its score is $32$1 under oracle context, $32$2 with BM25-Top1, $32$3 with BM25-Top3, and $32$4 with BM25-Top5. This near-oracle performance under BM25-Top1 is used to argue for strong context efficiency, whereas GPT-5 and GPT-4o are reported to fluctuate more with context size, suggesting sensitivity to noise or redundancy in larger retrieval sets.

The reward model validation further supports the hybrid evaluation design. On positive/negative review classification, the fine-tuned reward model with file context achieves Accuracy $32$5, Precision $32$6, Recall $32$7, and $32$8 $32$9, outperforming Kimi-K2-0711 Preview and Gemini 2.5 Pro on that classification task. This suggests that specialized evaluators can be more reliable than general-purpose LLMs when scoring review relevance.

6. Relation to adjacent benchmarks, limitations, and terminological overlap

CodeFuse-CR-Bench occupies a distinct position among recent code review benchmarks. A closely related comparison point is CR-Bench, which is constructed by transforming SWE-Bench into a defect-focused code review benchmark and pairs its dataset with CR-Evaluator, an LLM-as-judge pipeline that classifies each agent comment as BUG_HIT, VALID_SUGGESTION, or NOISE. CR-Bench reports two complementary corpora—584 PR tasks in the standard set and a manually vetted subset of 174 high-fidelity instances—and emphasizes the trade-off between issue resolution and spurious findings through metrics such as Recall, Precision, Usefulness Rate, and Signal-to-Noise Ratio (Pereira et al., 10 Mar 2026).

By contrast, CodeFuse-CR-Bench is explicitly presented as repository-level and comprehensiveness-aware. Its task instances contain issue, PR, diff-hunk, and merge-state context, and its evaluation combines rule-based location checks with model-based judgments of review quality. This suggests that the two benchmarks target different slices of the code review problem: CR-Bench concentrates on defect-identifying reviews and developer-facing noise, whereas CodeFuse-CR-Bench evaluates broader end-to-end review competence over richer repository context.

The benchmark also has explicit limitations. It is currently Python-only; extending to Java, C++, and multi-language repos is identified as a priority. Its nine problem domains cover most PR types, but additional categories such as UX changes and infrastructure could be added. The authors note prompt and hyperparameter sensitivity, limited model coverage due to GPU/time constraints, and the possibility that richer context in reward-model training—such as cross-file diffs and test results—could further improve semantic alignment (Guo et al., 18 Sep 2025).

A terminological caveat is necessary because the provided literature also uses closely related naming for a different benchmark line. One description refers to “CodeFuse-CR-Bench (a.k.a. CodeFuse-DeBench)” in the context of binary decompilation evaluation, where the benchmark contains 240 atomic C functions organized into 8 source files and compiled into 640 binaries, with a three-axis evaluation over readability, recompilability, and functionality (Liu et al., 28 May 2026). This suggests a naming overlap rather than a single unified benchmark family. In current usage, “CodeFuse-CR-Bench” most directly denotes the repository-level Python code review benchmark, while “CodeFuse-DeBench” denotes the multidimensional binary decompilation benchmark.