Code Change Quality Estimation

• Code change quality estimation is a set of techniques that predict the quality of modifications based on diverse metrics like correctness, maintainability, and readability.
• Methodologies range from LLM uncertainty sampling and validation frameworks to supervised feature-driven classifiers that combine semantic and process signals.
• Practical applications include integrating these estimations in CI/CD pipelines to enhance error detection, streamline review processes, and guide maintenance strategies.

Code change quality estimation is the family of techniques that infer the likely quality of a code modification—commit, diff, patch, pull request, method revision, or associated review artifact—before, during, or shortly after integration. In the literature, “quality” is not a single latent variable but a collection of operational targets: functional correctness, maintainability-oriented issue presence, readability, defect risk, maintenance intent, review clarity, merge viability, co-change completeness, and effort. Consequently, the field spans uncertainty estimation for LLM-generated code, industrial multi-agent review systems, metric- and history-based classifiers, review-activity predictors, learning-to-rank models for evolutionary coupling, and commit-level quality-improvement detectors (Sharma et al., 17 Feb 2025, Wong et al., 1 Aug 2025, Hönel et al., 2020, Olewicki et al., 2024, Almukhtar et al., 4 Nov 2025).

1. Operational definitions of quality

The first technical fact about code change quality estimation is that the target variable is highly task-dependent. Some works estimate whether a generated program is functionally correct without executing an external oracle; some estimate whether a diff contains valid maintainability issues; some predict whether a patch will attract comments, require revision, be merged, or later be associated with bugs. Others model whether a commit improves particular quality attributes such as Understandability, Reliability, Maintainability, Usability, and Modularity, or whether the review text itself is concise, comprehensive, and relevant.

This plurality is substantive rather than merely terminological. A method that is well calibrated for selective acceptance of LLM-generated patches is not necessarily suitable for ranking co-changed methods in a pull request, and a model trained on review-comment clarity is not equivalent to a defect predictor. Several papers therefore define quality at different granularity levels—method, file, diff, commit, pull request, or review comment—and expose either scalar scores, binary labels, ranked lists, or structured issue sets.

Operationalization	Unit and output	Representative papers
Functional correctness proxy	Prompt or patch family; uncertainty score, abstention decision	(Sharma et al., 17 Feb 2025)
Validated maintainability issues	Diff; set of scored issues with function, rationale, file, line, tag	(Wong et al., 1 Aug 2025)
Subjective code quality / readability	Method or class; binary label or 0–100 readability score	(Mahamud et al., 2023, Simoes et al., 5 Jul 2025)
Maintenance purpose as quality proxy	Commit; adaptive/corrective/perfective or perfective/corrective/other	(Hönel et al., 2020, Trautsch et al., 2021)
Review quality	Review comment; confidence score or conciseness/comprehensiveness/relevance	(Mahbub et al., 2023, Naik et al., 2024)
Review hot-spot / process viability	File or change; commented/revised/hot-spot or merged/abandoned	(Olewicki et al., 2024, Islam et al., 2019)
Change completeness / coupling risk	Query method; ranked co-changed methods	(Jia et al., 2024)
Effort as auxiliary signal	Commit; Standard Coding Hours	(Wright et al., 2019)
Multi-QA enhancement in Python MLS	Commit; Enhanced vs Not Enhanced for UN/RE/MA/US/MO	(Almukhtar et al., 4 Nov 2025)

2. Methodological families

One major family estimates quality from a model’s own uncertainty. In LLM-based code generation, multiple responses are sampled for the same prompt, clustered by semantic equivalence via bounded symbolic execution, and converted into semantic entropy or mutual-information scores. Low entropy indicates concentration in one or a few semantic clusters; high entropy indicates dispersed behaviors. The same machinery supports selective prediction: accept when uncertainty is below a threshold and abstain otherwise. This converts uncertainty into a correctness proxy and, in effect, into a code-change quality gate (Sharma et al., 17 Feb 2025).

A second family uses LLMs as evaluators or validators rather than primary generators. In the CQS system, an Issue Collector proposes tagged diff-level issues and a separately fine-tuned LLM-Judge scores validity and severity, after which hand-crafted filters suppress hallucinations and known failure modes. CORE uses a proposer LLM to generate candidate revisions for static-analysis warnings, retains only candidates that pass the static checks, and then applies a ranker LLM with an explicit ordinal rubric ranging from Strong Reject to Strong Accept. CRScore evaluates review comments rather than patches: it generates pseudo-references from the code change itself—claims, implications, and analyzer-detected smells—and then scores review conciseness, comprehensiveness, and relevance through semantic matching against those grounded pseudo-references (Wong et al., 1 Aug 2025, Wadhwa et al., 2023, Naik et al., 2024).

A third family remains fundamentally supervised and feature-driven. Examples include CodeBERT-based classifiers for subjective code quality, seBERT-based commit-intent classification, LightGBM prediction of merged versus abandoned changes, Random Forest and Naive Bayes models for file-level review hot-spots, k-means with cosine similarity over change metrics followed by expert mapping to change classes, and Random Forest learning-to-rank models for co-changed methods at the pull-request level (Mahamud et al., 2023, Trautsch et al., 2021, Islam et al., 2019, Olewicki et al., 2024, Kniazev, 16 Feb 2026, Jia et al., 2024). These methods typically separate representation engineering from prediction, and they remain competitive because many operational targets in this area are structured, imbalanced, and richly contextual rather than purely generative.

3. Signals, features, and scoring formulations

The predictive signals used in code change quality estimation fall into several recurrent groups. One group is direct semantic or behavioral evidence extracted from the changed code itself: token log-probabilities, symbolic-execution traces, semantically clustered response distributions, static-analysis findings, and metric deltas between pre- and post-change states. Another group is process evidence: gross and net change size, source code density, reviewer and author histories, directory and project histories, co-change frequencies, and developer experience. A third group is textual or embedding-based evidence derived from commit messages, diffs, or code representations (Hönel et al., 2020, Islam et al., 2019, Olewicki et al., 2024, Jia et al., 2024).

Several papers define explicit scoring rules. ReviewRanker computes three probabilities $C_{i1}, C_{i2}, C_{i3}$ corresponding to operation type, insertion understanding, and deletion understanding for a review $R_i$, then aggregates them as

$$C_i = \bigl(C_{i1} \times C_{i2} \times C_{i3}\bigr)^{1/3}.$$

CRScore defines sentence-level matching between review text and pseudo-references using cosine similarity and a threshold $\tau$, then computes conciseness, comprehensiveness, and relevance as precision-like, recall-like, and harmonic-mean quantities, respectively. These formulations are important because they expose review quality estimation as a structured matching problem rather than mere text similarity to a single human reference (Mahbub et al., 2023, Naik et al., 2024).

Metric-delta modeling is especially explicit in PyQu. There, each quality attribute is predicted from pre/post differences in low-level metrics such as $\Delta CC$, $\Delta HV$, $\Delta LoC$, $\Delta CH$, $\Delta CP$, $\Delta TC$, $R_i$0, and ML-specific conformance signals. By contrast, source code density defines change density at commit level as

$R_i$1

thereby separating potentially functional change from whitespace- and comment-dominated churn. At the other end of the abstraction spectrum, Standard Coding Hours estimate the effort of a change from its diff features through a learned “standard coder,” yielding a scalar effort signal that is not itself a quality score but is directly usable as an auxiliary feature in risk and maintainability models (Almukhtar et al., 4 Nov 2025, Hönel et al., 2020, Wright et al., 2019).

Label quality is itself a technical issue. CAT analysis shows that refactoring and refactoring propagation confound bug-inducing labels in just-in-time defect prediction, because SZZ-style blame can attribute defects to behavior-preserving moves or renames. CAT therefore assigns fine-grained line categories such as Add_Refactoring, Add_Propagation, and Add_Refactoring_Edit, then re-traces blame past pure refactoring-only edits. This makes quality estimation depend not only on model architecture but also on whether the supervision signal has been disentangled from refactoring noise (Niu et al., 25 Jul 2025).

4. Empirical behavior and deployment

In LLM-based correctness estimation, uncertainty has been shown to correlate negatively with functional correctness on LiveCodeBench, where correctness is measured as percentage of tests passed and sometimes binarized at a $R_i$2 pass threshold. Entropy-based semantic clustering outperformed a baseline that used only LLM-reported probabilities, and a tuned abstention policy reduced false positives to $R_i$3 with accuracy $R_i$4 for SE-CodeGen-Norm, whereas the log-probability baseline had accuracy $R_i$5 and false positives $R_i$6 (Sharma et al., 17 Feb 2025).

Industrial diff-level quality estimation emphasizes precision-oriented deployment. CQS reports that the collector-only system achieved precision $R_i$7 and recall $R_i$8, while the full collector+validator+filtering system reached precision $R_i$9 and recall $$C_i = \bigl(C_{i1} \times C_{i2} \times C_{i3}\bigr)^{1/3}.$$0, reflecting a deliberate shift toward few but trusted findings. The system has been deployed to more than 5,000 engineers and sustained about $$C_i = \bigl(C_{i1} \times C_{i2} \times C_{i3}\bigr)^{1/3}.$$1 week-over-week user helpfulness, which functions as an online quality metric for surfaced issues rather than for the underlying diffs alone (Wong et al., 1 Aug 2025).

Process- and history-based estimators remain strong in deployment-style settings. Source code density and parent-generation features enable automatic maintenance classification with up to $$C_i = \bigl(C_{i1} \times C_{i2} \times C_{i3}\bigr)^{1/3}.$$2 cross-project accuracy and Kappa $$C_i = \bigl(C_{i1} \times C_{i2} \times C_{i3}\bigr)^{1/3}.$$3, and up to $$C_i = \bigl(C_{i1} \times C_{i2} \times C_{i3}\bigr)^{1/3}.$$4 accuracy with Kappa approaching $$C_i = \bigl(C_{i1} \times C_{i2} \times C_{i3}\bigr)^{1/3}.$$5 in single-project settings. PredCR reaches about $$C_i = \bigl(C_{i1} \times C_{i2} \times C_{i3}\bigr)^{1/3}.$$6 AUC for early merged-vs-abandoned prediction and ER@20% near $$C_i = \bigl(C_{i1} \times C_{i2} \times C_{i3}\bigr)^{1/3}.$$7. File-level review-activity prediction improves over prior work by $$C_i = \bigl(C_{i1} \times C_{i2} \times C_{i3}\bigr)^{1/3}.$$8 to $$C_i = \bigl(C_{i1} \times C_{i2} \times C_{i3}\bigr)^{1/3}.$$9 F1 and, when used to reorder files in review, yields $\tau$0 more comments and higher hot-spot precision and recall. At the method-coupling level, Random Forest learning-to-rank reaches $\tau$1 and declines after about 60 days, motivating bi-monthly retraining (Hönel et al., 2020, Islam et al., 2019, Olewicki et al., 2024, Jia et al., 2024).

Method-level quality and metric-evaluation studies show that narrower targets are learnable but context-sensitive. TAPT-CodeBERT attains accuracy $\tau$2, precision $\tau$3, and F1 $\tau$4 for binary Java method-quality classification. In readability evaluation, nine LLMs were sensitive to comment removal, identifier obfuscation, and smell-removal refactoring, with $\tau$5 to $\tau$6 of executions showing non-zero standard deviation; identifier obfuscation produced the strongest readability degradation. CRScore achieves the highest alignment with human review-quality judgments among open-source metrics, with Spearman correlation $\tau$7 at review level and $\tau$8 for system ranking. PyQu, finally, reports best-model accuracies between $\tau$9 and $\Delta CC$0 and F1-scores between $\Delta CC$1 and $\Delta CC$2 across five quality attributes in Python ML systems (Mahamud et al., 2023, Simoes et al., 5 Jul 2025, Naik et al., 2024, Almukhtar et al., 4 Nov 2025).

5. Applications and workflow integration

A recurring deployment pattern is the use of quality estimation as a gate rather than merely as an analytic report. Uncertainty-based methods accept only low-entropy patch families; CORE surfaces only revisions that both satisfy the static analyzer and receive sufficiently strong ranker scores; refactoring-aware JIT-DP suppresses alerts on pure refactoring-only commits; and similar strategies can be embedded into CI/CD to decide when to trust automation, when to request more evidence, and when to escalate to human review (Sharma et al., 17 Feb 2025, Wadhwa et al., 2023, Niu et al., 25 Jul 2025).

Another application is review orchestration. CQS inserts inline diff comments into the same interface as human reviewers. ReviewRanker estimates whether a review clearly communicates the intended code modification. Review-activity predictors reorder changed files so that likely hot-spots appear earlier, improving reviewer attention allocation. Merge-vs-abandonment prediction estimates whether a change is likely to survive the review process, enabling early triage, reviewer assignment, or managerial intervention. These systems treat review behavior itself as a measurable substrate of code change quality, or at least as a powerful proxy for it (Wong et al., 1 Aug 2025, Mahbub et al., 2023, Olewicki et al., 2024, Islam et al., 2019).

A third application concerns change completeness and maintenance analytics. Co-change ranking identifies methods that historically evolve together, so omitted high-ranking methods become plausible signals of incomplete or risky changes. Maintenance classification distinguishes adaptive, corrective, and perfective work, supporting process-quality analysis and temporal monitoring of maintenance balance. Intent classification via commit messages differentiates perfective and corrective quality-improving changes, showing that perfective changes more often reduce complexity while corrective changes frequently increase it. Effort estimation via Standard Coding Hours can then be layered on top as an auxiliary indicator of intrinsic change difficulty. This suggests a multi-view architecture in which correctness, maintainability, completeness, review dynamics, and effort are estimated jointly rather than collapsed prematurely into a single scalar (Jia et al., 2024, Hönel et al., 2020, Trautsch et al., 2021, Wright et al., 2019).

6. Limitations, controversies, and open problems

The central limitation of the area is that “quality” is inherently plural and often proxy-defined. Binary good/bad labels for methods, $\Delta CC$3 readability scores, perfective-versus-corrective intent labels, and merged-versus-abandoned outcomes all capture different slices of the phenomenon. Some studies explicitly note that readability is subjective, that binary outputs should become scores, or that intended quality improvement can still increase complexity when the change is corrective rather than perfective. This means that agreement between metrics is neither expected nor always desirable (Mahamud et al., 2023, Simoes et al., 5 Jul 2025, Trautsch et al., 2021).

A second limitation is supervision and evaluation bias. Refactoring tangling can corrupt defect labels unless refactoring-aware tracing is used. Pseudo-reference generation for review evaluation has non-trivial error and missing rates, even after combining LLMs with static analyzers. Cluster-to-class mapping by expert judgment achieves only moderate purity, reported as $\Delta CC$4 and $\Delta CC$5, which is useful for triage but not equivalent to exact semantic classification. Some industrial evaluations also depend on issue pools initially produced by frontier LLMs, which can bias what counts as “ground truth” (Niu et al., 25 Jul 2025, Naik et al., 2024, Kniazev, 16 Feb 2026).

A third limitation is the persistent trade-off between precision, recall, compute, and context. Symbolic execution gives semantically meaningful clusters but is bounded and timeout-limited. CQS achieves very high precision by collapsing recall. Co-change ranking degrades after roughly 60 days, implying continual retraining. Many models are language- or ecosystem-specific, and several quality dimensions—especially security, performance, and cross-file architectural properties—remain only partially covered. Even PyQu’s broader five-attribute framework is presently specialized to Python ML systems, and CQS explicitly notes that security and performance are not yet the main focus (Sharma et al., 17 Feb 2025, Wong et al., 1 Aug 2025, Jia et al., 2024, Almukhtar et al., 4 Nov 2025).

A plausible next step is a unified, multi-objective formulation in which code change quality is treated as a vector over correctness, maintainability, readability, defect risk, review utility, and effort, with calibrated abstention and explanation. The surveyed literature already supplies most of the ingredients: semantic clustering, refactoring-aware labels, structured issue detection, review-grounded metrics, process/history features, and metric-delta QA models. The unresolved question is not whether code change quality can be estimated, but how to preserve the specificity of each operational target while still producing decisions that are useful in real development workflows.