CodeScore-R: Robust Code Evaluation

• The paper introduces CodeScore-R, a test-case-free metric that predicts functional correctness by leveraging execution-based behaviors in synthesized code.
• It employs a dual approach using a UniCE-based reference-only model and a contrastively pretrained sketch normalization framework to handle identifier renaming and syntactic rewrites.
• Empirical results demonstrate improved MAE and F1 scores over traditional metrics, highlighting robustness against syntactic and semantic perturbations.

CodeScore-R is an automated, test-case-free metric designed to assess the functional correctness of synthesized code by aligning with execution-based behaviors, while also achieving robustness against superficial code perturbations such as identifier renaming and syntactic rewrites. Two distinct research efforts under the CodeScore-R banner—one introduced as the reference-only variant of CodeScore within the UniCE framework (Dong et al., 2023) and the other leveraging contrastive pretraining and sketch-based normalization atop UniXcoder (Yang et al., 2024)—target the need for scalable, reliable evaluation of generated code without reliance on hand-written test suites. Both variants are motivated by the limitations of conventional match-based metrics (e.g., BLEU, CodeBLEU) and seek to better reflect semantic (functional) equivalence directly from code structure.

1. Motivation and Foundational Principles

Traditional code evaluation metrics fall into three broad categories: match-based (BLEU, ChrF, exact match), semantic-based (CodeBLEU, CodeBERTScore), and execution-based (Pass@k). Match- and semantic-based metrics are generally fast but are easily misled by surface similarities or shallow embeddings, exhibiting poor alignment with true functional correctness, especially for programs with considerable lexical or syntactic variability. Execution-based metrics such as Pass@k—fraction of generated solutions passing a test suite—directly target program functionality but suffer from substantial overhead due to test execution and maintenance.

The CodeScore-R paradigm addresses these shortcomings by offering a fully automated (test-case-free), functionally-aware metric, robust to benign code modifications, and sensitive to subtle semantic errors. The core insight is to replace surface-level similarity heuristics with learned estimators of functionality, trained to predict or mimic execution-derived metrics using only static code as input (Dong et al., 2023, Yang et al., 2024).

2. Architectural Overview

2.1 CodeScore-R in the UniCE Framework

Within the UniCE framework (Dong et al., 2023), CodeScore-R is a reference-only LLM-based metric that predicts the likelihood a generated solution would pass available test cases, conditioned solely on the candidate and a single reference implementation. Training proceeds by:

• Pairing generated code $g$ and reference code $r$ as input to a transformer backbone (UniXcoder).
• Using layer-wise pooling to aggregate representations, yielding a unified embedding.
• Adding task-specific heads for:
  • predicting a scalar score $s$ (CodeScore-R) corresponding to the expected “PassRatio”, and
  • predicting a binary executability flag.

Losses are the squared error between $s$ and true PassRatio and cross-entropy for executability.

2.2 CodeScore-R via Contrastive Pretraining

The approach described in (Yang et al., 2024) employs contrastive pretraining (ConCE) and sketch-based identifier normalization to learn robust code embeddings. The pipeline:

• Sketch extraction: Normalizes user-defined identifiers (functions, parameters, locals) to abstract away irrelevant lexical variance.
• Positive/negative pair generation:
  • Syntactic-equivalent transformations (e.g., loop exchange, expression exchange, branch swapping, condition inversion) yield functionally identical positives.
  • Fine-grained mutation (arithmetic, relational, logical, assignment operators) introduces likely semantic bugs for negatives.
• Contrastive loss training: Encourages embedding proximity for semantically equivalent pairs, separation for negatives, with an auxiliary MLM task.
• Inference: Given candidate and reference code, sketch and embed both, then compute a cosine similarity score. Functional equivalence is determined by a similarity threshold.

3. Mathematical Formulation

3.1 Execution-Based Ground Truth

Let problem $p$ have a test set $C_p = \{(I_{p,c}, O_{p,c})\}$. The baseline execution-derived metric is:

$$\mathrm{PassRatio}(g_p) = \frac{1}{|C_p|} \sum_{c \in C_p} \mathbb{1}\{\mathrm{Eval}(g_p, I_{p,c}) = O_{p,c}\}$$

where $\mathbb{1}\{\cdot\}$ is the indicator function.

3.2 Learned Metrics

• UniCE-based CodeScore-R: Model $f$ predicts $s \in [0,1]$ and a binary executability, trained via

$$L = (s - \mathrm{PassRatio}(g))^2 + \text{cross-entropy}(Exec, \text{true label})$$

using $(g, r)$ as input (Dong et al., 2023).

• Contrastive Pretraining-based CodeScore-R:
  • Embeddings $h(c)$ via ReLU-pooled UniXcoder [CLS] tokens.
  • Contrastive loss over batch $B$:

  $$L_{CL} = -\sum_{i \in B} \log \frac{e^{\mathrm{sim}(h_i, h^+_i)/\tau}}{e^{\mathrm{sim}(h_i, h^+_i)/\tau} + \sum_j e^{\mathrm{sim}(h_i, h^-_j)/\tau}}$$

  with $\mathrm{sim}(u, v) = \frac{u \cdot v}{\|u\| \|v\|}$ and temperature $\tau$ (Yang et al., 2024).

• Final Score:

$$\mathrm{CodeScore\textrm{-}R}(c_1, c_2) = \begin{cases} 1 & \text{if}\ \mathrm{sim}(h_1, h_2) > 0.5 \ 0 & \text{otherwise} \end{cases}$$

4. Implementation Details

• Identifier Normalization (Sketching): Tree-sitter is used to parse code. Function names become “f”, parameters “arg_n”, locals “var_n”, reducing spurious layout and identifier effects (Yang et al., 2024).
• Syntactic-Equivalent Augmentations: Apply rewrites such as for/while interchanges, $a+=b$ vs. $a=a+b$, swapping if/else branches, condition inversions—preserving semantics but varying syntax.
• Semantic-Preserving and Semantic-Breaking Mutations: For contrastive learning, functionally broken mutants are created by modifying relational, arithmetic, assignment, or control-flow operators.
• Pooling Methods: Embedding pooling is a hyperparameter, with CLS+ReLU pooling usually optimal for Python generation (Yang et al., 2024).

5. Empirical Evaluation

Experimental validation spans code generation and migration for Python and Java, using HumanEval, HumanEval-X, and AVATAR migration datasets, with ground-truth established via Pass@1.

Quantitative highlights include (all MAE relative to Pass@1):

Task	Best Baseline MAE	CodeScore-R MAE	Relative MAE ↓
Java Generation	0.342	0.317	7.3%
Python Generation	0.324	0.287	11.4%
Python→Java Migration	0.202	0.175	13.4%
Java→Python Migration	0.151	0.125	17.2%

F1 classification scores (score$>0.5$ vs. Pass@1$>0$) reach ≈0.8 (generation) and ≈0.9 (migration) (Yang et al., 2024).

In reference-only scenarios (APPS-Eval/MBPP-Eval/HE-Eval), CodeScore-R achieves higher correlation with functional correctness (e.g., Spearman $\rho$ ≈0.59–0.67) than BLEU (≈0.07–0.18) or CodeBLEU (≈0.29–0.41), at a fraction of the evaluation cost (Dong et al., 2023).

6. Robustness and Analysis

Multiple robustness studies demonstrate that CodeScore-R is invariant under:

• Identifier perturbations: Mean Absolute Error (MAE) for baselines can double, but CodeScore-R remains stable.
• Syntax perturbations: Application of random rewrites preserves CodeScore-R MAE (≈0.32–0.34) while baseline errors increase (up to 0.8).
• Semantic perturbations: Large-scale code operator mutations degrade CodeBERTScore’s MAE to ≈0.95+, but CodeScore-R degrades gracefully (e.g., up to 0.46 for migration).

Contrastive pretraining ensures functional-semantic distinctions are learned, while sketch normalization and augmentation yield invariance to irrelevant code aspects. Illustrative examples reveal CodeScore-R correctly rewards functional but superficially divergent implementations and penalizes syntactically similar but faulty ones (Dong et al., 2023, Yang et al., 2024).

7. Limitations and Future Directions

• Dependency on Compiler: CodeScore-R requires that code compiles for scoring (contrasting with text-only baselines).
• Computational Cost: Pretraining is compute-intensive compared to surface-matching methods, though inference is efficient.
• Representation Collisions: Some semantic collisions remain due to limitations in transformer embedding spaces.

Research directions for improving CodeScore-R include scaling to more languages, enriching representations with AST graphs or execution traces, and calibrating thresholds for capturing finer-grained Pass@k estimates (Yang et al., 2024). There is active interest in adapting these techniques to diverse tasks such as multilingual code synthesis and open-domain program generation.

---

References:

• (Dong et al., 2023)
• (Yang et al., 2024)