---
title: 'CodeBERTScore: NL→Code Evaluation Metric'
url: https://www.emergentmind.com/topics/codebertscore-cbs
type: topic
---

# CodeBERTScore: NL→Code Evaluation Metric

CodeBERTScore (CBS) is an unsupervised, embedding-based automatic evaluation metric for natural-language-to-code (NL→Code) generation tasks. It generalizes BERTScore to the code domain by leveraging code-aware Transformer models (such as CodeBERT) to produce contextualized embeddings for both generated code and references, conditioned on the natural language (NL) prompt. CBS computes pairwise token-level semantic similarity, enabling soft matching of syntactically and lexically divergent—but semantically equivalent—code fragments, while modeling consistency with the originating NL context. Correlations with human assessment and functional correctness demonstrate superior alignment compared to traditional metrics such as BLEU or METEOR [2302.05527].

## 1. Underlying Methodology and Mathematical Formulation

CBS operates by concatenating the NL prompt $x$ with the reference code $y^*$ and generated code $\hat{y}$, forming $S_\text{ref} = x \mathbin{\|} y^*$ and $S_\text{gen} = x \mathbin{\|} \hat{y}$, where “$\|$” denotes sequence concatenation. Each sequence is tokenized using the CodeBERT tokenizer $\mathcal{T}_\mathcal{B}$:
\[
\mathcal{T}_\mathcal{B}(S_\text{ref}) = \langle x_1,\ldots,x_k, y^*_1,\ldots,y^*_m\rangle
\]
\[
\mathcal{T}_\mathcal{B}(S_\text{gen}) = \langle x_1,\ldots,x_k, \hat{y}_1,\ldots,\hat{y}_n\rangle
\]

After passing these tokens through a pretrained CodeBERT encoder $\mathcal{B}_\ell$ (with selected layer $\ell$), context (NL) and punctuation tokens are masked. This yields code token embeddings:
- $H^* = \{h^*_i\ |\ M_\text{ctx}[i]=0\wedge M_\text{punc}[i]=0\}$ from reference
- $\hat{H} = \{\hat{h}_j\ |\ M_\text{ctx}[j]=0\wedge M_\text{punc}[j]=0\}$ from candidate

For each pair, cosine similarity $S_{ij}$ is computed:
\[
S_{ij} = \frac{\langle h^*_i,\ \hat{h}_j \rangle}{\|h^*_i\|\, \|\hat{h}_j\|}
\]
The similarity matrix $S\in \mathbb{R}^{m'\times n'}$ forms the basis for subsequent aggregation.

CBS aggregates these similarities into precision and recall with inverse-document-frequency (idf) token weighting:
\[
\text{Precision} = \frac{\sum_{j=1}^{n'} \operatorname{idf}(\hat{y}_j) \max_i S_{ij}}{\sum_{j=1}^{n'} \operatorname{idf}(\hat{y}_j)}
\quad
\text{Recall} = \frac{\sum_{i=1}^{m'} \operatorname{idf}(y^*_i) \max_j S_{ij}}{\sum_{i=1}^{m'} \operatorname{idf}(y^*_i)}
\]
The F1 score is their harmonic mean; optionally, $F_\beta$ (e.g., $F_3$ for functional emphasis) can be used:
\[
F_1 = \frac{2 PR}{P+R} \qquad
F_3 = \frac{10PR}{9P + R}
\]

Scores are linearly rescaled into $[0,1]$ via baseline subtraction: $F_1' = \frac{F_1 - b}{1 - b}$, with $b$ the mean score of random code pairs (typically $b\approx 0.76{-}0.78$).

## 2. Model Architecture and Implementation Specifics

CBS builds upon Microsoft's code-bert-base model: a 12-layer Transformer, hidden size $d=768$. Five language-specific variants (Java, Python, C, C++, JavaScript) are trained by continued masked language modeling (MLM) on CodeParrot-filtered corpora ($\sim$115M GitHub files, 1M steps, batch 32, learning rate 5e-5$\rightarrow$3e-5).

Embedding extraction is conducted at mid-to-high layers ($\ell\in\{7,8,9,10,11,12\}$), as optimal correlation with ground truth varies by language. Token idf weights are estimated on held-out development sets per language. The CBS reference implementation utilizes HuggingFace Transformers and Torch, and is distributed with openly available models and code [2302.05527].

## 3. Evaluation Protocols and Comparative Results

Evaluation benchmarks:
- Human preference: CoNaLa (472 NL→Python pairs, 5 model outputs per prompt, human rating 0–4).
- Functional correctness: HumanEval (164 Python prompts, reference and test cases), translated to Java, C++, JavaScript, using Codex outputs with pass/fail test case supervision.

Correlation analysis employs:
- Kendall’s $\tau$ (within each prompt’s 5 outputs)
- Spearman $\rho$ and Pearson $r$ (global across all generations)

Key outcomes (summarized for human preference and functional correctness): 

| Metric         | CoNaLa $\tau$ | CoNaLa $r_s$ |
| -------------- |:------------:|:------------:|
| BLEU           | 0.374        | 0.543        |
| chrF           | 0.470        | 0.623        |
| METEOR         | 0.366        | 0.540        |
| CodeBERTScore  | **0.517**    | **0.662**    |

| Lang       | BLEU $\tau$ | chrF $\tau$ | METEOR $\tau$ | CBS $\tau$ | CBS $r_s$ |
|:-----------|:-----------:|:-----------:|:-------------:|:----------:|:---------:|
| Java       | 0.481       | 0.532       | 0.558         | **0.553**  | 0.369     |
| C++        | 0.112       | 0.319       | 0.301         | **0.327**  | **0.393** |
| Python     | 0.393       | 0.394       | 0.418         | **0.422**  | **0.415** |
| JavaScript | 0.248       | 0.302       | 0.324         | **0.319**  | **0.402** |

CBS exhibits monotonic improvements in $\tau$ and $\rho$, exceeding BLEU by $\sim$0.05–0.14 $\tau$ on human preference and matching or outperforming all baselines on functional correctness [2302.05527].

## 4. Practical Usage and Software Integration

The CBS software is implemented in Python atop HuggingFace and Torch. Language-specific models are available on the HuggingFace Hub. Typical invocation entails:

```python
# 1. Install
pip install code-bert-score

# 2. Compute scores
from code_bert_score import score
nl = ["Return the factorial of n", ...]
preds = ["def fact(n): return 1 if n<2 else n*fact(n-1)", ...]
refs  = ["def fact(n): ...", ...]
P, R, F1 = score(
    preds, refs,
    lang="python",
    device="cuda",        # or "cpu"
    batch_size=8,
    idf=True,             # inverse document frequency weighting
    rescale_with_baseline=True
)
print("Mean CodeBERTScore F1:", F1.mean())
```

The package handles model loading, optimal layer selection, (NL∥code) tokenization, masking, pairwise similarity, IDF weighting, and final aggregation. Staff recommendations are to use language-specific pretrained models, extract from intermediate Transformer layers (preferably 7–10), employ $F_1$ for human preference and $F_3$ for functional correctness, apply idf weighting, and enable baseline scaling for normalized outputs [2302.05527].

## 5. Limitations and Failure Modes

CBS requires GPU acceleration for efficient computation; pairwise BERT embedding similarity is more computationally expensive than traditional n-gram counting. The quality of CBS is tied to that of its encoder model; future code LMs (such as CodeGen, SantaCoder) could further enhance score calibration. Layer selection, idf, scoring $b$, and $F_\beta$ parameter must be tuned per language and task.

Failure modes include:
- Very short predictions (1–2 tokens): susceptible to idf or similarity noise.
- Obfuscated or anomalous variable names: may yield unpredictable soft matches if substantially out of training distribution.

*This suggests* careful dataset preprocessing and per-language hyperparameter tuning are necessary for optimal results.

## 6. Recommendations and Comparative Advantages

Empirical findings confirm that using language-specific models confers a $\sim$1–2 point gain in $\tau/\rho$ over the generic CodeBERT-base. Extracting embeddings from mid-to-high layers achieves stronger correlation with reference-based and functional assessments than final-layer features. IDF weighting is shown to downweight trivial tokens (such as assignment or punctuation), and $F_3$ is advised where recall alignment (e.g., for correctness tests) is paramount.

In sum, CBS provides an effective drop-in replacement for BLEU/ROUGE/METEOR in code generation evaluation, delivering improved agreement with human judgments and test-case correctness. Its open-source implementation, together with pretrained language-targeted encoders, underpins a readily adoptable evaluation solution for NL→code systems [2302.05527].

Source: https://www.emergentmind.com/topics/codebertscore-cbs