CodeBLEU: Code-Aware Evaluation Metric

• CodeBLEU is an automatic evaluation metric for code synthesis that improves upon BLEU by integrating weighted n-gram, AST-based syntactic, and data-flow semantic matching.
• It computes a linear combination of standard BLEU, weighted n-gram match, syntax-aware AST match, and semantic data-flow match, with tunable weights to suit various code tasks.
• Widely used in code translation, decompilation, repair, and refactoring, CodeBLEU highlights both its strengths in structural evaluation and limitations like reference sensitivity and parser dependence.

CodeBLEU is an automatic evaluation metric for code synthesis that was introduced to address limitations of BLEU and exact-match accuracy for source code. In its original formulation, it combines standard BLEU, a weighted n‑gram match, an AST-based syntactic match, and a data-flow-based semantic match, thereby attempting to evaluate generated programs not only by token overlap but also by syntax and program semantics (Ren et al., 2020). Subsequent work has used CodeBLEU for code translation, decompilation, refactoring, vulnerability repair, geospatial workflow generation, and plagiarism-oriented ranking, while also exposing systematic limitations: sensitivity to reference form, parser dependence, weak handling of functional equivalence, and, in some settings, divergence from execution-based correctness or human judgment (Aljagthami et al., 16 Sep 2025, Abualazm et al., 7 Jul 2026, Evtikhiev et al., 2022).

1. Origins and motivation

CodeBLEU was proposed in response to three limitations of existing code evaluation practice. BLEU was originally designed for natural language and neglects important syntactic and semantic features of code; perfect accuracy is too strict because semantically equivalent programs can differ textually; and computational accuracy is powerful but requires compilers, interpreters, and test inputs, making it difficult to scale across settings (Ren et al., 2020). The original paper therefore framed CodeBLEU as a weighted syntactic and semantic BLEU score for code synthesis (Ren et al., 2020).

The metric was introduced and validated on three code synthesis tasks: text-to-code, code translation, and code refinement (Ren et al., 2020). Later literature adopted it as a standard or baseline code-aware metric in a wide range of tasks. In code translation, it is explicitly used to capture “surface correspondence and structural similarity” jointly with BLEU (Aljagthami et al., 16 Sep 2025). In decompilation, it is used as a primary automatic similarity metric because it adds AST and data-flow components to BLEU and is treated as a proxy for readability and semantic similarity (Abualazm et al., 2 Apr 2026). In code generation for security and refactoring, it is frequently retained as an important baseline even when the surrounding papers argue that it is insufficient on its own (Ouédraogo et al., 7 Jun 2025, Cheng et al., 6 Dec 2025).

A recurrent theme in this literature is that CodeBLEU is intended as an intermediate point between text similarity and full semantic verification. It is more code-aware than BLEU, but it is still a static, reference-based metric rather than an execution-based notion of correctness (Abualazm et al., 7 Jul 2026, Ren et al., 2020).

2. Formal construction

In the original definition, CodeBLEU is a linear combination of four sub-metrics (Ren et al., 2020):

$$\begin{aligned} \mathrm{CodeBLEU} &= \alpha \cdot \mathrm{BLEU} \ &\quad + \beta \cdot \mathrm{BLEU_{weight}} \ &\quad + \gamma \cdot \mathrm{Match_{ast}} \ &\quad + \delta \cdot \mathrm{Match_{df}} \end{aligned}$$

where $\alpha,\beta,\gamma,\delta$ are weights that sum to $1$ (Ren et al., 2020). The paper uses $\alpha=\beta=\gamma=\delta=0.25$ in examples and later reports that $\alpha=\beta=0.1,\ \gamma=\delta=0.4$ yields the best average correlation with human scores across its tasks (Ren et al., 2020).

The first component is standard BLEU, defined with clipped $n$‑gram precision and a brevity penalty (Ren et al., 2020). The second component, $\mathrm{BLEU_{weight}}$, is a weighted n‑gram match intended to emphasize programming-language keywords; in the original implementation, keywords are given five times the weight of other tokens, and this term is computed over unigrams only (Ren et al., 2020). This design reflects the intuition that tokens such as if, for, return, type names, and other reserved words are more semantically consequential than arbitrary identifiers.

The third component, $\mathrm{Match_{ast}}$, is a syntactic similarity score computed from AST subtrees. Candidate and reference programs are parsed, leaf nodes are removed, all subtrees are extracted, and a clipped-match ratio is computed between candidate and reference subtree multisets (Ren et al., 2020). The fourth component, $\mathrm{Match_{df}}$, is a semantic similarity score based on data-flow graphs: variables are extracted, dependencies are represented as edges indicating that the value of one variable comes from another, variable names are normalized, and a clipped-match ratio is computed over data-flow items (Ren et al., 2020).

Subsequent papers often restate the same conceptual decomposition with slightly different notation. For example, translation and benchmarking studies describe CodeBLEU as combining n‑gram match, weighted n‑gram match, syntax-aware AST similarity, and semantic or data-flow similarity, sometimes with explicit weightings suited to their application (Zhang et al., 7 Sep 2025, Bhattarai et al., 2024). A GeoAI benchmark gives an explicit fixed combination of $0.2$ n‑gram, $\alpha,\beta,\gamma,\delta$0 weighted n‑gram, $\alpha,\beta,\gamma,\delta$1 AST syntax match, and $\alpha,\beta,\gamma,\delta$2 data-flow match for its Python/ArcPy setting (Zhang et al., 7 Sep 2025).

3. Computation and implementation

In practice, CodeBLEU is computed by comparing generated code against a reference implementation using language-specific tokenization, parsing, and data-flow extraction (Ren et al., 2020). The original paper uses tree-sitter to construct ASTs (Ren et al., 2020). Later work generally adopts the standard implementation rather than redefining the metric, and several papers explicitly state that they use the public or canonical CodeBLEU toolkit with default settings (Dong et al., 2022, Aljagthami et al., 16 Sep 2025).

Implementation is inherently language dependent. Tokenization requires lexical rules for the target language; weighted n‑gram scoring requires a keyword list; AST matching requires a parser; and data-flow matching requires rules for extracting variable definitions and uses (Ren et al., 2020). Because of this dependence, multiple later papers either adapt CodeBLEU to new languages or note that they rely on its existing language support. One decompilation study contributes a Dart-adapted CodeBLEU with Dart-specific AST parsing and data-flow analysis, explicitly noting that no prior Dart implementation existed (Abualazm et al., 7 Jul 2026). Another EDA automation study introduces an extended CodeBLEU for TCL scripts with stage-specific weights, TCL-specific data-flow extraction, and line-based syntax matching because robust tree-sitter support for TCL was unavailable (Lu et al., 1 Aug 2025).

Several papers also highlight that preprocessing choices materially affect CodeBLEU. Removing comments and formatting noise can reduce superficial variance in code-translation or plagiarism settings (Aljagthami et al., 16 Sep 2025, Ebrahim et al., 28 Apr 2026). In some domains, reference choice matters as much as implementation. GeoAnalystBench fixes a single expert or ModelBuilder-generated ArcPy implementation as ground truth, which makes CodeBLEU a measure of alignment to that specific reference rather than to the space of all valid GIS workflows (Zhang et al., 7 Sep 2025). In retrieval-augmented Fortran-to-C++ translation, pairwise CodeBLEU scores between generated C++ snippets are even used as continuous supervision signals in a contrastive learning objective for embedding alignment (Bhattarai et al., 2024).

4. Applications across code tasks

CodeBLEU has become a general-purpose metric for code-generation and code-transformation research, but its role varies by task.

In code translation, it is used to evaluate whether translations are structurally faithful to target-language references. A 2025 study on C++, Java, Python, and C# translation reports CodeBLEU direction-wise across models, prompt styles, and prompt languages, and treats it as the main structural and semantic evaluation metric alongside BLEU (Aljagthami et al., 16 Sep 2025). In HPC translation between Fortran and C++, it is the central quantitative metric used to demonstrate gains from domain-specific datasets and fine-tuning, with scores reported on a $\alpha,\beta,\gamma,\delta$3 scale and compared against GPT‑4 and human translations (Lei et al., 2023). In retrieval-augmented Fortran-to-C++ translation, CodeBLEU is both the main evaluation metric and the optimization target for retrieval embeddings, raising average CodeBLEU from $\alpha,\beta,\gamma,\delta$4 to $\alpha,\beta,\gamma,\delta$5 on the HPC Fortran2C++ dataset and from $\alpha,\beta,\gamma,\delta$6 to $\alpha,\beta,\gamma,\delta$7 on Numerical Recipes without fine-tuning the LLM (Bhattarai et al., 2024).

In decompilation, CodeBLEU is used as a proxy for code readability or semantic similarity, usually paired with compile@k and increasingly contrasted with pass@k. A Dart decompilation study reports a mean CodeBLEU of $\alpha,\beta,\gamma,\delta$8 for a specialized 4B model on a 73-function test set, approximately comparable to a much larger code model at $\alpha,\beta,\gamma,\delta$9 (Abualazm et al., 2 Apr 2026). A later study on HumanEval-Dart uses a Dart-specific CodeBLEU and shows that CodeBLEU and compile@k can improve significantly while pass@k moves in the opposite direction, arguing that pass@k must be the primary evaluation metric for neural decompilation (Abualazm et al., 7 Jul 2026).

In code generation and repair, CodeBLEU appears both as a central metric and as an object of critique. CodePAD, a pushdown-automaton-based framework for grammar-constrained generation, uses CodeBLEU to show that enforcing grammatical correctness yields improvements such as a relative 17% CodeBLEU gain on CONALA and a rise for CodeGen-350M on MBPP from $1$0 to $1$1 in zero-shot (Dong et al., 2022). SynthFix uses CodeBLEU and CrystalBLEU as primary repair-quality metrics and reports up to 18% relative improvement in CodeBLEU/CrystalBLEU over strong baselines on FixJS and CodeFlaws (Zhang et al., 19 Apr 2026). Human-attention-guided training for code summarization reports a CodeBLEU increase from $1$2 to $1$3 on CodeXGlue Java summarization, with especially large improvement in the data-flow component (Zhang et al., 19 Mar 2025).

In workflow and domain-specific scripting, CodeBLEU has also been specialized. GeoAnalystBench uses it to evaluate Python geospatial workflows against expert reference code, reporting overall CodeBLEU values such as $1$4 for ChatGPT‑4o‑mini and $1$5 for DeepSeek‑R1‑7B (Zhang et al., 7 Sep 2025). AutoEDA introduces an extended CodeBLEU for EDA TCL scripts, arguing that syntax correctness and logical flow are more important than vocabulary repetition and reporting large gains over baselines in multi-stage RTL-to-GDSII automation (Lu et al., 1 Aug 2025).

5. Empirical behavior and what the metric captures

A consistent empirical finding is that CodeBLEU is usually more informative than BLEU when code structure matters. The original paper shows higher Pearson correlation with programmer-assigned scores than BLEU on text-to-code, code translation, and code refinement (Ren et al., 2020). Translation and code-generation papers routinely use CodeBLEU to detect cases where BLEU remains high because of shared tokens but structure, AST shape, or data-flow diverges (Aljagthami et al., 16 Sep 2025, Dong et al., 2022).

At the same time, later studies make clear that CodeBLEU is not a uniform proxy for human or functional judgment. “Out of the BLEU” conducts a human study on CoNaLa and Hearthstone and finds that CodeBLEU does not perform significantly better than several machine-translation metrics in emulating human judgment; on CoNaLa, none of the studied metrics can correctly emulate human judgment with more than 95% certainty when model-score differences are below 5 points, and the paper concludes that ChrF is a better fit than BLEU and CodeBLEU for those datasets (Evtikhiev et al., 2022). In LLM-based test refactoring, CodeBLEU is described as overly sensitive to renaming and structural edits, producing many false negatives because readability-improving refactorings deviate from the original lexical form (Ouédraogo et al., 7 Jun 2025). In that setting, CTSES is proposed as a composite of CodeBLEU, METEOR, and ROUGE‑L (Ouédraogo et al., 7 Jun 2025).

Domain-specific studies also reveal characteristic response patterns. In code translation, CodeBLEU is sensitive to prompt design and prompt language; detailed prompts yield consistent gains across models and directions, and English prompts outperform Arabic by about 13–15% in CodeBLEU (Aljagthami et al., 16 Sep 2025). In CodePAD, CodeBLEU rises in lockstep with grammatical correctness percentage, indicating that parser success and valid structure strongly affect the AST and data-flow components (Dong et al., 2022). In GeoAnalystBench, AST syntax scores are generally higher than data-flow and n‑gram scores, suggesting that current LLMs produce syntactically plausible Python more readily than workflows whose logical dependencies fully match the reference (Zhang et al., 7 Sep 2025).

A further empirical use is as a ranking signal rather than a strict correctness score. In plagiarism research, CodeBLEU is repurposed to rank plagiarized against non-plagiarized Java pairs. It performs strongly overall, outperforming JPlag in pooled AUROC and AP, with very strong performance at plagiarism levels L1–L3 and clear degradation from L4 onward (Ebrahim et al., 28 Apr 2026). This use highlights that CodeBLEU can be useful whenever similarity to a reference-like artifact is the operative notion, even outside classical code generation.

6. Limitations, controversies, and alternatives

The major limitation of CodeBLEU is that it remains a static, reference-based similarity metric. It does not execute code, so it cannot guarantee functional equivalence. Multiple papers make this point explicitly. The original proposal contrasted CodeBLEU with computational accuracy as a more practical but less direct proxy for semantics (Ren et al., 2020). CodePAD notes that CodeBLEU and grammatical correctness are not sufficient for compilation or execution correctness (Dong et al., 2022). The Dart neural decompilation study demonstrates metric divergence directly: CodeBLEU and compile@k can improve significantly while pass@k regresses, implying that CodeBLEU can reward outputs that look more like reference code without being more functionally correct (Abualazm et al., 7 Jul 2026).

A second limitation is sensitivity to single-reference form. CodeBLEU may penalize alternative but valid implementations, especially when reordering, helper-function extraction, different API choices, or refactoring alter AST shape or data-flow topology relative to the reference. This issue is central in test refactoring, where renaming and modularization are desirable but can lower CodeBLEU (Ouédraogo et al., 7 Jun 2025). It is also central in security evaluation, where the CFCEval paper argues that CodeBLEU suffers from imprecise tokenization, structural limitations on short snippets, and low reference diversity, and that its correlation with human or LLM-judge assessments is substantially below that of the proposed ELRM metric (Cheng et al., 6 Dec 2025).

A third limitation is parser and language dependence. CodeBLEU requires language-specific parsing and data-flow extraction; minor syntax errors can zero out or degrade AST and data-flow components even when intent is clear (Aljagthami et al., 16 Sep 2025). For unsupported languages or DSLs, researchers must implement adaptations, as in Dart decompilation and TCL-based EDA automation (Abualazm et al., 7 Jul 2026, Lu et al., 1 Aug 2025).

These limitations have motivated several alternatives and critiques. “Out of the BLEU” argues that ChrF may be a better default metric than BLEU or CodeBLEU for some code-generation tasks (Evtikhiev et al., 2022). CSSG positions itself as a semantic-graph alternative, arguing that CodeBLEU, like BLEU and TSED, relies on surface-level string overlap or AST structure and fails to capture deeper semantic relationships encoded in program dependence graphs (Xu et al., 7 Jan 2026). CFCEval introduces ELRM for security-oriented code relevance and reports much higher correlation with human and LLM-as-judge scores than CodeBLEU (Cheng et al., 6 Dec 2025). Other papers recommend composite or multi-metric evaluation, such as CTSES for test refactoring (Ouédraogo et al., 7 Jun 2025), or pairing CodeBLEU with compile@k, pass@k, execution-based checks, and human review (Abualazm et al., 7 Jul 2026, Abualazm et al., 2 Apr 2026).

Taken together, this literature establishes a stable but qualified role for CodeBLEU. It is one of the most important automatic metrics for code similarity because it extends BLEU with syntax and data-flow awareness, and it remains broadly useful for translation, generation, repair, and decompilation (Ren et al., 2020). But it is not a substitute for dynamic correctness, and its best use is as one component in a broader evaluation stack rather than as a stand-alone definition of code quality (Abualazm et al., 7 Jul 2026, Cheng et al., 6 Dec 2025).