CodeXGLUE: Unified Code Benchmark
- CodeXGLUE is a unified benchmark suite for code intelligence, encompassing 10 tasks across 14 datasets to assess code understanding, generation, and translation.
- It supports diverse modalities (code→code, text→code, code→text, text→text) with practical applications in code search, completion, repair, and summarization.
- The platform standardizes evaluation metrics and provides baseline models, enabling reproducible comparisons and guiding multi-task research in program analysis.
CodeXGLUE is a machine learning benchmark dataset and evaluation platform for code understanding and generation, introduced as a collection of 10 tasks across 14 datasets together with standardized splits, task-specific metrics, and three reference baselines. It spans code→code, text→code, code→text, and text→text settings across Python, Java, JavaScript, PHP, Ruby, Go, C/C++, C#, and C, and was designed as a unified comparison point for program analysis, synthesis, translation, repair, search, and summarization systems (Lu et al., 2021).
1. Origins, scope, and benchmark rationale
CodeXGLUE was proposed to play for code intelligence a role analogous to ImageNet in vision and GLUE in natural language processing. Its stated motivation is that, although strong pretrained models for code already existed, the community lacked a comprehensive, multi-task benchmark for evaluating generalization across both program understanding and program generation. The benchmark therefore targets practical developer-facing tasks such as code search, code completion, code translation, code repair, and code summarization, while also providing a public evaluation platform and baseline implementations (Lu et al., 2021).
The original release comprises 10 tasks over 14 datasets and includes three baseline system families: a BERT-style encoder, a GPT-style decoder, and an encoder–decoder model. The platform is distributed through a public repository and supports leaderboard-style comparison. In later literature the name is occasionally written as “CodexGlue,” but the benchmark identity remains the same; subsequent work treats it as a comprehensive or well-known benchmark suite for code intelligence and uses it as a common testbed for architecture design, prompting, retrieval augmentation, symbolic guidance, structure-aware pretraining, and efficiency studies (Lu et al., 2021, Icoz et al., 24 Jul 2025).
2. Task taxonomy and dataset composition
CodeXGLUE organizes its tasks into four modality groups. The taxonomy is central to its design, because it places retrieval, classification, autoregressive completion, sequence transduction, and summarization under a single evaluation umbrella (Lu et al., 2021).
| Category | Tasks | Representative datasets and languages |
|---|---|---|
| Code→Code | Clone detection, defect detection, cloze test, code completion, code repair, code translation | BigCloneBench, POJ-104, Devign, CT-all, CT-maxmin, PY150, GitHub Java Corpus, Bugs2Fix, Java↔C# |
| Text→Code | Natural language code search, text-to-code generation | CodeSearchNet AdvTest, WebQueryTest, CONCODE |
| Code→Text | Code summarization | Filtered CodeSearchNet in 6 languages |
| Text→Text | Documentation translation | Microsoft Docs EN↔LA/DA/NO/ZH |
Within code→code evaluation, clone detection is split into binary classification on BigCloneBench and retrieval on POJ-104. BigCloneBench contains 901,028/415,416/415,416 train/dev/test Java pairs, while POJ-104 contains 104 problems with 500 solutions per problem and is split by problem IDs into 64/16/24 train/dev/test problems, corresponding to roughly 32K/8K/12K programs. Defect detection uses Devign, a C-language vulnerability dataset of 27,318 labeled functions from QEMU and FFmpeg, split about 80/10/10. The cloze tests CT-all and CT-maxmin are derived from CodeSearchNet dev/test data across Go, Java, JavaScript, PHP, Python, and Ruby; CT-all has 176,115 test instances and CT-maxmin 2,615. Code completion uses PY150 for Python and the GitHub Java Corpus for Java, with both token-level and line-level settings. Bugs2Fix supports code repair with two Java subsets: Small, 46,680/5,835/5,835, and Medium, 52,364/6,545/6,545. Java↔C# translation uses 11,800 aligned function pairs mined from Lucene, POI, JGit, and Antlr, with about 10,300 training pairs, 500 dev, and 1,000 test (Lu et al., 2021).
The text→code side includes natural-language code search and text-to-code generation. Code search combines CodeSearchNet AdvTest, with 251,820/9,604/19,210 train/dev/test Python pairs, and WebQueryTest, a real-query dataset whose final binary test portion has 642 negative and 422 positive examples. Text-to-code generation uses CONCODE, a Java dataset with 100K train, 2K dev, and 2K test instances, pairing natural-language intent and class environment with target methods (Lu et al., 2021).
For code→text summarization, CodeXGLUE uses filtered CodeSearchNet corpora in six languages: Go 167,288/7,325/8,122; Java 164,923/5,183/10,955; JavaScript 58,025/3,885/3,291; PHP 241,241/12,982/14,014; Python 251,820/13,914/14,918; and Ruby 24,927/1,400/1,261. For text→text, it adds multilingual documentation translation on Microsoft Docs for EN↔LA, EN↔DA, EN↔NO, and EN↔ZH, with 43K, 19K, 44K, and 50K training pairs respectively and 1K/1K dev/test per direction (Lu et al., 2021).
3. Evaluation protocols, metrics, and baseline systems
A defining feature of CodeXGLUE is that it standardizes metrics by task rather than forcing a single measure across heterogeneous problem types. It uses F1 for clone detection on BigCloneBench, MAP for POJ-104 retrieval, Accuracy for defect detection and cloze tests, token Accuracy plus line-level Exact Match and Levenshtein edit similarity for code completion, MRR and classification Accuracy/F1 for code search, Exact Match together with BLEU and CodeBLEU for repair, translation, and text-to-code generation, and smoothed BLEU for summarization. CodeBLEU is explicitly code-aware, combining lexical, syntax, and data-flow components:
with (Lu et al., 2021).
The baseline BERT-style system is CodeBERT, a 12-layer Transformer encoder with hidden size 768, 12 attention heads, maximum length 512, and about 125M parameters, pretrained with Masked Language Modeling and Replaced Token Detection on CodeSearchNet. The GPT-style baseline is CodeGPT or CodeGPT-adapted, a GPT-2–style 12-layer decoder with hidden size 768, 12 heads, maximum length 1,024, and about 124M parameters, trained autoregressively on code. The encoder–decoder baseline initializes the encoder from CodeBERT and uses a randomly initialized 6-layer Transformer decoder (Lu et al., 2021).
The initial benchmark results established a characteristic division of labor across model families. CodeBERT led clone detection with F1 96.5 and defect detection with Accuracy 62.08; CodeGPT-adapted led code completion with overall score 71.28 and text-to-code generation on CONCODE with EM 20.10, BLEU 32.79, and CodeBLEU 35.98; the CodeBERT-initialized encoder–decoder led code repair and code translation, with overall code-repair Exact Match 0.108 and Java↔C# translation overall CodeBLEU 82.26. On code summarization, CodeBERT reached average smoothed BLEU 17.83 across the six CodeSearchNet languages (Lu et al., 2021).
4. Code understanding, vulnerability analysis, and completion research
One major line of follow-on work uses CodeXGLUE to study vulnerability and defect detection. CoTexT, a T5-style encoder–decoder pretrained on code and code–text data, reported Devign defect-detection Accuracy 66.62 with its CoTexT (1-CCG) variant, surpassing PLBART by 3.44 points even though C was not among its pretraining languages (Phan et al., 2021). VulBERTa, a RoBERTa-based C/C++ model with Clang-aware tokenization and approximately 125M parameters, followed the official Devign protocol and reached 64.75% Accuracy with an MLP head and 64.42% with a CNN head, ranking third on the May 2022 CodeXGLUE leaderboard snapshot behind CoTexT and C-BERT (Hanif et al., 2022). A contrasting low-footprint approach evaluated Code2vec on CodeXGLUE’s defect-detection benchmark using Devign and, after AST path-context extraction with astminer, obtained 61.43% Accuracy, 57.50% Precision, 61.77% Recall, and 59.56% F1, with about 5 minutes of training on a single GTX 1050 Ti, around 600 MB memory per training step, and 31M parameters (Coimbra et al., 2021).
Later studies also repurposed CodeXGLUE-style defect detection beyond the original C setting. One automated code review study uses what it describes as the Python defect-detection sub-dataset as a proxy for review, framing buggy-versus-clean classification as the core reviewer decision. In that setting, the hybrid combination of fine-tuning, few-shot prompting, and a textual “knowledge map” of 20 Python bug patterns yielded Accuracy 0.687 with GraphCodeBERT, alongside Precision 0.485, Recall 0.532, and F1 0.381 (Icoz et al., 24 Jul 2025).
Code completion has been another major beneficiary of the benchmark. ReACC introduced retrieval augmentation for token- and line-level completion on PY150 and JavaCorpus, using a hybrid retriever with score and fragment alignment. Its best ReACC-hybrid variant reached PPL 2.311, EM 46.26, and Edit Similarity 74.41 on PY150, and PPL 3.327, EM 30.70, and Edit Similarity 64.73 on JavaCorpus (Lu et al., 2022). MultiCoder studied multi-programming-lingual pretraining and a programming-language-level Mixture-of-Experts design on the same completion tasks, reporting 76.66 Accuracy and 83.84 Edit Similarity on PY150 and 77.13 Accuracy and 80.64 Edit Similarity on Java, with larger gains on Java as the relatively low-resource language (Gong et al., 2022). eWASH approached the long-context problem through syntax-hierarchy-based window selection and also introduced a normalized-literals benchmark variant; on files exceeding the 1,024-token window, XGPT-C with normalized literals achieved ROUGE-L Precision 0.91, ROUGE-L Recall 0.96, Edit distance 94.1, and EM@5 63.9%, with similarly strong gains on shorter files (Clement et al., 2021).
Completion work has also exposed inference-time considerations not captured by traditional accuracy-only leaderboards. A prompt-engineering study on 1,000 random Java completion snippets from CodeXGLUE used Llama 3 8B Instruct and showed that explicit prompt tags such as <code> and <incomplete> reduced GPU energy consumption and execution time while improving Exact Match and Edit Distance; by contrast, removing the system role produced much worse energy and accuracy outcomes, including zero exact matches in the zero-shot setting (Rubei et al., 10 Jan 2025).
5. Repair, translation, summarization, and generation trajectories
CodeXGLUE’s code repair and code translation tasks have become a focal point for systems that move beyond direct full-sequence generation. NSEdit framed repair as generation of a neural-symbolic edit script under a regular-language edit grammar with insert and delete operations, plus a pointer network for locations and reranker ensembling. On the Tufano small abstract subset it achieved 24.04% top-1 Exact Match, outperforming CoTexT’s 22.64% and establishing a new state of the art for that subset; on Tufano medium abstract it reached 13.87%, below CoTexT’s 15.36% (Hu et al., 2022). Toggle then pushed this direction further by decoupling token-level bug localization from repair and constructing prompts around the predicted edit region. Its best end-to-end CodeXGLUE instances reached 25.07% Exact Match on Tufano Small and 16.19% on Tufano Medium, surpassing the strongest previously reported baselines on both (Hossain et al., 2024).
Structure-aware pretraining has also been evaluated directly on CodeXGLUE’s code-to-code tasks. AST-T5 introduced AST-Aware Segmentation and AST-Aware Subtree Corruption without changing the T5 architecture, and improved Bugs2Fix Exact Match to 23.8 on the small split and 16.1 on the medium split. On Java↔C# translation it reached 68.9 EM for Java→C# and 72.3 EM for C#→Java, while also reporting 98.6 F1 on BigCloneBench clone detection and 65.8 Accuracy on Devign defect detection (Gong et al., 2024). Domain-shift work around the CodeXGLUE APR task showed that cross-project distribution mismatch is severe: on the small dataset the weighted average Exact Match dropped from 31.53% in the “Included” condition to 5.40% when the target project was excluded from training, and full fine-tuning adaptation raised it to 45.04%; a synthetic bug generator, CodeXBUG, raised the small-set weighted average from 5.40% to 28.82% in the zero-shot setting (Zirak et al., 2022).
On the code→text and text→code side, CodeXGLUE has also been a testbed for data-centric, multi-task, and prompting-based interventions. A self-improvement framework that generates pseudo-labels with the fine-tuned model itself improved CodeBERT, CodeT5, and UniXCoder on both summarization and CONCODE generation; for example, CodeT5 base summarization improved from 19.17 to 19.78 overall smoothed BLEU-4 at beam 1, and on CONCODE its EM improved from 21.75 to 22.40 and BLEU from 39.00 to 39.75 (To et al., 2023). DocChecker combined code–text contrastive learning, binary consistency classification, and comment generation on top of a UniXcoder backbone, and reported 33.64 BLEU-4 overall across the six CodeXGLUE summarization languages, far above the baseline numbers cited for RoBERTa, CodeBERT, PLBART, CodeT5, and UniXcoder (Dau et al., 2023). HumanLLM used Java code summarization with eye-tracking-derived attention motifs and reward-guided fine-tuning of CodeT5-base; on a 10% sample of the Java summarization data it improved BLEU from 5.36 to 10.32, ROUGE-L from 10.80 to 14.91, METEOR from 7.22 to 12.88, and CodeBLEU from 8.51 to 15.67 (Zhang et al., 19 Mar 2025).
Prompting studies have used CodeXGLUE to evaluate closed models and small LLMs under controlled conditions. A ChatGPT study on CONCODE and C#→Java translation showed that prompt structure strongly affected both BLEU and CodeBLEU: the best text-to-code prompt with a conciseness directive reached BLEU 26.86 and CodeBLEU 50.18 on CONCODE, while the best code-to-code setting, obtained with continuous sessions, reached BLEU 16.82 and CodeBLEU 48.80 on C#→Java translation (Liu et al., 2023). A later small-language-model benchmark used CodeXGLUE code-to-text in zero-shot mode across Python, JavaScript, Java, PHP, Ruby, and Go and reported language ranking frequencies rather than per-model BLEU: Python ranked first for 14 of 20 models, PHP ranked first for 6, Java was most often third, and Ruby and JavaScript most often occupied the lower ranks (Hasan et al., 3 Jul 2025). In another direction, a study of multi-task code LLM construction used Python Code-to-Text with 417,000 training pairs and 25,834 test examples, showing that data mixing was preferable around the 2B scale, whereas at 7B post-hoc merging with DARE or DELLA preserved summarization quality near specialist levels; for Qwen2.5-Coder 7B, the summarization specialist reached BLEU-4 0.0635, the mixed model 0.0598, DARE 0.0614, and DELLA 0.0578 (Zhu et al., 28 Jan 2026).
6. Interpretation, limitations, and evolving uses
CodeXGLUE’s influence is inseparable from recurrent debates about what its metrics actually measure. Several later studies explicitly argue that lexical overlap can misrepresent program correctness. In code refinement, Toggle treats Exact Match as fairer than BLEU or CodeBLEU because buggy and fixed methods typically share large non-buggy prefixes and suffixes, so lexical metrics can reward unchanged regions rather than successful repair (Hossain et al., 2024). The ChatGPT prompt study similarly found that strong CodeBLEU does not guarantee semantic correctness: in a manual check of 100 outputs per task, only 31 text-to-code generations and 59 code-to-code translations were judged functionally equivalent to the ground truth (Liu et al., 2023). For code summarization, the small-language-model evaluation cautions that higher BLEU for Python and PHP likely reflects shorter, more formulaic documentation styles and training-distribution effects rather than deeper semantic understanding (Hasan et al., 3 Jul 2025).
Generalization beyond the benchmark distribution is another recurring issue. The domain-adaptation study on APR shows that excluding the target project from training can collapse repair performance, especially on the small Tufano subset (Zirak et al., 2022). The Code2vec defect-detection study notes that its findings are specific to Devign and CodeXGLUE’s binary defect-detection setting, with labels that do not specify vulnerability type and may therefore limit nuanced analysis (Coimbra et al., 2021). VulBERTa likewise identifies label noise and external-validity concerns when moving beyond benchmark datasets such as Devign and D2A (Hanif et al., 2022). The multilingual summarization evaluation for small LLMs also notes that contamination by benchmark-like data cannot be ruled out for CodeXGLUE (Hasan et al., 3 Jul 2025).
Resource trade-offs have become increasingly visible through CodeXGLUE-based studies. Code2vec’s 61.43% Devign accuracy came with a markedly smaller memory footprint and shorter training time than CodeBERT, illustrating a low-compute baseline that remained competitive with non-code-pretrained transformers (Coimbra et al., 2021). The energy-focused Java completion study argues that benchmark practice should report energy and latency alongside lexical accuracy, since prompt design alone changed GPU energy consumption materially without changing model size or hardware (Rubei et al., 10 Jan 2025). A plausible implication is that CodeXGLUE has evolved from a static leaderboard into a methodological probe: later work uses the same benchmark not only to compare models, but also to compare context-selection schemes, retrieval strategies, pretraining objectives, symbolic guidance, prompt templates, weight-merging regimes, and efficiency profiles.
The benchmark has also proved extensible. eWASH introduced normalized-literals variants and file-level-context completion settings on top of CodeXGLUE-style evaluation (Clement et al., 2021). Other work has used CodeXGLUE selectively rather than exhaustively, such as automated code review framed through defect detection or small-model studies using only code summarization. This suggests that CodeXGLUE functions simultaneously as a benchmark suite, a common evaluation protocol, and a reusable substrate for benchmark variants and task re-interpretations. Across these uses, its lasting contribution is less a single score table than a standardized experimental language for research on machine learning over source code (Lu et al., 2021).