Large Language Models for Code

Updated 7 July 2026

Large Language Models for Code are models pretrained with both source code and natural language to support tasks like generation, debugging, summarization, and translation.
These models span encoder-only, encoder–decoder, and decoder-only architectures, each optimized for distinct objectives such as code understanding or generative performance.
Research on LLMs4Code emphasizes robust benchmarks, advanced adaptation techniques, and privacy-aware deployment strategies to enhance reliability in real-world software development.

LLMs for code (LLMs4Code, or Code LLMs) are LLMs whose pretraining and/or fine-tuning corpora are enriched with source code alongside natural language, with the aim of supporting code generation, transformation, summarization, debugging, translation, and related software-engineering tasks (Raihan et al., 2024). Their rapid expansion has been driven by publicly accessible source-code corpora, advances in modern LLMs, and increasing computational resources, producing a code–model ecosystem in which datasets, pretrained models, downstream adaptations, and publication practices are tightly coupled (Yang et al., 2024).

1. Conceptual scope and model taxonomy

Code LLMs occupy the interface between natural languages and programming languages. A widely used taxonomy partitions their task space into four mappings: NL $\rightarrow$ NL, NL $\rightarrow$ PL, PL $\rightarrow$ PL, and PL $\rightarrow$ NL. In this scheme, NL $\rightarrow$ PL covers code generation from natural-language prompts; PL $\rightarrow$ PL covers refactoring, infilling, translation, and clone detection; PL $\rightarrow$ NL covers summarization, comment generation, and identifier normalization; and NL $\rightarrow$ NL includes natural-language tasks adjacent to software engineering, such as document translation and log analysis (Raihan et al., 2024).

The model space is conventionally divided into encoder-only, encoder–decoder, and decoder-only architectures. Encoder-only models, including BERT, RoBERTa, CodeBERT, GraphCodeBERT, TreeBERT, and Code-MVP, emphasize understanding-oriented objectives such as masked language modeling, replaced-token detection, and NL–PL contrastive learning. Encoder–decoder models, including T5, BART, CodeT5, CodeT5+, SPT-Code, and AlphaCode, support both generation and understanding. Decoder-only models dominate current generative practice and include general foundational models such as GPT-4, Claude 3, Gemini, LLaMA, Falcon, Mistral, Nemotron, and DBRX, as well as code-focused families such as Code-LLaMA, StarCoder, SantaCoder, phi-1/phi-2, WizardCoder, MagiCoder, OctoCoder, Phind-CodeLLaMA, and CodeGemma (Raihan et al., 2024).

This architecture-level taxonomy is matched by a corpus taxonomy. Natural-language sources include CommonCrawl, ROOTS, and The Pile, while programming-language sources include The Stack v1, The Stack v2, CodeParrot, CodeSearchNet, and synthetic code generated by other LLMs. Training objectives range from next-token prediction to MLM, span prediction, identifier prediction, AST-edge prediction, and CFG prediction. Fine-tuning practices include supervised tuning on prompt–code pairs, repair examples, and summarization pairs, together with efficient adaptation methods such as quantization, LoRA, and QLoRA, and alignment-oriented procedures such as instruction tuning and reinforcement learning (Raihan et al., 2024).

A plausible implication is that LLMs4Code should not be treated as a single homogeneous class. They are better understood as a layered family whose behavior depends on objective design, code–NL mixing ratios, tokenizer choices, and the degree to which the model is optimized for generation, understanding, or both.

2. Ecosystem structure, reuse, and publication norms

The LLM4Code ecosystem has been formalized as a directed heterogeneous graph

$E = (D, M, H, R_{md}, R_{mm}, R_{hm}),$

where $D$ is the set of code datasets, $\rightarrow$ 0 the set of pretrained code models, $\rightarrow$ 1 the set of Hugging Face owners and contributors, $\rightarrow$ 2 the model-to-data dependency relation, $\rightarrow$ 3 the model-to-model dependency relation, and $\rightarrow$ 4 the owner-to-model relation. In an empirical study of Hugging Face, this graph comprised 73 datasets, 366 models, 60 model-to-data edges, and 435 model-to-model reuse edges (Yang et al., 2024).

Popularity in this ecosystem is measured by likes $\rightarrow$ 5, downloads $\rightarrow$ 6, and reuse-degree $\rightarrow$ 7, defined as the out-degree in $\rightarrow$ 8. All three proxies follow a power-law distribution of the form $\rightarrow$ 9, with fitted $\rightarrow$ 0 values in the range 2.1–2.5 and $\rightarrow$ 1. The reported interpretation is a heavy-tailed “rich-get-richer” pattern in which a small number of hub models and datasets—such as t5-small, bigcode/starcoder, and codexglue—attract disproportionate downstream reuse (Yang et al., 2024).

Model reuse is not monolithic. Manual inspection of 435 reuse edges yields nine reuse categories.

Reuse category	Count	Share
Fine-tuning	205	47.1%
Architecture sharing	84	19.3%
Quantization	32	7.4%
Continue-pretraining	15	3.5%
Model conversion	14	3.2%
Distillation	12	2.8%
LoRA adaptation	9	2.1%
Instruction-tuning	9	2.1%
Adversarial training	6	1.4%

The top three reuse types are therefore fine-tuning, architecture sharing, and quantization. The same study notes that approximately 10% of reuse edges are accounted for by quantization and distillation together, reflecting growing efficiency and edge-deployment demands (Yang et al., 2024).

Publication practices in this ecosystem differ from those in broader AI and open-source repositories. README files for LLM4Code models contain fewer lists, code blocks, inline code spans, images, and URLs than 1,149 general AI GitHub repositories, although they contain more tables on average. Wilcoxon tests with $\rightarrow$ 2 indicate that LLM4Code READMEs systematically omit lists, examples, and metadata. YAML front matter is also sparse: 220 of 366 models omit any license field, only 68 specify datasets:, and only 2 specify base_model: (Yang et al., 2024).

Licensing is similarly distinctive. Among the 146 models declaring a single license, Apache-2.0 accounts for 25.23%, RAIL for 21.10%, BSD-3-Clause for 21.10%, MIT for 16.06%, and the AI2 Impact License for 2.19% (8 models). The heavy use of AI-specific licenses and the 60% non-declaration rate distinguish LLM4Code from typical open-source repositories (Yang et al., 2024).

3. Task landscape, benchmarks, and measured performance

The benchmark landscape reflects the breadth of LLM4Code usage. For code generation, representative benchmarks include HumanEval, HumanEval+, HumanEval-XL, mHumanEval, MBPP, MBPP+, MultiPL-E, and DS-1000. For understanding-oriented tasks, commonly used benchmarks include CodeSearchNet, ETH Py150, CoSQA, BigCloneBench, and GREAT, with task-specific metrics such as pass@k, BLEU, CodeBLEU, MRR, F1, and accuracy (Raihan et al., 2024).

Performance reported in survey form is heterogeneous across model families and task types. On HumanEval pass@1, GPT-4 is reported at approximately 85%, Claude 3 at approximately 82%, Code-LLaMA 70B at approximately 77%, StarCoder 15B at approximately 34%, and SantaCoder 1.1B at approximately 15%. On MBPP pass@1, GPT-4 is reported at approximately 83%, while Codestral 22B and Mistral 123B are both reported at approximately 86%. For PL $\rightarrow$ 3 NL summarization on ETH Py150, CodeBERT attains BLEU-4 of 17.83 and TreeBERT 20.49. For code search on CodeSearchNet, MRR rises from 0.64 with CodeBERT to 0.68 with GraphCodeBERT and 0.70 with SynCoBERT. Clone detection on BigCloneBench is reported in the range of approximately 0.94–0.97 F1, while defect detection on GREAT reaches approximately 0.86–0.89 accuracy (Raihan et al., 2024).

Code explanation highlights a recurrent pattern: code-specialized models outperform generic chat models even when the latter are larger. On the IRSE benchmark of 100 Python snippets with human-written explanations, CodeLlama-13B-Instruct and Llama-2-Coder-7B achieve BLEU1 of 0.189 in zero-shot evaluation, whereas Llama-2-70B-Chat achieves 0.019. The same study reports that code-specific LLMs achieve 8–10 $\rightarrow$ 4 higher BLEU1 than the generic Llama-2-70B, and that few-shot prompting often underperforms zero-shot under distribution mismatch between CoNaLa-style examples and the IRSE test distribution (Bhattacharya et al., 2023).

These results indicate that benchmark choice materially changes what “performance” means in LLMs4Code. A model that is strong on NL $\rightarrow$ 5 PL generation may not be correspondingly strong on explanation, retrieval, or semantic judgment, and the reverse can also hold.

4. Code comprehension, robustness, and explainability

A central controversy in LLMs4Code concerns the relation between code generation proficiency and genuine code understanding. A large-scale debugging study evaluates nine LLMs on 600,010 debugging tasks derived from 670 Java and 637 Python programs, using fault localization as a proxy for deep code comprehension. Mean Phase I fault-localization accuracy is approximately 37% for Python and approximately 23% for Java. However, when semantic-preserving mutations are applied to programs for which Phase I localization succeeded, LLMs fail under those mutations in 81% of cases, corresponding to a robustness ratio of approximately 19%. Faults in the first 25% of lines are localized approximately twice as often as faults in the last 25%, and multimodal/chat models outperform pure coding/reasoning models by approximately 20 percentage points (Haroon et al., 6 Apr 2025).

The reported interpretation is that current LLMs rely heavily on lexical and syntactic cues rather than durable semantic representations. Misleading comments, misleading variable names, dead-code insertion, and cumulative perturbations all degrade performance substantially, and the study explicitly attributes this brittleness to lexical/syntactic cues, tokenization bias, and context drift (Haroon et al., 6 Apr 2025).

Benchmark design has responded accordingly. A code-comprehension benchmark introduces six tasks—Subjectivity Grading Task, Code Question-Answering Task, Code Search Task, Test Case Task, Bug Fix Task, and Code Comparison Task—to probe semantic analysis rather than surface-level next-token competence. On the Subjectivity Grading Task, QWQ-32B improves from 70.00% to 83.47% after fine-tuning, Granite-8B improves from 68.00% to 76.55%, and the DPO-fine-tuned Codestral-22B reaches the highest micro-accuracy of 87.66% (Havare et al., 14 Jul 2025).

Explainability work has paralleled this shift from token-level outputs to structure-aware analysis. ASTxplainer aligns normalized token logits to AST nodes through an alignment function $\rightarrow$ 6 and aggregates them into per-node scores $\rightarrow$ 7, enabling syntax-level evaluation, causal analysis, and visualization. In a study over 12 decoder-only models on approximately 50,000 Python functions, codegen-2B-mono attains the highest global syntax-level evaluation score at 0.84, and identifier nodes show the largest negative Average Treatment Effect on cross-entropy for several models, indicating that poor identifier handling is a major driver of loss (Palacio et al., 2023).

Intervention studies also target comprehension directly. ProCURE combines counterfactual code augmentation with concept-aware tuning based on token masks and attention constraints. Averaged across models, it yields a gain of +0.7 percentage points in Pass@1, +4.7 percentage points in Pass@5 over standard fine-tuning, and a +10.47 percentage-point gain in Concept Consistency Score, with a total +18.77 percentage-point gain over the pretrained model (Ren et al., 18 Aug 2025).

Taken together, these findings argue against equating high pass@k on generation benchmarks with semantic competence. They also suggest that future progress in LLMs4Code depends on representations that are invariant to semantics-preserving perturbations and sensitive to data flow, control flow, and program structure.

5. Adaptation, repair, and model maintenance

Much of the technical work on LLMs4Code after pretraining concerns how to adapt models to subtle bugs, outdated knowledge, and incorrect code behaviors without full retraining. Code Comparison Tuning (CCT) is a representative example. It augments instruction tuning with token-level preference loss over correct versus corrupted code tokens and sequence-level comparison samples that explicitly pair buggy code with corrected code. On HumanEvalFix, CCT-StarCoder improves pass@1 from 33.7 to 38.3, while CCT-CodeLlama improves from 43.5 to 47.7. An ablation on CodeLlama-Python-13B reports 43.53±0.49 for instruction tuning, 45.76±0.23 for sequence-level data only, and 47.71±0.39 for full CCT, indicating that both granular token ranking and sequence-level comparison contribute to the gain (Jiang et al., 2024).

Hotfixing addresses a related but broader maintenance problem: rapidly steering a released model away from undesired behaviors while preserving other capabilities. In this formulation, training tokens are partitioned into desired tokens $\rightarrow$ 8, undesired tokens $\rightarrow$ 9, and other tokens $\rightarrow$ 0, and the update jointly pursues three objectives: learn desired behaviors, unlearn undesired behaviors, and retain knowledge of other code. Across CodeGen models, the best setting reported is LoRA with Dual Loss. For CodeGen-350M, buggy completions decrease by 44.32% and fixed completions increase by 108.42%; for CodeGen-2B, buggy completions decrease by 50.47% and fixed completions increase by 103.38%. With a KL-based retention term, Wilcoxon tests on HumanEval pass@100 show no statistically significant drop relative to the original model ( $\rightarrow$ 1) (Yang et al., 2024).

Model editing pushes this logic from behavior-level steering to targeted knowledge correction. CLMEEval introduces CoNaLa-Edit for NL $\rightarrow$ 2 PL editing and CodeSearchNet-Edit for PL $\rightarrow$ 3 NL editing, and evaluates six editing techniques on CodeLlama-7B, CodeQwen1.5-7B, and Stable-Code-3B. On CodeLlama-7B for CoNaLa-Edit, GRACE reaches 93.67% effectiveness EM and 92.89% specificity EM but only 0.03% generalization EM; A-GRACE preserves 93.57% effectiveness while increasing generalization to 64.54%, albeit with lower specificity at 37.46%. On CodeSearchNet-Edit, A-GRACE reaches 99.69% effectiveness and 97.05% generalization with 91.15% specificity. The general conclusion is that external memorization-based editing has strong effectiveness and specificity, whereas generalization across semantically equivalent inputs remains a universal challenge (Li et al., 2024).

These adaptation methods indicate that post-pretraining maintenance in LLMs4Code is becoming its own subfield. Fine-tuning, hotfixing, and editing are no longer interchangeable labels: they correspond to different granularities of change, different preservation requirements, and different assumptions about the locality of the knowledge to be updated.

6. Multilingual and cross-language code intelligence

Multilingual LLMs4Code are defined by two linked capabilities: multilingual code generation from shared natural-language requirements and multilingual code translation that preserves semantics across languages. Formally, generation learns a mapping $\rightarrow$ 4, while translation learns $\rightarrow$ 5, with semantic preservation checked by equivalence of execution outcomes (Jiang et al., 27 Apr 2026).

Recent surveys identify four main paradigms for eliciting multilingual competence: prompt engineering, multilingual pretraining and fine-tuning, multi-agent collaborative frameworks, and retrieval-augmented generation. Benchmarks include HumanEval-XL, mHumanEval, and PSEUDOEVAL for generation, and XLCoST, CodeTransOcean, and RepoTransBench for translation and repository migration. The empirical picture is strongly shaped by high-resource versus low-resource gaps: on Python, GPT-4 reaches Pass@1 of approximately 70–75% and CodeBLEU of approximately 0.65, while on Rust and OCaml the same models drop to Pass@1 of approximately 30–40% and CodeBLEU of approximately 0.35. In XLCoST, Java $\rightarrow$ 6 C++ exceeds 85% Computational Accuracy, whereas Python $\rightarrow$ 7 Rust is approximately 45%; in RepoTransBench, JavaScript $\rightarrow$ 8 TypeScript reaches approximately 90% build success, while C $\rightarrow$ 9 Rust is approximately 55% (Jiang et al., 27 Apr 2026).

Cross-language maintenance is not limited to whole-program translation. Codeditor reframes multilingual co-evolution as edit translation. It represents changes as explicit edit sequences, concatenates source-language edits, target-language old code, and source-language new code, and trains an encoder–decoder model to generate the target-language edit sequence. On 6,613 aligned Java–C# changes mined from eight pairs of open-source projects, Codeditor achieves approximately 67.2% exact-match accuracy on the time-segmented test split, compared with approximately 60.4% for CodeT5Up; a hybrid of edit translation and CodeT5Up reaches approximately 71.8% (Zhang et al., 2023).

Cross-lingual evaluation also reveals limits. In clone detection, prompt-based LLMs can achieve F1 up to 0.99 for straightforward examples, but they do not necessarily understand the meaning of “code clones” in a cross-lingual setting. On XLCoST and CodeNet, a basic classifier trained on cross-lingual code embeddings outperforms all evaluated LLMs by approximately 1 and approximately 20 percentage points, respectively, suggesting that representation alignment can matter more than generative reasoning for some non-generative code tasks (Moumoula et al., 2024).

The broader lesson is that multilingual LLMs4Code are not merely monolingual models with wider tokenizer coverage. Cross-language generalization is constrained by data imbalance, paradigm shifts such as Rust’s ownership model, and the gap between language translation and edit propagation in real repositories.

7. Privacy, deployment, and practitioner-facing constraints

The deployment of LLMs4Code raises privacy concerns because training corpora can contain sensitive data such as credentials, e-mail addresses, and personally identifiable information. Quantization has recently been studied not only as a compression technique but also as a factor in privacy exposure. Across Pythia, GPT-Neo, and CodeGen families, 8-bit static quantization yields a CodeBLEU drop under 1.5% for all Pythia sizes and reduces membership-inference effectiveness by approximately 0.1–0.3% AUC, while 4-bit static quantization produces larger privacy reductions—up to -10.01% ROC_AUC in one Pythia setting—at the cost of larger task degradation, with an average CodeBLEU drop of -3.15%. The same study reports a positive correlation between task performance and privacy risk, with Pearson $\rightarrow$ 0 in full precision and $\rightarrow$ 1 when aggregating all precisions (Haque et al., 31 Jul 2025).

Machine unlearning targets privacy leakage more directly. Evaluated on AIXCoder-7B, CodeLlama-7B, and CodeQwen-7B, gradient-ascent-based unlearning methods substantially reduce leak rate while preserving code generation capability. For CodeLlama-7B, leak rate falls from 28.6% in zero-shot to 5.6% with GA+GD, while Pass@1 moves from 69.3% to 68.8%. The same study reports that unlearning shifts leakage from direct leakage toward indirect leakage, meaning that targeted fields may be suppressed while associated attributes from the same profile can still appear (Geng et al., 9 Feb 2025).

Deployment concerns are not confined to privacy. A practitioner study built a unified sandboxed platform around eight code-generation models and surveyed 60 software practitioners from 11 countries. Reported usability was high: 92% found the system easy to use, and 92% said it met or exceeded their code-generation expectations. GPT-4 o was chosen by 31 of 60 practitioners (52%) as the top model, while GPT-3.5 Turbo was widely rated the worst. Practitioners also argued that widely used academic benchmarks emphasize synthetic, function-level tasks and overlook context propagation across modules, maintainability, integration with existing codebases and libraries, and error-message clarity (Rasheed et al., 28 Jan 2025).

These findings suggest that the mature evaluation of LLMs4Code must combine algorithmic, legal, and operational dimensions. Compression, unlearning, documentation quality, license declarations, and project-level usability are not peripheral issues; they are part of the technical definition of whether an LLM4Code system is reliable enough for software-engineering practice.