Large Language Models for Code

• LLM4Code are specialized transformer models trained on large-scale code corpora, enabling tasks such as generation, summarization, and fault localization.
• Methodologies like semantic-preserving mutations and counterfactual code augmentation enhance deep code understanding and robustness under challenging transformations.
• Research addresses efficiency with model compression and green AI while tackling challenges in privacy, interpretability, and integration into real-world workflows.

LLMs for Code (LLM4Code) are a specialized class of transformer-based models trained on large-scale source code corpora and associated natural language artifacts (comments, docstrings, specifications). These models have rapidly become central to software engineering automation, enabling tasks such as code generation, summarization, explanation, repair, testing, code search, and translation. The research literature on LLM4Code spans methodology, ecosystem analysis, evaluation, interpretability, robustness, privacy, and efficiency, with increasing focus on the semantic depth, trustworthiness, and operational integration of these models in real-world workflows.

1. Methodologies for Measuring and Enhancing Code Understanding

A fundamental research front in LLM4Code is the quantification and improvement of “code comprehension”: the ability to model program semantics rather than syntactic or lexical patterns. Traditional code generation benchmarks (e.g., HumanEval, CodeSearchNet) primarily test sequence-to-sequence synthesis, but recent work emphasizes tasks probing deep semantics:

• Fault localization as a code understanding proxy: The methodology pioneered in “How Accurately Do LLMs Understand Code?” involves systematically injecting single-line bugs (off-by-one, misplaced return, boolean inversion, operator swap) into real Java and Python programs, then requiring the LLM to localize the injected fault given the code and a natural language specification. Crucially, after successful localization, authors apply a battery of semantic-preserving mutations (SPMs)—dead code insertion, misleading comments, identifier renaming, function definition shuffling—in order to stress-test the model’s reliance on non-semantic cues. LLMs on these tasks exhibit severe robustness collapse: on programs that a model could previously debug, SPMs induce a loss of bug-finding ability in 78% of cases (up to 83% in certain conditions). The effect is universal across both state-of-the-art closed-source models (Gemini 1.5 Pro, GPT-4o) and top open-source models (Llama 3.1, Qwen-QWQ) (Haroon et al., 6 Apr 2025).
• Counterfactual code augmentation and concept-aware tuning: Beyond fault localization, frameworks such as ProCURE automate the generation of concept-oriented, semantics-preserving code perturbations—including if-else flipping, def-use chain breaking, independent statement swaps, and identifier randomization—combined with masked loss functions that explicitly focus the model’s learning signal on the altered code regions and penalize confounding attention elsewhere. Empirically, this delivers consistent improvements in both pass@k generation metrics and semantic consistency scores across multiple LLM platforms (Ren et al., 18 Aug 2025).

The core insight is that existing LLMs for code function as powerful pattern matchers over lexical and surface-syntactic features, but lack stable representations of code execution or semantic invariants under transformation. Both code comprehension and bug-finding accuracy degrade rapidly under surface form changes that leave semantics invariant, highlighting the need for new training objectives, tokenization schemes, and representational architectures that explicitly capture program structure.

2. Model Architectures, Training Protocols, and Ecosystem Dynamics

The LLM4Code landscape is architecturally dominated by large-scale decoder-only transformers, with model sizes ranging from hundreds of millions (CodeGen-350M) to tens of billions (CodeLlama-70B, StarCoder-15B). Training pipelines broadly consist of:

• Pretraining: Next-token prediction on massive deduplicated corpora of source code mined from GitHub, StackOverflow, open-source datasets, with frequent mixing of natural language contexts (comments, docstrings). Tokenization is typically byte-pair encoding (BPE) or unigram subwords, not structurally aligned to code semantics.
• Instruction tuning and adaptation: Supervised fine-tuning on constructed instruction–response pairs, bug-fix corpora, Evol-Instruct or Self-Instruct datasets, and in some models, multi-task or fill-in-the-middle (FIM) objectives (Jiang et al., 28 Mar 2024, Di et al., 2023).

The ecosystem paper of Hugging Face models reveals a rapidly growing, highly skewed network of >366 transformer models for code and 73 code datasets (Yang et al., 27 May 2024). The model/dataset dependency structure displays a canonical power-law: a small set of foundational models and datasets are heavily reused for fine-tuning, quantization, architecture sharing, and distillation. Fine-tuning is by far the most common reuse pattern (47%), closely followed by architecture sharing (19%). Model documentation and licensing lag behind those of general NLP models, with >60% of models lacking formal license declarations and only 19% specifying training datasets. This indicates urgent need for metadata and documentation standardization to enable reproducible, legally compliant research.

3. Evaluation, Generalization, Robustness, and Non-Functional Properties

LLM4Code evaluation has diversified beyond pass@k generation accuracy to include a broad array of metrics capturing semantic robustness, privacy, explainability, and operational usability (Yang et al., 12 Mar 2024, Das et al., 4 Dec 2025).

• Code explanation and summarization: Specialized LLMs (CodeLlama, StarCoder, DeepSeekCoder) outperform generic LLMs (Llama-2-70B) on code-to-text tasks, confirmed by BLEU, ROUGE, and CodeBERTScore on CodeSearchNet, HumanEvalExplain, and IRSE datasets. Instruction tuning and infilling-based adaptation further improve code summarization (Szalontai et al., 29 May 2024, Bhattacharya et al., 2023).
• Testing and oracle reasoning: LLMs exhibit the ability to synthesize test cases for arbitrary Python functions, with correctness (pass rate) and coverage metrics sometimes exceeding the LLM’s solution synthesis rate itself. Fine-grained prompt and post-processing (self-generated tests, rank-weighted selection) reinforce the tight link between test-generation ability and synthesis accuracy (Xiong et al., 2023).
• Robustness and adversarial vulnerability: Taxonomies of adversarial attacks demonstrate that LLM4Code are brittle to word-level perturbations in comments, prompts, or code, with substantial drops in pass@1 and altered output sets. Word-level changes (variable renames, synonym swaps) are significantly more damaging than statement- or character-level ones. Models trained solely on code in one language (e.g., Python-only) outperform multilingual models in robustness (Liu et al., 9 Jun 2025).
• Non-functional properties: Empirical reviews synthesize metrics and methods for robustness (adversarial training, norm-bounded attack evaluation), privacy (membership inference, data extraction), efficiency (model size, FLOPs, inference latency), explainability (counterfactual analysis, rationales), usability (developer studies, productivity surveys), and security (backdoor and poisoning resistance) (Yang et al., 12 Mar 2024). Significant research gaps remain: large-scale adversarial training for 10–100B parameter models is computationally prohibitive, formal privacy guarantees are unestablished, and explainability methods are underdeveloped for generative code tasks.

4. Interpretability, Semantic Alignment, and Trust

Interpretability methods for LLM4Code are increasingly focused on post-hoc, causal, and concept-driven explanations of generation decisions.

• CodeQ rationales: This framework executes greedy search for minimal subsets of input tokens (“rationales”) sufficient to explain the generation of each output token, maps these to higher-level concepts (AST node types or parts-of-speech), and aggregates explanations across datasets. Global heatmaps reveal that LLMs often over-rely on low-value code features (indentation, punctuation) rather than truly causative concepts, exposing an “overinterpretation” phenomenon (Palacio et al., 21 Mar 2025). User studies confirm high perceived usefulness for debugging and prompt engineering, but alignment of model and human rationales remains low.
• Graph-structural infusion: GALLa introduces auxiliary graph alignment during fine-tuning by integrating the outputs of a Graph Neural Network (processing AST/DFG representations) into the LLM’s embedding space as auxiliary tokens. The key is that graph-structured data is used only during training (alignment and cross-modal generation tasks); inference remains graph-free. Multi-task evaluation shows consistent (>2–36%) improvements in code summarization, translation, and clone detection, with the largest gains accruing for smaller models (Zhang et al., 6 Sep 2024).
• Semantic adaptation and counterfactual loss: When fine-tuning on code comprehension, explicit loss masks over concept-altered tokens (from counterfactual code augmentation) improve both task accuracy and semantic consistency, surpassing traditional instruction tuning (Ren et al., 18 Aug 2025).

These advances collectively signal a move toward LLMs operating with explainable, structure-aware, and semantically regularized prediction, as opposed to opaque, surface-level pattern discovery.

5. Privacy, Memorization, and Mitigation Strategies

Memorization and privacy leakage are critical concerns for LLM4Code trained on public code corpora.

• Empirical memorization and extraction attackability: Experiments demonstrate that code LLMs (e.g., CodeGen-Mono-16B) memorize and regurgitate ~47% of code substrings marked as “extractable” with simple k-token prefix attacks; exact-match memorization scales logarithmically with parameter count. Category-specific analysis shows that data-carrier code (e.g., embedded datasets, keys) is more vulnerable than regular code (Al-Kaswan et al., 2023).
• Causal modeling of PII leakage: Fine-grained analysis links ease-of-learning (measured via training dynamics: confidence μ and variability σ per PII token) to the empirical risk of leakage under black-box extraction attacks. Easy-to-learn types (e.g., IP addresses) leak at 25–35%, while “hard” keys and passwords leak <8%. Causal ATE estimates confirm that moving from “easy” to “hard” directly reduces leakage; ambiguous tokens are context-sensitive. Defense recommendations include targeted PII scrubbing, learnability-aware differential privacy, and adaptive injection of decoy secrets (Yang et al., 8 Dec 2025).
• Hotfixing as privacy and reliability mitigation: Instead of global retraining, hotfixing leverages parameter-efficient tuning (LoRA, dual loss) over small code diffs to unlearn memorized leaks or buggy-code completions, simultaneously penalizing undesired continuations and preserving overall model utility. This process achieves >80% reduction in private-token exposure within minutes and negligible impact on canonical pass@k metrics (Yang et al., 11 Aug 2024).

These results underscore the need for dynamic, targeted, and type-aware privacy mitigation strategies as models and codebases evolve.

6. Efficiency, Green AI, and Model Compression

Deployment of LLM4Code in “edge” and on-premises scenarios necessitates radical model compression and energy savings.

• Avatar multi-objective model optimization: Avatar tailors student models via a Pareto optimization over model size (<3 MB), inference latency, energy, and carbon, subject to constrained accuracy loss (≤2%) (Shi et al., 2023). The framework employs SMT solvers to prune feasible configuration spaces (layers, head count, hidden size), genetic search for Pareto-optimal configurations, and knowledge distillation on task-labeled data. Distilled students (e.g., CodeBERT-Avatar) offer 160x compression, 76x latency speedup, 184x energy savings, and <2% drop in vulnerability and clone detection accuracy compared to baseline. This establishes a new frontier for “green” LLM4Code absent in early generations of deployment.
• PEFT, quantization, and distillation: Parameter-efficient fine-tuning (LoRA, QLoRA, IA^3), 4/8-bit quantization, and distillation to tiny models (Compressor, CodeBERT-3MB, etc.) are now widespread; however, the field needs systematic evaluation of the side effects of such compression on semantic robustness, privacy, and trustworthiness (Yang et al., 27 May 2024).

7. Current Limitations and Future Directions

LLM4Code research faces several persistent limitations and open challenges:

• Existing LLMs exhibit brittle, primarily surface-based code modeling; truly semantic comprehension remains inadequate as measured by fault-localization robustness and concept consistency (Haroon et al., 6 Apr 2025, Ren et al., 18 Aug 2025, Havare et al., 14 Jul 2025).
• Training and evaluation pipelines must evolve to use dynamically generated, contamination-robust benchmarks with semantic mutation and adversarial augmentation, moving away from fixed static datasets.
• New architectures are needed that admit hybrid representations: incorporating AST, DFG, or execution traces directly into learned embeddings or interleaving symbolic with neural computation, without compromising scale or training tractability.
• Formally certified privacy (ε,δ-differential privacy), robustness (norm-bounded certified training), and explainability for generative tasks remain largely unaddressed in code contexts (Yang et al., 12 Mar 2024).
• Practical deployment demands improved documentation, licensing transparency, and UI/UX affordances for uncertainty, debugging, and human-in-the-loop correction.

Future LLM4Code work will likely center on hybrid neuro-symbolic architectures, graph- or tree-aligned input/output representations, dynamic and adaptive training objectives (rewarded on downstream measurement oracles), and a holistic, multi-objective perspective on safety, cost, efficacy, and human integration.