CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation (2109.00859v1)

Abstract: Pre-trained models for Natural Languages (NL) like BERT and GPT have been recently shown to transfer well to Programming Languages (PL) and largely benefit a broad set of code-related tasks. Despite their success, most current methods either rely on an encoder-only (or decoder-only) pre-training that is suboptimal for generation (resp. understanding) tasks or process the code snippet in the same way as NL, neglecting the special characteristics of PL such as token types. We present CodeT5, a unified pre-trained encoder-decoder Transformer model that better leverages the code semantics conveyed from the developer-assigned identifiers. Our model employs a unified framework to seamlessly support both code understanding and generation tasks and allows for multi-task learning. Besides, we propose a novel identifier-aware pre-training task that enables the model to distinguish which code tokens are identifiers and to recover them when they are masked. Furthermore, we propose to exploit the user-written code comments with a bimodal dual generation task for better NL-PL alignment. Comprehensive experiments show that CodeT5 significantly outperforms prior methods on understanding tasks such as code defect detection and clone detection, and generation tasks across various directions including PL-NL, NL-PL, and PL-PL. Further analysis reveals that our model can better capture semantic information from code. Our code and pre-trained models are released at https: //github.com/salesforce/CodeT5 .

This paper is about a new deep learning model designed to understand and generate computer programs. The model is built on an encoder-decoder framework and is called CodeT5. In simple terms, it learns to read code (as well as accompanying natural language comments) and then use that learning for tasks like summarizing what a piece of code does, generating new code based on a description, translating code from one programming language to another, and even spotting defects in code.

Background and Motivation

The work draws inspiration from models used in NLP such as BERT and GPT. Those models were very successful in understanding human language, and researchers wanted to see if similar ideas could be applied to programming languages. However, code has its own special characteristics. For example, in code, variable names and function names—called identifiers—often carry a lot of meaning. CodeT5 is specifically designed to pay attention to these identifiers so that it understands the code’s meaning better.

Main Contributions

The key ideas in CodeT5 include:

• Unified Approach: CodeT5 uses a single model for both understanding and generating code. Unlike some earlier approaches that either focused only on the understanding part (encoding) or on generating outputs (decoding), this model handles both tasks within one framework.
• Identifier-Aware Pre-training: The model is trained not only by trying to predict missing parts of code (as in standard denoising tasks) but also by learning to recognize and recover identifiers. Since identifiers are important in conveying what the code does, this helps the model capture the structure and meaning of programs even more accurately.
• Dual Generation with Bimodal Data: Many codebases come with code comments. The model is additionally trained to convert code into natural language descriptions and vice versa. This dual training helps the model understand how programming languages and human language relate to each other.

How the Model Works

1. Input Representation:
   • The model takes either a code snippet by itself or a combination of code and natural language (NL) comments.
   • For inputs that include both, the text and code are combined into one sequence with a special delimiter. This allows the model to learn the alignment between the code and its description.
   • The code part is also processed to mark the type of each token – for example, indicating whether a token is an identifier or some other piece of code.
2. Pre-training Tasks:

CodeT5 is pre-trained using several tasks designed to help the model learn different aspects of code: - Masked Span Prediction (MSP): Like many NLP models, it randomly hides parts of the sequence and trains the model to predict these missing pieces. In this case, the hidden pieces can be parts of code. - Identifier Tagging (IT): The model is taught to label each token in the code as being an identifier (like variable or function names) or not. This helps it learn which parts of the code are especially important. - Masked Identifier Prediction (MIP): All identifiers are masked and replaced with a placeholder, and the model must predict the correct identifier names. This is harder than the usual span prediction because correct identifiers are key to understanding the code’s intent. - Bimodal Dual Generation: When the model sees paired code and comments, it is trained in both directions—generating code from a description and generating a description from code. This training strategy helps align how code and natural language relate.

1. Fine-tuning on Downstream Tasks:

After pre-training, CodeT5 is adjusted (fine-tuned) for specific tasks. This includes tasks like: - Code Summarization: Generating a natural language explanation from a piece of code. - Code Generation: Creating code given a natural language description. - Code Translation and Refinement: Converting code from one programming language to another or correcting buggy code. - Understanding Tasks: For example, predicting whether a code snippet has a defect or checking if two pieces of code perform the same function.

The same model can be fine-tuned for multiple tasks by providing a special “prompt” that tells it what to do, such as “Translate Java to C#:” at the beginning of the input.

Experimental Results and Findings

The experiments in this work show that CodeT5 outperforms several previous models on a wide range of tasks:

• On tasks that involve generating code or summaries, CodeT5 produces outputs that more accurately capture both the syntax and semantics of the code.
• For tasks requiring code understanding—like defect detection or clone detection—the model also performs very well.
• The identifier-aware training methods help the model especially in tasks where knowing the exact names of variables or functions is important.

Technical Insights and Practical Considerations

• Pre-training Data: CodeT5 is trained on millions of code snippets from various programming languages. The model also uses specialized techniques to ensure that code tokens are efficiently processed, reducing the number of tokens and helping the model focus on important context.
• Tokenization: A custom tokenizer is used for code, which better preserves the structure of programming languages than standard text tokenizers. This is important because the proper handling of symbols like braces or parentheses is crucial in code.
• Balanced Multi-task Learning: When the same model is used for more than one task during fine-tuning, balanced sampling is applied so that tasks with more data do not overwhelm those with less data.

Conclusion

The paper introduces CodeT5, a unified model that intelligently bridges the gap between code and natural language. By paying attention to identifiers and employing multiple pre-training tasks, the model can better understand and generate code. This research not only advances the field of automated code analysis and generation but also opens up practical applications—such as helping developers write code faster or aiding in automated code maintenance.

Overall, the work represents a significant step toward more effective machine learning techniques tailored for programming languages, combining ideas from natural language processing with insights specific to code.