SkCoder: A Sketch-based Approach for Automatic Code Generation (2302.06144v4)

Abstract: Recently, deep learning techniques have shown great success in automatic code generation. Inspired by the code reuse, some researchers propose copy-based approaches that can copy the content from similar code snippets to obtain better performance. Practically, human developers recognize the content in the similar code that is relevant to their needs, which can be viewed as a code sketch. The sketch is further edited to the desired code. However, existing copy-based approaches ignore the code sketches and tend to repeat the similar code without necessary modifications, which leads to generating wrong results. In this paper, we propose a sketch-based code generation approach named SkCoder to mimic developers' code reuse behavior. Given a natural language requirement, SkCoder retrieves a similar code snippet, extracts relevant parts as a code sketch, and edits the sketch into the desired code. Our motivations are that the extracted sketch provides a well-formed pattern for telling models "how to write". The post-editing further adds requirement-specific details to the sketch and outputs the complete code. We conduct experiments on two public datasets and a new dataset collected by this work. We compare our approach to 20 baselines using 5 widely used metrics. Experimental results show that (1) SkCoder can generate more correct programs, and outperforms the state-of-the-art - CodeT5-base by 30.30%, 35.39%, and 29.62% on three datasets. (2) Our approach is effective to multiple code generation models and improves them by up to 120.1% in Pass@1. (3) We investigate three plausible code sketches and discuss the importance of sketches. (4) We manually evaluate the generated code and prove the superiority of our SkCoder in three aspects.

This paper proposes a new method for automatically generating source code that is inspired by the way developers reuse and modify existing code. The method, called SkCoder, is built around a three-step process that mimics human problem-solving: first, finding similar code; second, extracting the most useful parts of that code to form a “sketch”; and third, editing the sketch to produce the final, working code.

Background and Motivation

Developers often search for code snippets that solve similar problems and then modify those snippets to suit their needs. Traditionally, many automated systems try to copy parts of similar code directly, which can lead to mistakes when unnecessary parts are not removed or changed. The idea behind this approach is to more closely capture the human behavior of recognizing which parts of the similar code are useful and then adapting them. This not only helps ensure that the generated code is correct but also makes the code easier to understand and maintain.

Core Components of the Method

The method is divided into three main components:

1. Retriever
   • The system first takes a natural language description (a requirement) and searches through a large collection of existing code.
   • It uses a technique based on measuring the similarity between the description and the available code comments or documentation.
   • This component uses a well-known scoring method to find the code snippets that are most similar in content to what is required.
2. Sketcher
   • Once a similar code snippet is retrieved, not all of it is useful. The sketcher is responsible for picking out the parts of that code that are most relevant to the current programming need.
   • It operates by treating the code as a sequence of tokens (words or symbols) and deciding, for each token, whether it should be kept or replaced by a placeholder.
   • The chosen method for training the sketcher uses the longest common subsequence between the retrieved code and the target (desired) code. This means it finds the longest sequence of code elements that the retrieved code and the correct code have in common.
   • The resulting “sketch” highlights the structure and important parts of the code while leaving out details that are likely to be unnecessary or even misleading.
3. Editor
   • In the final step, an editing model takes the sketch and the natural language description to generate the complete code.
   • This component uses an encoder-decoder framework. The encoder processes the input (the description and the sketch), and the decoder generates the final code token-by-token.
   • The editor not only fills in placeholders with appropriate details (like variable names or specific conditions) but also adds any missing components that were not present in the sketch.
   • This step ensures that the resulting code not only follows a correct structure but also satisfies the exact requirements described by the user.

Training Strategy and Datasets

Because the process involves making hard decisions (such as whether to keep a token or replace it with a placeholder), the training is done in two stages:

• First, the sketcher is trained using examples where the common parts between retrieved and actual code are used as the “gold standard” for the sketch.
• Next, the editor is trained to generate the target code using the sketches produced by the first stage.

The method is evaluated on several datasets that include natural language descriptions paired with real code. Some datasets come from game development scenarios, while others are drawn from real software projects. In addition to comparing generated code against reference code using metrics like exact match and BLEU scores, performance is also tested using unit tests to check if the code runs correctly.

Key Observations and Results

• Improved Correctness: By first extracting a sketch that represents a well-formed pattern and then editing it, the proposed approach tends to generate code that is more exactly correct compared to methods that merely copy code.
• Better Use of Retrieved Code: The retrieval component helps bring in code that already has a sound structure, while the sketcher ensures that only relevant parts are used. As a result, the system avoids common mistakes that occur when irrelevant parts are copied directly.
• Robustness: The paper also shows that even when the retrieved code is not very similar to the target, the system can still pick out useful patterns and generate correct programs.
• Versatility Across Models: The approach has been tested with different underlying neural network architectures, including traditional sequence models and more recent pre-trained models, and it shows improvements across the board.

Conclusion

In summary, this paper introduces a sketch-based method for automatic code generation that more closely replicates how human developers reuse and edit code. The three-part strategy—retrieving similar code, extracting a code sketch, and editing the sketch based on precise requirements—leads to improvements in the generation of correct and maintainable code. This method highlights an important step forward in making automated programming tools that not only generate code but do so in a way that maintains clarity and correctness.