CodePlan: Repository-level Coding using LLMs and Planning (2309.12499v1)

Abstract: Software engineering activities such as package migration, fixing errors reports from static analysis or testing, and adding type annotations or other specifications to a codebase, involve pervasively editing the entire repository of code. We formulate these activities as repository-level coding tasks. Recent tools like GitHub Copilot, which are powered by LLMs, have succeeded in offering high-quality solutions to localized coding problems. Repository-level coding tasks are more involved and cannot be solved directly using LLMs, since code within a repository is inter-dependent and the entire repository may be too large to fit into the prompt. We frame repository-level coding as a planning problem and present a task-agnostic framework, called CodePlan to solve it. CodePlan synthesizes a multi-step chain of edits (plan), where each step results in a call to an LLM on a code location with context derived from the entire repository, previous code changes and task-specific instructions. CodePlan is based on a novel combination of an incremental dependency analysis, a change may-impact analysis and an adaptive planning algorithm. We evaluate the effectiveness of CodePlan on two repository-level tasks: package migration (C#) and temporal code edits (Python). Each task is evaluated on multiple code repositories, each of which requires inter-dependent changes to many files (between 2-97 files). Coding tasks of this level of complexity have not been automated using LLMs before. Our results show that CodePlan has better match with the ground truth compared to baselines. CodePlan is able to get 5/6 repositories to pass the validity checks (e.g., to build without errors and make correct code edits) whereas the baselines (without planning but with the same type of contextual information as CodePlan) cannot get any of the repositories to pass them.

Repository-level Code Editing with CodePlan

The paper presents CodePlan, a sophisticated framework developed to address repository-level coding tasks. These tasks include complex activities at the repository level such as package migration and temporal code edits, which involve interdependent modifications across the entire codebase. Traditional tools like GitHub Copilot, while adept at localized coding problems, fall short in handling such tasks where the scope extends beyond individual files. CodePlan provides an innovative approach by reframing repository-level coding as a planning problem that organizes and orchestrates a sequence of interdependent code changes.

Core Methodology: CodePlan

CodePlan is engineered around three principal elements: incremental dependency analysis, change may-impact analysis, and an adaptive planning algorithm. The primary objective is to effectively navigate the challenges posed by repository-level tasks by synthesizing a multi-step edit chain. Each edit in the chain results from calling an LLM with contextual information derived from the repository state, historical changes, and task-specific guidance. By constructing a plan graph, CodePlan methodically identifies and executes code modifications necessary to transition a repository from an initial to a target state, consistent with correctness conditions defined by an oracle.

Evaluation and Results

CodePlan was evaluated against two repository-level tasks: package migration for C# and temporal code edits for Python, applying it to multiple repositories. The evaluation demonstrates that CodePlan significantly outperforms simpler approaches, such as those solely relying on build or static checking tools for error detection and correction, commonly referred to as oracle-guided repair techniques. Importantly, CodePlan managed to lead 5 out of 6 repositories to pass the defined validity checks, showcasing the framework's robustness and efficacy. The empirical results indicate CodePlan's superiority not just in achieving correctness akin to the ground truth, but also in handling complex, interdependent coding tasks, which have proven challenging with previous methodologies.

Technical Contributions

1. Problem Formalization: The paper is pioneering in formalizing repository-level coding tasks using LLMs, emphasizing the need to analyze and propagate code changes across a repository.
2. Planning as a Solution: By conceiving repository-wide coding as a planning problem, CodePlan presents a task-agnostic framework that leverages static analysis for planning, thereby ensuring the effective propagation of changes and maintaining code repository consistency.
3. Experimental Insights: The experiments show CodePlan achieving a closer match to ground truth edits compared to competing techniques, with specific numerical evidence highlighting its success rate in passing repository validity checks.

Implications and Future Directions

Practically, CodePlan holds significant implications for large-scale software development, notably in tasks requiring holistic changes across codebases like library migrations or consolidating widespread API changes. From a theoretical perspective, it underscores the value of combining static analysis with LLM capabilities and adaptive planning to enhance automated code modification techniques.

Future investigations could expand CodePlan's utility across additional programming languages and types of software artifacts. Addressing the challenge of incorporating dynamic dependencies found in multi-threading or distributed systems will further solidify its applicability in enterprise environments. There is also potential to enhance the mechanism employed for context provision to LLMs, which might involve few-shot learning or other advanced machine learning techniques.

In summary, CodePlan represents a significant advancement in repository-level code automation, not only streamlining the developer workflow but also setting a foundation for future research in automated software engineering.