Fixing Rust Compilation Errors using LLMs (2308.05177v1)

Abstract: The Rust programming language, with its safety guarantees, has established itself as a viable choice for low-level systems programming language over the traditional, unsafe alternatives like C/C++. These guarantees come from a strong ownership-based type system, as well as primitive support for features like closures, pattern matching, etc., that make the code more concise and amenable to reasoning. These unique Rust features also pose a steep learning curve for programmers. This paper presents a tool called RustAssistant that leverages the emergent capabilities of LLMs to automatically suggest fixes for Rust compilation errors. RustAssistant uses a careful combination of prompting techniques as well as iteration with an LLM to deliver high accuracy of fixes. RustAssistant is able to achieve an impressive peak accuracy of roughly 74% on real-world compilation errors in popular open-source Rust repositories. We plan to release our dataset of Rust compilation errors to enable further research.

Fixing Rust Compilation Errors using LLMs

The paper addresses the challenge of fixing Rust compilation errors using LLMs, presenting a tool named RustAssistant. With Rust's increasing popularity due to its safety features, tackling compilation errors, especially for new adopters, has become essential. This work leverages LLMs to provide automated solutions for these errors.

Key Contributions

The authors introduce RustAssistant, a novel tool designed to resolve compilation errors in Rust. The tool combines several strategies such as prompt engineering, error localization, and iteration with LLMs, specifically utilizing models like GPT-3.5 and GPT-4. RustAssistant achieves a peak accuracy of 74% on real-world errors from popular open-source Rust repositories.

Methodology

RustAssistant operates through a meticulous process:

• Prompt Construction: The tool constructs detailed prompts including error messages and relevant code snippets. These are crafted to help the LLM generate precise fixes.
• Iteration with LLMs: It iteratively interacts with the LLM to apply and verify suggested fixes. The process continues until the code successfully compiles or the system determines that progress is halted.
• Change Log Format: A specific format is used to record changes, ensuring that the LLM focuses on key modifications.

Dataset and Evaluation

The dataset comprises three categories:

• Micro-benchmarks: Derived from 270 Rust error codes, these are smaller programs designed to trigger specific compilation errors.
• Stack Overflow Snippets: 50 real-world examples focusing on memory and thread safety errors.
• Top-100 Crates: Compilation errors from top Rust libraries on GitHub.

Evaluation results are promising, with RustAssistant fixing 92.59% of micro-benchmarks and 73.63% of GitHub commits. Notably, GPT-4 outperforms GPT-3.5, highlighting the benefits of advanced LLMs. The paper also shows that RustAssistant outperforms Clippy's auto-fix with a 75% success rate on non-compilation linting errors.

Implications and Future Work

The findings suggest that LLMs, when combined with effective prompt strategies and iteration mechanisms, can substantially aid in fixing code errors. This approach emphasizes the potential for LLMs to automate not only compilation error fixes but also tackle other areas of software maintenance.

Future work could explore integrating few-shot learning or additional context for improved accuracy. Further, expanding the dataset could enhance the tool's robustness across diverse scenarios.

The paper establishes a foundation for employing LLMs in the field of automated code repair, specifically within the Rust ecosystem, offering significant promise for development efficiency and error resolution.