Multi-line AI-assisted Code Authoring (2402.04141v1)

Abstract: CodeCompose is an AI-assisted code authoring tool powered by LLMs that provides inline suggestions to 10's of thousands of developers at Meta. In this paper, we present how we scaled the product from displaying single-line suggestions to multi-line suggestions. This evolution required us to overcome several unique challenges in improving the usability of these suggestions for developers. First, we discuss how multi-line suggestions can have a 'jarring' effect, as the LLM's suggestions constantly move around the developer's existing code, which would otherwise result in decreased productivity and satisfaction. Second, multi-line suggestions take significantly longer to generate; hence we present several innovative investments we made to reduce the perceived latency for users. These model-hosting optimizations sped up multi-line suggestion latency by 2.5x. Finally, we conduct experiments on 10's of thousands of engineers to understand how multi-line suggestions impact the user experience and contrast this with single-line suggestions. Our experiments reveal that (i) multi-line suggestions account for 42% of total characters accepted (despite only accounting for 16% for displayed suggestions) (ii) multi-line suggestions almost doubled the percentage of keystrokes saved for users from 9% to 17%. Multi-line CodeCompose has been rolled out to all engineers at Meta, and less than 1% of engineers have opted out of multi-line suggestions.

• The paper demonstrates that transitioning from single-line to multi-line suggestions overcomes usability challenges with a scope-based algorithm.
• It introduces latency reduction techniques like Flash Attention and persistent K-V caching to enhance user experience.
• The study nearly doubled keystrokes saved, validating multi-line suggestions as a significant productivity booster in code completion.

Multi-line AI-assisted Code Authoring: An Evaluation of CodeCompose

The presented paper discusses the development and deployment of CodeCompose, an advanced AI-assisted code authoring tool used internally at Meta. This tool leverages the capabilities of LLMs to provide inline code suggestions to developers. The paper specifically focuses on the transition from single-line to multi-line suggestions, a transition that posed unique technical and usability challenges.

Overview of Challenges and Solutions

The shift from single-line to multi-line code suggestions introduced a number of challenges, which were meticulously addressed through innovative solutions. The main challenges identified were:

1. The Jarring Effect: Multi-line suggestions can alter the layout of the existing code, disrupting the developer's flow. To address this, the team implemented a scope-based algorithm ensuring that multi-line suggestions only trigger when the cursor is at the end of a scope. This approach minimizes intrusiveness and ensures seamless integration with the developer's workflow.
2. Latency and User Experience: The generation time for multi-line suggestions is significantly longer, leading to potential delays and decreased interaction rates. To mitigate this, several optimizations were introduced, including the use of Flash Attention and persistent key-value (K-V) caching. Moreover, an inline spinner was added to the interface, transparently indicating ongoing suggestion processing, thereby managing user expectations and enhancing the perceived responsiveness.
3. Effectiveness and Measurement in Production: Evaluating the impact of multi-line suggestions involved tracking key metrics such as acceptance rate and keystrokes saved. Though initial implementation showed a decrease in suggestion display, subsequent optimizations improved performance metrics. Multi-line suggestions accounted for 42% of accepted characters, despite being only 16% of the displayed suggestions, indicating a significant impact on developer productivity.

Quantitative Results and Implications

The paper highlights the profound efficacy of multi-line suggestions with quantitative data: multi-line suggestions nearly doubled the percentage of keystrokes saved (from 9% to 17%). This finding emphasizes the substantial productivity gains that can be derived from properly optimizing AI-assisted code suggestions.

Such results indicate that multi-line AI-assisted code authoring can lead to more advanced levels of code completion, supporting developers in writing boilerplate code and discovering best practices and APIs. These improvements pave the way for more sophisticated techniques in machine learning-based code suggestion systems, potentially bringing LLMs into mainstream code authoring environments.

Future Directions

The results of this paper underscore the need for continued innovation in AI-assisted tools within software development environments. Future work could focus on further decreasing latency, expanding the contexts in which multi-line suggestions can be effectively applied, and exploring user feedback to refine model accuracy and suggestion usefulness.

The lessons learned in optimizing both user interface and model-serving infrastructure can guide future efforts to integrate AI capabilities into developer tools on a broader scale, highlighting both the technological and human-centric aspects critical to successful deployment.

In conclusion, the paper presents an insightful case paper on optimizing AI-assisted code authoring through thoughtful intersection of user-centric design principles, advanced machine learning techniques, and empirical validation. This sets a standard for future research and development in intelligent code completion tools.