A Large-Scale Survey on the Usability of AI Programming Assistants: Successes and Challenges (2303.17125v2)

Abstract: The software engineering community recently has witnessed widespread deployment of AI programming assistants, such as GitHub Copilot. However, in practice, developers do not accept AI programming assistants' initial suggestions at a high frequency. This leaves a number of open questions related to the usability of these tools. To understand developers' practices while using these tools and the important usability challenges they face, we administered a survey to a large population of developers and received responses from a diverse set of 410 developers. Through a mix of qualitative and quantitative analyses, we found that developers are most motivated to use AI programming assistants because they help developers reduce key-strokes, finish programming tasks quickly, and recall syntax, but resonate less with using them to help brainstorm potential solutions. We also found the most important reasons why developers do not use these tools are because these tools do not output code that addresses certain functional or non-functional requirements and because developers have trouble controlling the tool to generate the desired output. Our findings have implications for both creators and users of AI programming assistants, such as designing minimal cognitive effort interactions with these tools to reduce distractions for users while they are programming.

Insights into Developer Interaction with AI Programming Assistants: An Analysis

The paper, "A Large-Scale Survey on the Usability of AI Programming Assistants: Successes and Challenges," presents a comprehensive examination of developers' practices and usability challenges encountered while using AI programming assistants like GitHub Copilot. The authors conducted a survey involving 410 developers, tapping into a wide spectrum of coding backgrounds to highlight prevalent successes and persistent issues.

Key Observations and Findings

The core findings of the research illuminate several aspects of AI programming assistant usage:

1. Usage Patterns: The paper reports that a significant proportion of developers, particularly those using GitHub Copilot, attributed approximately 30.5% of their code to the assistant. These tools are heavily utilized for repetitive tasks, simple logic, and code autocompletion, underscoring their role in accelerating coding tasks rather than pioneering new solutions.
2. Motivations and Challenges: One major motivation is the significant reduction in development time, with tools effectively assisting in syntax recall and task completion. Conversely, the paramount challenges include the lack of control over the generated output and the generated code not meeting certain functional or non-functional requirements. Notably, the cognitive load imposed by understanding specific parts of code influenced by the tool and aligning them with developer expectations stands out as a pressing usability issue.
3. Successful Use Cases and Strategies: Use cases where AI programming assistants excel include generating boilerplate code, aiding in quality assurance tasks like test generation, and facilitating the learning of new programming concepts. Key strategies adopted by developers include providing explicit input and context, employing clear explanations, and sometimes relying on existing code to improve the model's output.
4. Implications for Learning and Understanding: The findings suggest that developers are increasingly viewing these tools not just as productivity aids but as learning partners. By assisting in the recall of APIs and programming constructs, these assistants are pushing boundaries in educational contexts.

Opportunities for Enhancement and Future Research Directions

The findings indicate an avenue for improving AI programming assistants by refining the models to better understand developer contexts and offering more nuanced control mechanisms. Developers express a desire for interactions that cater to personal coding styles and project-specific requirements, indicating a need for models to support personalization and adaptive learning from user feedback. Moreover, adopting more conversational interfaces inspired by chatbot models like ChatGPT emerges as a potential avenue to facilitate natural language interactions, improving usability in exploration modes.

Encouraging explicit feedback to fine-tune the programming assistant models could align tool suggestions closer to user expectations. Additionally, investigating the incorporation of non-functional requirements such as performance, readability, and security within the generated code could address a key aspect of developers' reluctances.

Conclusion

This paper delivers critical insights into the usability of AI programming assistants in modern software development environments. It suggests focusing on aligning these tools more closely with the needs of developers through improved control, contextual understanding, and code quality. The path forward includes leveraging developers' feedback effectively and fostering tools that are intelligent, personalized, and contextually aware. These advances could herald significant gains in developers' productivity and the wider adoption of such assistants in intricate coding environments.