Open questions on the usability of AI programming assistants

Investigate the usability of AI programming assistants (such as GitHub Copilot and similar tools) in real-world developer workflows to determine why developers frequently do not accept the tools’ initial code suggestions and to characterize the factors that influence suggestion acceptance and overall usability.

Background

In the Related Work section, the authors note that developers often do not accept AI programming assistants’ initial suggestions. They explicitly state that this observation leaves open questions regarding the usability of these tools.

Understanding these usability issues is important for interpreting empirical findings on productivity and for guiding the design of AI programming assistants used in industry settings.

References

However, in practice, developers do not accept AI programming assistants' initial suggestions at a high frequency. This leaves a number of open questions related to the usability of these tools.

— How much does AI impact development speed? An enterprise-based randomized controlled trial  (2410.12944 - Paradis et al., 2024) in Section 2: Related Work

Our use cases and colleagues' experiences are concentrated in academic writing by experienced researchers. They illustrate how Reactant can fit ongoing work, but we have not conducted a formal comparative usability study. We therefore cannot establish whether Reactant is easier to learn, more efficient, or preferable to conventional chat-based or editor-integrated workflows.

— Who Asked for This? Inline Annotations as Authoring Transactions for Provenance in Agentic Authoring  (2609.40126 - Xiao, 30 Sep 2026) in Discussion and Conclusion, Section 6