What is it like to program with artificial intelligence? (2208.06213v2)

Abstract: LLMs, such as OpenAI's codex and Deepmind's AlphaCode, can generate code to solve a variety of problems expressed in natural language. This technology has already been commercialised in at least one widely-used programming editor extension: GitHub Copilot. In this paper, we explore how programming with LLMs (LLM-assisted programming) is similar to, and differs from, prior conceptualisations of programmer assistance. We draw upon publicly available experience reports of LLM-assisted programming, as well as prior usability and design studies. We find that while LLM-assisted programming shares some properties of compilation, pair programming, and programming via search and reuse, there are fundamental differences both in the technical possibilities as well as the practical experience. Thus, LLM-assisted programming ought to be viewed as a new way of programming with its own distinct properties and challenges. Finally, we draw upon observations from a user study in which non-expert end user programmers use LLM-assisted tools for solving data tasks in spreadsheets. We discuss the issues that might arise, and open research challenges, in applying LLMs to end-user programming, particularly with users who have little or no programming expertise.

An Exploration of the Influence and Challenges of AI-Assisted Programming

The paper "What is it like to program with artificial intelligence?" presents a detailed exploration of programming with LLMs through experiences with existing tools like GitHub Copilot. The authors examine how these AI-assisted programming tools transform the programming process and outline the distinct characteristics and challenges these tools introduce compared to traditional methods like compilation, pair programming, and search-based development.

The paper starts by highlighting the technological advancements that underpin the capabilities of LLMs in programming, such as transformer architecture and large pre-trained models. These models, epitomized by technologies such as OpenAI's Codex and DeepMind's AlphaCode, have demonstrated considerable aptitude in generating functional code from natural language descriptions. This capability has been commercialized in tools like GitHub Copilot, redefining programmer assistance beyond classical autocomplete functions.

Comparison with Previous Programmer Assistance Models

The authors articulate that while AI-driven solutions share procedural aspects with traditional contexts such as pair-programming and search-driven development, distinct differences are evident. For instance, compared to static code autocompletion, tools like Copilot are capable of synthesizing entire function bodies or generating test cases, hence resembling more of a sophisticated, dynamic partner in the coding process rather than a mere platform for error detection or template completion.

Despite the apparent speed gain in generating code, interviews with developers and comprehensive usability studies reveal several enduring challenges. The success of code generation heavily depends on the development of precise prompts, requiring programmers to refine their queries iteratively. The effort to ensure generated code's correctness—particularly when the output might escape traditional comprehension boundaries—often counterbalances the initial gains.

Experiences and Challenges with AI-Generated Code

The paper cites longitudinal experiences of programmers, underscoring instances where these tools expedite the programming process, notably in repetitive or boilerplate code scenarios. However, these trials are not without risk; developers note the potential for AI systems to produce syntactically correct but semantically flawed code that may deviate subtly from intended functionality. The task of verifying AI-generated code for correctness and security remains a manual and somewhat arduous process.

Moreover, this new programming paradigm introduces novel debugging challenges; the unexpected logic or erroneous outcomes driven by learned data biases necessitate a re-conceptualization of existing debugging strategies that might have been sufficient for human-generated code.

Implications for End-User Programmers

The authors extend the discussion to encompass implications for end-user programmers, who might not have extensive programming expertise and might therefore benefit substantially from such tools. The AI's ability to automatically transform complex tasks into executable solutions by understanding natural language promises significant utility. Nonetheless, this application reignites challenges around comprehending generated code, verifying its correctness, and developing adequate problem decomposition skills typical in structured software development.

Future Prospects

The paper discusses the profound implications on programming structures and the potential shift in programming practices. As AI tools reach higher maturity levels, exploring strategies for integrating intelligent assistance beyond simple code generation—such as more interactive feedback (akin to compiler diagnostics) or context-driven programming suggestions—may become crucial. The potential impacts on task completion, economic productivity, and job roles in software engineering underscore a need for more in-depth studies as these technologies continue evolving.

In conclusion, this paper provides valuable insights into a nascent yet transformative evolution in software development facilitated by AI. It highlights both the potential and the limitations of current LLM-based programming assistants and paves a path for future explorations in making AI a reliable partner in the art and science of software engineering.