"It's Weird That it Knows What I Want": Usability and Interactions with Copilot for Novice Programmers (2304.02491v1)

Abstract: Recent developments in deep learning have resulted in code-generation models that produce source code from natural language and code-based prompts with high accuracy. This is likely to have profound effects in the classroom, where novices learning to code can now use free tools to automatically suggest solutions to programming exercises and assignments. However, little is currently known about how novices interact with these tools in practice. We present the first study that observes students at the introductory level using one such code auto-generating tool, Github Copilot, on a typical introductory programming (CS1) assignment. Through observations and interviews we explore student perceptions of the benefits and pitfalls of this technology for learning, present new observed interaction patterns, and discuss cognitive and metacognitive difficulties faced by students. We consider design implications of these findings, specifically in terms of how tools like Copilot can better support and scaffold the novice programming experience.

Usability and Interactions with Copilot for Novice Programmers

This research paper investigates how novice programmers interact with GitHub Copilot, a code generation tool powered by LLMs, and explores their perceptions of its utility during programming tasks. The paper is grounded in the context of introductory programming education and aims to assess the usability, cognitive implications, and potential benefits and risks associated with Copilot.

The researchers employed a qualitative methodology, combining observation and interviews with novice programmers performing a typical introductory programming assignment. The goal was to capture interaction patterns, perceptions of usability, and personality-driven concerns. The primary findings reveal interaction patterns termed as "shepherding" and "drifting," commentary on cognitive load, metacognition, and broader concerns about reliance and ethical implications.

Key Findings and Observations

The participants—novice programmers in an introductory programming course—were initially unfamiliar with tools like Copilot. The observations recorded two prominent interaction patterns. Shepherding refers to the behavior where students attempt to guide Copilot into producing code which aligns with their objectives. This often involved students meticulously typing out suggestions rather than immediately accepting them, highlighting a possible skepticism and need for perceived control over the code composition process. Drifting involves students navigating between multiple suggestions in an exploratory but aimless manner, leading to confusion and cognitive overload.

The paper identified enhancement in coding efficiency as an explicit perceived benefit. Students reported Copilot's suggestions saved time, reduced syntax errors, and facilitated faster iteration through programming tasks. This aligns with findings in previous research that noted increased productivity when using auto-generated code, even though it can lead to cognitive distractions and a superficial learning process if unchecked.

Cognitive and Metacognitive Aspects

The cognitive responses of participants indicated a potential double-edged effect of Copilot. While suggestions accelerated progress, they simultaneously posed a challenge by increasing the cognitive burden—students found deciphering suggested code mentally taxing. Participants' reflections revealed both positive and negative sentiments, with positive emotions emerging from Copilot's ability to anticipate code solutions, prompting excitement and engagement. However, negative emotions arose from frustrations linked with incorrect suggestions or unwelcome intrusions into the thought process.

From a metacognitive standpoint, Copilot served as both a supportive scaffolding tool and a potential crutch. Some students used it as a stepping surface for deeper problem-solving, while others feared developing dependency, which could hinder their learning. This bifurcation points to a need for explicit metacognitive guidance within programming support tools to bolster learning without fostering over-dependence.

Speculations on Broader Implications

Ethical considerations, trust, and the perceived intelligence of Copilot surfaced prominently in participant feedback. Concerns about over-reliance on auto-generated code and reduced problem-solving skills suggest implications that educators must address by setting clear guidelines on how to balance genuine understanding with the utility of such tools.

Moreover, the paper discusses ethical concerns surrounding code generation, particularly related to licensing and potential misuse. Students expressed fears of plagiarism and unintentional reliance on snippets without understanding their contexts or origins. This raises a broader question of how educators might integrate Copilot into curricula that incorporate conceptual understanding along with practical tool use.

Design Implications

The paper offers several design implications aimed at improving the interaction experience for novices. They recommend:

1. Prompt Control: Offering users control over when suggestions are made could reduce cognitive overload and provide a clearer workflow.
2. Explainable AI: Ensuring Copilot provides transparent logic and confidence levels behind suggestions may bolster trust and educate users by offering insights into the AI's decision-making process.
3. Metacognitive Scaffolding: Designing user interfaces that incorporate metacognitive cues can guide novice programmers through learning stages, enhancing their problem-solving skills and cognitive frameworks.

Conclusion

This paper contributes to the understanding of AI tools’ impact on programming education, specifically for novices. It underscores the potential benefits of tools like Copilot in enhancing coding efficiency, while also highlighting cognitive, emotional, and ethical challenges. Future developments should strive towards creating balanced, ethically-informed, and educatively beneficial AI-integrated learning environments.