The Robots are Here: Navigating the Generative AI Revolution in Computing Education (2310.00658v1)

Abstract: Recent advancements in AI are fundamentally reshaping computing, with LLMs now effectively being able to generate and interpret source code and natural language instructions. These emergent capabilities have sparked urgent questions in the computing education community around how educators should adapt their pedagogy to address the challenges and to leverage the opportunities presented by this new technology. In this working group report, we undertake a comprehensive exploration of LLMs in the context of computing education and make five significant contributions. First, we provide a detailed review of the literature on LLMs in computing education and synthesise findings from 71 primary articles. Second, we report the findings of a survey of computing students and instructors from across 20 countries, capturing prevailing attitudes towards LLMs and their use in computing education contexts. Third, to understand how pedagogy is already changing, we offer insights collected from in-depth interviews with 22 computing educators from five continents who have already adapted their curricula and assessments. Fourth, we use the ACM Code of Ethics to frame a discussion of ethical issues raised by the use of LLMs in computing education, and we provide concrete advice for policy makers, educators, and students. Finally, we benchmark the performance of LLMs on various computing education datasets, and highlight the extent to which the capabilities of current models are rapidly improving. Our aim is that this report will serve as a focal point for both researchers and practitioners who are exploring, adapting, using, and evaluating LLMs and LLM-based tools in computing classrooms.

Navigating the Generative AI Revolution in Computing Education

The unprecedented capabilities of generative AI, particularly LLMs like GPT and Codex, have poised them as potentially transformative agents in the field of computing education. The paper, "The Robots are Here: Navigating the Generative AI Revolution in Computing Education," presents a comprehensive analysis of the intersections between LLMs and computing education, exploring their potential impacts, challenges, and opportunities.

Overview of Contributions

This multi-authored working group report delineates five key contributions in understanding and integrating LLMs within computing education.

1. Literature Review: The authors conducted an exhaustive survey of 71 articles, providing insights into the rapidly evolving role of LLMs in computing education. The review highlights that LLMs have demonstrated competence in resolving programming exercises comparable to that of students, enabling the generation of learning materials like problem sets and explanations. However, the potential for student over-reliance on LLMs, posing a risk to foundational skills development, was a notable concern across the literature.
2. Perceptions and Attitudes: By surveying 171 students and 57 educators from diverse geographical locations, the paper highlights a general consensus regarding the necessity for some form of regulation in employing LLMs. Both students and educators agree on the potential and limitations of LLMs, yet express the need for clarity in policy, especially as many educators find the current guidelines surrounding LLM use to be ambiguous.
3. Curricular Adjustments: Interviews with 22 educators indicate shifts toward focusing on critical thinking, code comprehension, and developing new learning objectives that include using LLMs effectively. There is also an emphasis on process-oriented assessments over content, with increased weight given to exams and proctored assessments to mitigate academic integrity concerns.
4. Ethical Implications: The complexities surrounding the ethical use of LLMs are addressed with an analysis grounded in the ACM Code of Ethics. The discussion extends to institutional responses and guidelines, recommending that educators foster awareness among students regarding ethical AI use. Explicit statements about permissible uses and transparency in academic work involving LLMs are suggested.
5. Benchmarking and Performance Analysis: The evolving capability of LLMs is captured through benchmark tests using earlier publications as a reference. The paper replicates prior work to assess the performance of contemporary models like GPT-4, noting significant improvements, underscoring the necessity for continuously updated benchmarks to evaluate educational tools accurately.

Implications and Future Directions

The intersection of LLMs and education is a burgeoning field with profound implications. As LLMs dramatically enhance their performance, instructional practices must adapt to harness these tools effectively without eroding foundational skills in computing. There lies a balance to be struck between leveraging these models for pedagogical benefit and managing their risks, such as the propagation of biases and dependency.

The authors argue for a reimagining of computer science curricula, recommending a progressive shift in focus towards competencies that complement the strengths of LLMs. As reliance on LLMs for generating straightforward code solutions increases, emphasis might shift towards higher-order skills such as critical analysis, ethics, and creative problem-solving.

Overall, the report serves as a crucial resource for educators and policy-makers, equipping them with the knowledge to navigate the integration of LLMs into computing education thoughtfully and effectively. As technological advancements continue, ongoing dialogue and research will be indispensable in shaping the future of learning in computing disciplines.