Understanding Student Perceptions, Mistakes, and Debugging Approaches when Solving Natural Language Programming Tasks

Abstract: Learning to communicate with code-generating AI models is an emerging skill for novice programmers. One recent pedagogical approach, Prompt Problems, has students solve computational tasks by writing natural-language prompts for code-generating AI models. However, little is known about the specific prompt-level mistakes novice programmers make, the kinds of computational details they fail to communicate, and what strategies they use to recover when generated code is incorrect. In a CS1 course, we studied attempts by more than 900 students to solve dialogue-based Prompt Problems. We analyzed student reflections, unsuccessful prompts, and reported debugging strategies. Compared to traditional coding tasks, students generally found prompting easier, more enjoyable, and better targeted at developing problem-solving skills. The most common mistakes are related to the omission of key details, suggesting both a failure to acknowledge their importance and over-reliance on AI to infer them. When prompts failed, students focused more on clarifying their intent and reflecting on the provided problem details than on tracing generated code or examining test cases.

• The paper shows that prompt-based programming reduces cognitive load by offloading syntactic reasoning, as evidenced by a Likert mean of 3.80/5.
• The paper details common error patterns such as omission of type details, misnamed arguments, and incomplete functionality in novice prompts.
• The paper advocates for instructional scaffolds and hybrid workflows to balance prompt refinement and traditional code comprehension for robust learning.

Understanding Student Prompting Practices in Natural Language Programming Tasks with Generative AI

Introduction and Context

Prompt-based programming with generative AI models has rapidly shifted the paradigm of how novices engage with computational problem solving. The investigated study systematically analyzes the experiences, error profiles, and debugging strategies of over 900 introductory programming students as they interact with dialogue-driven Prompt Problems using a GPT-4o mini backend. Prompt Problems require students to solve computational tasks by crafting natural language prompts that direct the code-generating model to produce the intended implementation. This work fills critical gaps in understanding novice behavior at prompt granularity, characterizing the novel mistakes endemic to natural language interface with code-generation systems, and cataloging how novices respond to model errors.

Task and Tool Design

The study deployed Prompt Programming, a web-based environment enabling iterative, multi-turn dialogue between learners and LLMs, without exposing the raw problem specification to the model. The interface supports prompt sending, interaction reset, and execution of generated code against predefined test cases, shifting debugging activities to the meta-level of prompt composition and refinement.

Six custom-designed Prompt Problems were partitioned into two difficulty tiers: arrays-centric batch (B1) and higher-complexity data-structured batch (B2). The former emphasizes counting and searching among integer sequences (e.g., count negatives, sum evens), while the latter tests array/matrix manipulation and in-place updates (e.g., binary addition, propagate ones).

Figure 1: Problem description and input-output specs.

Figure 2: Problem description and input-output specs.

Figure 3: Specifications for `count negatives' problem (B1-1).

Student Perceptions versus Traditional Coding

A rigorous thematic analysis of 400 targeted, saturated student reflections highlighted a strong trend: students generally reported Prompt Problems to be easier, faster, and less cognitively taxing than traditional coding, in alignment with Cognitive Load Theory predictions. This was both self-reported and corroborated by a Likert mean of 3.80/5 for perceived increased ease with prompting versus code authoring. The prevalent belief was that natural language offloads syntactic and low-level reasoning, thereby foregrounding algorithmic analysis and code comprehension Figure 4.

Notably, a subpopulation remains skeptical, expressing reservations about AI reliability for complex problem domains, a desire for hybrid workflows (prompting + manual code editing), and doubts regarding the depth of learning afforded by prompt-centric tasks. Several respondents acknowledged that as problem complexity escalates, confidence in AI-generated artifacts declines, and traditional coding competencies become more valued.

Figure 4: Themes identified in students' responses to Ref1 and Ref2. Bars show the number of coded occurrences for each theme across 200 student reflections, with stacked segments indicating whether the theme appeared in responses to Ref1 or Ref2.

Error Taxonomy in Natural Language Prompts

A granular coding of 1,286 initial, unsuccessful prompts exposed the spectrum of novice error patterns unique to prompt-based programming. The most prevalent mistakes were:

• Omission of function return type and expected output: Students commonly neglected to specify critical type-level details or failed to articulate the desired return semantics, especially in batch B1.
• Absent or misnamed arguments: Students frequently omitted or mislabeled arguments, deviating from problem specifications even when explicit names were presented visually.
• Incomplete high-level functionality descriptions: Failure to communicate comprehensive behavioral intent led to underspecified or misdirected code outputs.
• Argument order and function signature mismatches: Confusion about ordering and an overreliance on AI inference were notable, particularly in more complex batch B2 tasks.
• Extraneous elements: Prompts sometimes incorporated irrelevant information, reflecting confusion or attempts to compensate for earlier failures.

Correlational analysis revealed these omissions often co-occurred, indicating that when students omitted one structural detail, multiple aspects of the specification were typically absent. Importantly, there was little evidence that making more initial mistakes required substantially more conversational exchanges to reach a solution, suggesting beginner learners may correct multiple omission categories simultaneously upon receiving feedback.

Debugging and Refinement Strategies

Analysis of students' self-reported prompt refinement strategies in B2 highlighted strong reliance on clarifying intent and more thoroughly specifying missing details in follow-up prompts. The majority of students anchored their debugging in re-examining the graphical problem depiction rather than systematically tracing generated code or scrutinizing failing test cases—both higher-order, but less frequently employed, repair strategies.

Figure 5: Strategies reported to refine prompts following unsuccessful attempts on the second batch of problems (B2) (n = 174).

A minority of students engaged in suboptimal or non-learning-aligned strategies such as repeatedly pasting failing test cases into the prompt, supplying code directly as prompt content, or "gambling" by resubmitting identical prompts. Frustration with AI misunderstanding was reported, particularly as problem complexity increased or paralinguistic prompt nuances were not disambiguated by the model.

Theoretical Implications and Recommendations

Prompt-based programming with LLMs effectively reduces extraneous cognitive load, helping novices achieve higher-level problem decomposition and analytical thinking. However, this offloading risks blunt engagement with the underlying code, and can produce "vibe coding" phenomena—where users iterate prompts until acceptable outputs are achieved without ever interrogating code correctness, a scenario detrimental to robust skill acquisition or transfer (cf. [gerlich2025ai]).

Results underscore the urgency of pedagogically scaffolded prompting. Promising interventions include structured prompt templates, Parsons problem-inspired distractor frameworks to sensitize learners to detail inclusion/exclusion, and prompt-level AI critique middleware for immediate, low-latency feedback on prompt completeness and precision.

The empirical finding that many students default to prompt clarification rather than code or test case analysis suggests a gap in code reading and debugging integration that must be addressed, possibly via deliberate alignment of prompt refinement cycles with code examination checkpoints or injected error discovery activities.

Practical and Future Directions

Practically, the study advocates for instructional designs that balance cognitive efficiency gains from prompting with mechanisms requiring engagement in code comprehension and test-driven iteration—essential for transfer to authentic professional workflows and higher-complexity problems. There is a need to track how prompting skills evolve longitudinally and to determine the transferability of prompt engineering competencies across problem domains and programming languages.

Future research should also disentangle the effects of single-shot versus dialogue-based prompting, quantify learning gains versus traditional code writing, and develop both formative and summative assessment models for prompt quality and prompt-driven debugging behaviors.

Conclusion

Prompt-based programming tasks are perceived by novices as easier and more enjoyable than traditional code writing, with clear advantages in engaging problem-solving and analysis skills by leveraging cognitive offloading. However, the transition to natural language interfaces introduces novel error patterns and suboptimal debugging strategies centered on omission of detail and excessive reliance on AI inferencing. Comprehensive curricular and tool scaffolds are warranted to ensure that learners emerge not just as competent prompt engineers, but as informed, analytic programmers equipped for the realities of modern AI-assisted software development.

The analysis above synthesizes empirical findings and theoretical implications from "Understanding Student Perceptions, Mistakes, and Debugging Approaches when Solving Natural Language Programming Tasks" (2607.05034), as well as relevant literature on GenAI integration in computing education and prompt engineering.