Design and Deployment of a Course-Aware AI Tutor in an Introductory Programming Course

Abstract: LLMs have become part of how students solve programming tasks, offering immediate explanations and even full solutions. Previous work has highlighted that novice programmers often heavily rely on LLMs, thereby neglecting their own problem-solving skills. To address this challenge, we designed a course-specific online Python tutor that provides retrieval-augmented, course-aligned guidance without generating complete solutions. The tutor integrates a web-based programming environment with a conversational agent that offers hints, Socratic questions, and explanations grounded in course materials. Students used the system during self-study to work on homework assignments, and the tutor also supported questions about the broader course material. We collected structured student feedback and analyzed interaction logs to investigate how they engaged with the tutor's guidance. We observed that students used the tutor primarily for conceptual understanding, implementation guidance, and debugging, and perceived it as a course-aligned, context-aware learning support that encourages engagement rather than direct solution copying.

• The paper presents a domain-specific AI tutor for introductory Python that integrates course materials using a retrieval-augmented LLM approach.
• It employs a three-tier architecture with configurable context awareness and Socratic prompting to provide guided, incremental hints.
• Empirical evaluations show positive student feedback with improved usability, pedagogical alignment, and effective scaffolding.

Course-Aware AI Tutoring for Introductory Programming: Design, Deployment, and Evaluation

Introduction

The widespread adoption of LLMs has catalyzed substantial shifts in programming education, fundamentally altering the ways students interact with instructional content and seek assistance. While generic LLM-based assistants such as ChatGPT and GitHub Copilot can provide instant explanations, coding examples, and even end-to-end solutions, over-reliance on such tools may undermine the development of conceptual understanding and independent problem-solving proficiency. Addressing this pedagogical risk, the paper "Design and Deployment of a Course-Aware AI Tutor in an Introductory Programming Course" (2604.11836) details the design, integration, and classroom evaluation of a domain-restricted, context-aware Python programming tutor aligned tightly with course materials and learning objectives.

System Architecture and Pedagogical Constraints

The Python Online Tutor comprises a three-tier architecture, balancing flexibility, scalability, and rigorous curricular alignment. The frontend, developed in React, merges code editing, prompt-based dialogue, and course-task context configuration in a cohesive web-based interface. The Prompt Manager backend orchestrates asynchronous interaction, session management, and dynamic prompt engineering, enabling granular control over the context-awareness level per student session. Response generation utilizes the OpenAI Assistant API with a Retrieval-Augmented Generation (RAG) scheme referencing a comprehensive, vectorized repository of all course documents—including slides, annotated code samples, assignment texts, and explanatory materials.

Figure 1: Architecture of the Python Online Tutor, illustrating the integration of the frontend, prompt manager backend, and retrieval-augmented LLM via API.

This design enforces several critical pedagogical guardrails:

• Course-Conform Responses: The system grounds all explanations in officially endorsed course content, precluding off-topic or advanced material.
• Socratic Prompting: The tutor eschews generation of full solutions, instead delivering incremental hints, guiding questions, and didactic feedback, modulated by the contextual history of the session.
• Configurable Context Awareness: Students and instructors can calibrate whether current code state and task descriptions are factored into LLM prompts, supporting focused guidance.
• Asynchronous Operation and Monitoring: The backend supports concurrent users with full interaction logging (while respecting privacy), enabling post hoc analysis of help-seeking behaviors and system performance.

The vector-based course material retrieval, executed at prompt time, ensures that all LLM completions are normatively consistent with current learning objectives and assignment scope.

User Interaction Design and Workflow

The tutor positions itself as an auxiliary layer directly within students' self-study workflow, accessible during homework completion and general course content exploration. The frontend exposes a tightly integrated chat interface alongside code editing and assignment viewing, with configurable context sensitivity and clear session continuity via thread identifiers.

Figure 2: Tutor Frontend, indicating conversational prompt/answer box, context-awareness controls, task description view, and code editing region.

Interaction logging and session data collection underpin both live monitoring and retrospective analysis, making visible both the taxonomy of student queries and the efficacy of the tutor’s corrective and formative interventions.

Empirical Study: Methodology and Data Analysis

A mixed-methods study was conducted in a first-year business administration undergraduate course (39 participants, predominantly novice programmers). Students engaged with the AI tutor during two core assignment topics (collections and functions). The study combined:

• Automated log analysis: Extraction and open coding of 304 non-trivial student-tutor interactions.
• Structured survey responses: 33 Likert-scale and open-ended items capturing didactic, contextual, and usability perceptions.

The primary research aims were to categorize student query types (help-seeking patterns) and to evaluate perceived instructional value, with particular focus on the effectiveness of context/awareness features and fidelity to course-aligned scaffolding.

Results: Usage Patterns and Student Perceptions

Inquiry Categories and Interaction Patterns

Analysis delineated twelve question categories. The most prominent clusters were conceptual explanations (20.1%), implementation guidance ("How to," code correctness, and debugging collectively 34.2%), and explicit implementation requests (15.5%). Code comprehension and assignment clarification requests were also observed, indicating substantial demand for both foundational skill-building and task-specific scaffolding.

Despite enforced solution-avoidance mechanisms, students frequently submitted prompts with the intent to elicit explicit implementations; the system, however, maintained restriction to hints and conceptual prompts. This affirms both the necessity for such didactic guardrails and students’ tendency to seek direct assistance when struggling.

Student Evaluation and Comparative Metrics

Survey responses indicated uniformly positive perceptions across all measured dimensions. The mean rating for overall system support was 3.84/5, with the highest scores recorded for usability (4.06) and context-awareness (4.01). Didactic behavior, including avoidance of full solutions (M = 4.18) and alignment with course material (M = 4.21), was particularly valued.

Figure 3: Mean survey scores across question categories: didactics, awareness, usability, summary, and ChatGPT comparison.

Qualitative feedback highlighted strong appreciation for code-aware, contextually grounded explanations and the effectiveness of the tutor’s Socratic strategy. Trust in answer quality correlated with the system’s exclusive reliance on course materials. However, system responsiveness (latency in LLM response), while not a primary focus of the implementation, was flagged as a barrier to uninterrupted learning flow.

Comparative items revealed a modest, but consistent, preference for the course-specific tutor over general-purpose LLMs such as ChatGPT in terms of learning support and exam preparation, especially for students with limited prior programming experience.

Practical Implications and Theoretical Significance

This work demonstrates that LLM-powered, course-aligned AI tutors can function as effective, scalable scaffolding tools in programming education, provided they incorporate strict guardrails against direct solution delivery and leverage granular context-awareness. The system’s RAG-based integration ensures both consistency and fidelity to curricular goals, while its adaptive, didactically constrained dialogue avoids fostering counterproductive help-seeking or surface learning strategies.

The findings underscore several actionable design implications:

• Guardrail Design: Continuous refinement of prompt engineering and feedback templates is necessary to maintain guidance without solution leakage, particularly for students prone to exploitative queries.
• Context Integration: Rich, configurable context awareness (e.g., dynamic access to current code and task specification) is crucial for perceived utility, trust, and personalization.
• System Responsiveness: Low-latency answer generation is essential in production deployments, suggesting further research into local LLM inference or more efficient API designs.
• Instrumentation for Monitoring: Persistent, privacy-aware logging is indispensable for both iterative educational design and future learning analytics.

Given the highly positive student reception, there is strong evidence that course-aware AI tutors can drive deeper engagement and more independent problem-solving practices compared to generic LLM use—while maintaining alignment with pedagogical intent.

Future Directions in AI-Augmented Education

Several avenues for future research and development arise from this deployment:

• Longitudinal, summative assessment studies assessing tangible impact on learning outcomes and retention.
• Expansion of coverage to advanced programming concepts and cross-course integration.
• Fine-grained adaptivity, including real-time recognition of repeated solution-seeking patterns and corresponding escalation of scaffolded interventions.
• Analysis of tutor usage across different demographics and experience levels to enhance inclusivity and instructional adequacy.

Evolving LLM architectures and local deployment models may also mitigate current latency challenges and reduce resource dependencies, further increasing the feasibility of institutional adoption.

Conclusion

The implementation and in situ evaluation of a course-aware, RAG-augmented AI tutor for introductory Python education confirms that carefully regulated, contextually aware conversational agents can serve as robust, scalable educational support systems. Such tutors are favorably perceived by students, foster independent engagement, and effectively substitute for generic LLMs when properly constrained. Critical to success are rigorous alignment with course materials, adaptive context use, and responsive interaction. These results provide a foundation for expanded investigation of AI-powered formative assessment and scaffolding across diverse STEM learning environments.