ChatGPT-Based Tutor: Efficacy & Limitations

• ChatGPT-based tutor is an AI-powered instructional agent that emulates human tutoring through interactive feedback and explanations.
• Empirical studies show that while ChatGPT-generated hints lead to measurable learning gains, they are statistically inferior to human-authored hints.
• Systematic quality checks and advanced prompt engineering are essential to reduce errors and enhance the tutor's domain-specific performance.

A ChatGPT-based tutor is an artificial intelligence-driven instructional agent built on LLM technology—specifically OpenAI’s ChatGPT platform—that delivers automated, interactive educational support across a range of disciplines and scenarios. Such tutors emulate or extend traditional human tutoring practices by providing explanations, hints, feedback, code review, assessment items, and dialogic engagement via text (and sometimes voice or multimodal) interfaces. Recent empirical studies across mathematics, programming, language acquisition, and higher education settings document both the efficiency gains and the persistent limitations of deploying ChatGPT-based tutors for scalable, adaptive, and context-aware learning.

1. Experimental Evaluation of Learning Gains

The first controlled evaluation of ChatGPT-generated algebra hints in comparison with human tutor-authored hints was conducted using a randomized experiment with 77 participants distributed across Elementary Algebra and Intermediate Algebra modules (Pardos et al., 2023). Participants experienced a three-phase lesson structure (pre-test, acquisition with embedded hints, post-test), allowing for the quantification of learning gains:

$$\text{Learning Gain (LG)} = (\text{Post-test Score}) - (\text{Pre-test Score})$$

Results indicate:

• Both human and ChatGPT-generated hints produced positive learning gains, but statistical significance was achieved only for human-authored hints.
• For Elementary Algebra, manual hints yielded 24.63% average gain versus 11.14% for ChatGPT; for Intermediate Algebra the difference was 23.65% vs. 1.7%.
• Mann–Whitney U tests confirmed statistical significance of these discrepancies ($p = 0.038$).
• Approximately 70% of ChatGPT-generated hints passed manual correctness checks (including final answer accuracy and validity of intermediate steps), with a 30% rejection rate due to errors.

This experimental evidence reveals that, although ChatGPT-generated instruction can drive learning gains above baseline, it remains inferior to human curation both in effect size and statistical robustness. These findings point to both the promise and the necessity of human oversight in high-stakes educational contexts.

2. Hint Generation Quality and Systematic Error Analysis

The process of generating high-quality hints with ChatGPT entails nontrivial validation. The manual quality check protocol used in experiments verifies:

1. Correctness of the final solution,
2. Validity and completeness of all intermediate steps,
3. Absence of any inappropriate or off-topic language.

The observed 30% rejection rate for ChatGPT hints in (Pardos et al., 2023) was principally due to incorrect answers or erroneous solution steps. In contrast, the human-authored hint process, involving dedicated content editing, led to higher fidelity and much stronger learning effects. This highlights a bottleneck in direct LLM content authoring for education, as contemporary models remain susceptible to semantic errors undetectable to novice learners.

A plausible implication is that LLM output in educational settings should be systematically screened (either by expert reviewers or AI-assisted quality control pipelines) to mitigate the risk of propagating instructional misconceptions.

3. Student Interaction Patterns and Self-Directed Learning

Studies of student-ChatGPT interaction logs in self-directed learning settings identify five dominant use categories (Ammari et al., 30 May 2025):

• Information Seeking: Factual retrieval, clarification of concepts, application of theory to problem solving.
• Content Generation: Drafting essays, code, MCQs, application materials; iteratively debugging outputs.
• Language Refinement: Rephrasing, translation, stylistic improvement, grammar checking.
• Meta-Cognitive Engagement: Self-assessment, goal setting, reflection, explicit uncertainty expression (“Can you double check?”).
• Conversational Repair: Prompt adaptation, clarification requests, correction of AI errors, emotional regulation.

Behavioral modeling demonstrates that goal-driven tasks such as coding or job application writing strongly predict continued tool engagement ($R^2 = 0.79$ in lagged linear regression, HR = 0.01 in a Cox model). Conversely, frequent repair actions and cognitive/emotional burden from breakdowns reduce return likelihood, unless mitigated by prompt system apologies or effective clarifications.

These findings define the ChatGPT-based tutor as an active partner in a co-regulated learning process, with students leveraging both the tool’s generative affordances and repair capabilities in an iterative progression toward mastery.

4. Prompt Engineering, Feedback, and Assessment

Prompt engineering emerges as a critical determinant of ChatGPT-based tutor performance across feedback, question generation, and assessment tasks. Exemplary structured prompts—incorporating Role, Task, Context, Example, and Format (“RTCEF” structure)—significantly improve the syntactic and semantic quality of generated outputs (Vu et al., 26 Jul 2025). For example:

\begin{array}{l} \textbf{Role:} \ \text{You are a lecturer of [subject]} \ \textbf{Task:} \ \text{Create N questions} \ \textbf{Context:} \ \text{Focusing on [topic]} \ \textbf{Example:} \ \text{[Sample question]} \ \textbf{Format:} \ \text{MCQ/T/F/Scenario} \ \end{array}

Blind evaluation surveys in higher education show that ChatGPT-generated questions are frequently indistinguishable from human-authored ones by students (with >60% misattribution), though expert reviewers remain more discerning.

Advanced prompt designs for automated code feedback—using in-context learning (ICL) and chain-of-thought (CoT) architectures—enable programmatically analyzable, sectioned outputs (“Brief Code Explanation,” “Main Issues,” and “Corrected Version”), supporting semi-automatic error rate estimation and instructor review (Ballestero-Ribó et al., 24 Jan 2025).

Challenges include duplication rates in MCQ generation (up to 19%), missing assumptions in calculative exercises (c. 28%), and persistent hallucination of feedback or error diagnoses by the LLM, even in high-performing versions such as GPT-4T.

5. Domain-Specific Capabilities and Limitations

Empirical studies span a wide range of coursework, demonstrating both the versatility and the persistent boundaries of ChatGPT-based tutors:

• Programming and Code Feedback: ChatGPT matches or outperforms prior models in code generation, code review, and summarization (Tian et al., 2023, Chen et al., 2023, Popovici, 20 Jan 2024, Bassner et al., 9 May 2024). However, the feedback is often cosmetic or superficial, with critical logical or syntactic errors escaping detection. Structured prompt templates and careful calibration improve diagnostic coverage but do not eliminate the requirement for instructor oversight (Ballestero-Ribó et al., 24 Jan 2025, Popovici, 20 Jan 2024, Anishka et al., 2023).
• Mathematics and Quantitative Reasoning: In algebra and linear algebra, ChatGPT can produce correct worked solutions, often in LaTeX, but is prone to algorithmic mistakes (misapplied elimination schemes, misclassification of solution multiplicity, failure in edge-case handling) (Pardos et al., 2023, Bagno et al., 18 Feb 2024). In the physical sciences, its reasoning about graphical representations is strong but visual decoding of graphs is weak (Polverini et al., 2023), suggesting the system should not be deployed as a sole accessibility tool for vision-impaired learners.
• Language and Communication: ChatGPT-based voice tutors provide real-time pronunciation feedback and support for communicative competence in EFL learners (Zhou, 2023). Nevertheless, limitations in feedback nuance, variable recognition accuracy, and the risk of over-reliance on AI-to-AI conversations constrain the achievable outcomes without human mediation.

6. Systems Integration, Context Awareness, and Knowledge Tracing

Modern ChatGPT-based tutoring systems often combine LLM engines with information retrieval (IR) modules, course-specific databases, automated test harnesses, or classroom learning management systems to achieve rapid, context-aware instructional support (Wang et al., 2023, Bassner et al., 9 May 2024, Popovici, 20 Jan 2024). Architectural advances such as ChatEd integrate course materials via vector search and aggregate conversational context to produce referenced, verifiable, and contextually relevant answers (Wang et al., 2023).

In large-scale deployments (e.g., Iris on Artemis), chat-based tutors draw not only from problem statements and automated testing but also from chain-of-thought prompting and self-check capabilities, balancing immediate feedback with policy controls against full-solution disclosure (Bassner et al., 9 May 2024).

Innovations in knowledge tracing embed LLMs both for annotating dialogue turns with knowledge component (KC) tags and for fine-tuning mastery prediction models that operate on full-dialog textual data (Scarlatos et al., 24 Sep 2024). For example, the LLMKT method leverages dialogue and KC descriptions as joint context:

$$\hat{z}_{jk} = P_\theta(z_{jk} = 1) = \frac{\exp(v^T_\theta(\mathbf{context}))}{\exp(v^T_\theta(\mathbf{context})) + \exp(v^F_\theta(\mathbf{context}))}$$

with the turn-level correct prediction given by averaging over involved KCs. This approach outperforms traditional deep knowledge tracing models, particularly in settings with open-ended dialogicity and multiple concurrent skills.

7. Limitations, Risks, and Directions for Future Research

Current ChatGPT-based tutors are limited by:

• Imperfect diagnostic accuracy and feedback quality, including a consistent (e.g., 30%) error or rejection rate for algebra hints (Pardos et al., 2023) and unreliable program correctness or code review (Anishka et al., 2023, Ballestero-Ribó et al., 24 Jan 2025).
• The risk of artificial hallucination, where confidently erroneous answers or non-existent references can be generated (Ling, 2023, Ballestero-Ribó et al., 24 Jan 2025).
• Inconsistency and unreliability in response to the same query (multiple possible answers), requiring robust verification and human oversight (Ling, 2023, Joshi et al., 2023).
• Domain-specific weaknesses, such as graphical input misinterpretation in physics or incomplete edge-case handling in math/functional programming.

Identified future priorities include:

• Systematic integration of external knowledge bases and course-specific materials to enhance retrieval-augmented generation (Wang et al., 2023, Vu et al., 26 Jul 2025).
• Development of high-granularity prompt engineering and meta-prompt strategies to increase output validity and context specificity.
• Deployment of semi-automated quality estimation modules to triage and filter LLM outputs prior to learner exposure (Ballestero-Ribó et al., 24 Jan 2025).
• Human-in-the-loop architectures for hybrid AI/human tutoring and dynamic error correction.
• Expanded evaluation across more diverse learner populations, problem domains, and real-world educational contexts, particularly in settings that require personalization, accessibility, and robust dialogic repair.

In sum, the ChatGPT-based tutor constitutes a rapidly evolving paradigm in educational technology, characterized by scalable content delivery, adaptive dialogic support, and opportunities for rigorous, semi-automated assessment. However, consistent with research findings, widespread deployment for high-stakes learning currently requires careful scaffolding, systematic quality verification, robust prompt engineering, and human educator partnership.