Tutor CoPilot: A Human-AI Approach for Scaling Real-Time Expertise (2410.03017v1)

Abstract: Generative AI, particularly LLMs (LMs), has the potential to transform real-world domains with societal impact, particularly where access to experts is limited. For example, in education, training novice educators with expert guidance is important for effectiveness but expensive, creating significant barriers to improving education quality at scale. This challenge disproportionately harms students from under-served communities, who stand to gain the most from high-quality education. We introduce Tutor CoPilot, a novel Human-AI approach that leverages a model of expert thinking to provide expert-like guidance to tutors as they tutor. This study is the first randomized controlled trial of a Human-AI system in live tutoring, involving 900 tutors and 1,800 K-12 students from historically under-served communities. Following a preregistered analysis plan, we find that students working with tutors that have access to Tutor CoPilot are 4 percentage points (p.p.) more likely to master topics (p<0.01). Notably, students of lower-rated tutors experienced the greatest benefit, improving mastery by 9 p.p. We find that Tutor CoPilot costs only $20 per-tutor annually. We analyze 550,000+ messages using classifiers to identify pedagogical strategies, and find that tutors with access to Tutor CoPilot are more likely to use high-quality strategies to foster student understanding (e.g., asking guiding questions) and less likely to give away the answer to the student. Tutor interviews highlight how Tutor CoPilot's guidance helps tutors to respond to student needs, though they flag issues in Tutor CoPilot, such as generating suggestions that are not grade-level appropriate. Altogether, our study of Tutor CoPilot demonstrates how Human-AI systems can scale expertise in real-world domains, bridge gaps in skills and create a future where high-quality education is accessible to all students.

Overview of Tutor CoPilot: A Human-AI Approach to Scaling Real-Time Expertise

This paper introduces Tutor CoPilot, a Human-AI system designed to scale expertise in real-time educational settings using generative AI, specifically LLMs (LMs). The paper aims to alleviate the challenges of providing quality educational guidance to novice tutors, particularly in under-served communities where access to skilled educators is limited. The paper presents the first randomized controlled trial of such a system involving 900 tutors and 1,800 K-12 students.

Key Findings and Methodology

Tutor CoPilot leverages the latent reasoning of expert educators to provide on-the-fly, expert-like suggestions to tutors. In the randomized controlled trial, students whose tutors used Tutor CoPilot showed a 4 percentage point improvement in mastering mathematics topics, with a p-value of less than 0.01. Notably, the improvement was more pronounced for students of lower-rated tutors, who exhibited a 9 percentage point increase, indicating the system's potential to enhance learning outcomes where it's most needed.

The system operates at a low cost of \$20 per tutor annually, making it a scalable alternative to traditional, resource-intensive training programs. Analysis of chat messages revealed that Tutor CoPilot promotes pedagogical strategies aligning with high-quality teaching practices, such as asking guiding questions, rather than simply delivering answers.

Implications for Education and AI

The paper highlights crucial implications for the integration of AI in educational environments. By demonstrating that AI can provide real-time, context-specific guidance, it opens pathways to improving the quality of education delivered by novice tutors without the need for extensive and expensive human-led training programs. It underscores the potential for AI to democratize access to high-quality education, particularly in under-served communities that benefit the most from expert guidance.

In terms of future AI developments, this work suggests the possibility of expanding human-AI collaborative systems across various domains that require nuanced decision-making and expert intervention, such as healthcare and law. It also points to the importance of incorporating multi-modality into these systems, potentially integrating vision and speech for a more comprehensive interaction model.

Limitations and Future Work

The paper acknowledges limitations regarding the generalizability of the findings, both within the educational field and across other domains. The paper is conducted in a specific demographic setting, and further research is required to assess its efficacy across different educational contexts and international settings. Additionally, there is scope to investigate the retention of skills among novice educators facilitated by AI systems and to explore long-term educational outcomes beyond immediate mastery metrics.

Moreover, considerations around user privacy and safety are emphasized, with the paper setting a precedent for safeguarding sensitive educational interactions in AI deployments. Future research could focus on addressing privacy challenges, particularly as the field moves towards multi-modal and more comprehensive AI systems.

In conclusion, Tutor CoPilot represents a viable Human-AI approach to enhancing educational experiences and scaling expertise in real-time, where traditional methods fall short. Its implications for scalable, cost-effective, and high-quality educational interventions provide a significant step towards more equitable education systems globally.