Effective and Scalable Math Support: Evidence on the Impact of an AI- Tutor on Math Achievement in Ghana (2402.09809v2)

Abstract: This study evaluates the impact of Rori, an AI powered conversational math tutor accessible via WhatsApp, on the math performance of approximately 1,000 students in grades 3-9 across 11 schools in Ghana. Each school was assigned to a treatment group or control group; the students in the control group continued their regular math instruction, while students in the treatment group engaged with Rori, for two 30-minute sessions per week over 8 months in addition to regular math instruction. We find that the math growth scores were substantially higher for the treatment group with an effect size of 0.37, and that the results were statistically significant (p < 0.001). The fact that Rori works with basic mobile devices on low-bandwidth data networks gives the intervention strong potential to support personalized learning on other low-and-middle-income countries (LMICs), where laptop ownership and high-speed internet - prerequisite for many video-centered learning platforms - remain extremely limited. While the results should be interpreted judiciously, as they only report on year 1 of the intervention, and future research is necessary to better understand which conditions are necessary for successful implementation, they do suggest that chat-based tutoring solutions leveraging artificial intelligence could offer a costeffective approach to enhancing learning outcomes for millions of students globally.

• The paper demonstrates that Rori significantly improves math scores with a statistically significant effect size of 0.37 (p < 0.001).
• It employs a controlled design dividing schools into treatment and control groups, using two 30-minute weekly sessions over eight months.
• The study highlights the scalability and cost-effectiveness of AI tutoring via mobile platforms in enhancing educational outcomes in LMICs.

Evaluation of an AI-Powered Math Tutor in Ghana's Education System

This paper presents a comprehensive evaluation of "Rori," an AI-driven conversational math tutor deployed via WhatsApp, and its impact on math achievement among approximately 1,000 students in grades 3-9 across 11 schools in Ghana. The paper aims to assess the potential of leveraging mobile platforms to deliver personalized learning experiences in Low- and Middle-Income Countries (LMICs) where access to personal computers and high-speed internet is limited.

Research Design and Implementation

The schools were divided into treatment and control groups, with the treatment group utilizing Rori for two 30-minute sessions weekly over eight months, supplementing their regular math instruction. The control group continued with standard instruction. The use of basic mobile devices makes Rori an accessible tool, circumventing the infrastructural constraints typical in LMICs.

Numerical Results and Impact

The analysis demonstrated statistically significant improvements in math growth scores for the treatment group, with an effect size of 0.37 (p < 0.001). These results signify a moderate to large effect size, aligning with established benchmarks in educational research. This showcases Rori's efficacy in enhancing math learning outcomes without necessitating substantial technological investments.

Theoretical and Practical Implications

The findings suggest that Rori effectively adapts to different learning levels across diverse grade bands, highlighting its potential utility in various educational contexts. The scalability of such interventions is underscored by the relatively low cost and the strategic utilization of widely available mobile technologies like WhatsApp. This aligns with existing literature on Intelligent Tutoring Systems (ITSs) and adaptive learning environments, reinforcing the viability of chat-based tutoring as a model for improving educational outcomes in resource-constrained settings.

Future Directions and Considerations

While the results are promising, further research is needed to validate these outcomes across different contexts and refine implementation strategies. Future studies may adopt more comprehensive assessment tools and standardized grading protocols to enhance result reliability and external validity. Investigating the interaction between dosage and learning gains could also yield insights into optimizing student engagement with AI-driven platforms.

Moreover, extending this research across various geographic locations or examining Rori's effectiveness in home-learning scenarios could contribute to understanding its broader applicability and scalability. Pursuing fully randomized controlled trial designs in subsequent research would likely bolster the robustness of the findings.

Conclusion

The paper sheds light on the potential of AI-driven chat-based tutoring systems like Rori in addressing educational challenges in LMICs. While initial results are significant, highlighting the intervention's scalability and cost-effectiveness, continuous evidence generation and methodological refinements are essential to establish comprehensive insights into its efficacy. The ongoing exploration of personalized learning interventions remains crucial in enhancing educational equity and optimizing learning outcomes for students in diverse environments.