- The paper introduces SocratiQ, a generative AI learning companion that applies the Socratic method to provide personalized and accessible education, initially integrated into a Harvard Machine Learning course.
- SocratiQ features include personalized explanations at varying difficulty levels, adaptive LLM-generated assessments tied to a knowledge graph, bounded learning focused on course material, and gamification to enhance student engagement.
- The system uses a local-first, serverless architecture with cost optimization strategies, supports secure student progress sharing via cryptographic hashing, and was evaluated through Bloom's Taxonomy analysis and user feedback.
The paper introduces SocratiQ, an AI-powered learning companion designed to enhance personalized education and accessibility, particularly in STEM fields. SocratiQ implements the Socratic method through adaptive learning technologies, dynamically creating personalized learning pathways based on student responses and comprehension. The core principle behind SocratiQ is "Generative Learning," which emphasizes actively connecting new information to prior knowledge through interactive dialogue, personalized explanations, and adaptive assessments.
The authors integrated SocratiQ into an online Machine Learning Systems textbook developed at Harvard University for the CS249r course. The objectives of this integration are threefold: enhancing student engagement, providing personalized learning pathways, and broadening accessibility to high-quality machine learning education.
The paper reviews the evolution of AI-enabled educational tools, categorizing them into three waves:
- First Wave: Adaptive Learning systems like Intelligent Tutoring Systems (ITS) such as MATHia, ALEKS, and DreamBox, which offered rule-based personalization and automated assessments.
- Second Wave: Specialized AI tools like Duolingo, Wolfram Alpha, and Grammarly, which enhanced domain expertise and provided task-specific AI assistance.
- Third Wave: Generative AI systems like ChatGPT and Gemini, which enable natural conversation, context understanding, and dynamic adaptation.
The authors present a framework (Table 1) for understanding the possible AI roles in a classroom setting, including learning companion, teaching assistant, and assessment coordinator, along with design considerations ranging from individual learning to institutional implementation. SocratiQ primarily focuses on the role of a learning companion.
The paper discusses the importance of balancing AI with human-centered education, arguing that AI systems should complement rather than replace traditional educational roles. The authors emphasize the need for systematic frameworks to guide the integration of AI into education, carefully balancing AI capabilities with traditional pedagogical strengths.
The key features of the SocratiQ system design include:
- Personalized Explanations: SocratiQ provides four difficulty levels (Beginner, Intermediate, Advanced, Expert) to tailor explanations to the student's level of understanding. These levels are implemented as system prompts provided to the Language Learning Model (LLM).
- Adaptive Assessments: The system leverages an LLM to dynamically generate assessments, providing learners with frequent opportunities to check their understanding and track their progress. As students progress through the textbook sections and complete quizzes, a knowledge graph is constructed to outline various topics and subtopics relevant to their coursework.
- Bounded Learning: The solution emphasizes textbook material as the primary focus of the LLM through in-context prompts while retaining access to its broader pre-trained knowledge.
- Gamification: SocratiQ uses gamification and progress tracking to enhance user engagement and provide opportunities for self-assessment. This is achieved by implementing progress tracking, streaks, passing quiz attempts, badges, and engagement heatmaps.
The implementation of SocratiQ is organized into four main stages: Initial Setup, Learning Flow, Quiz Generation, and Progress {paper_content} Gamification. The authors use Groq for inference on LLMs.
To address API call limitations, the authors use a combination of LLMs, including Mixstral-8x7b, Gemma 7b, and LLama 3.2. In the event of Groq unavailability, they rely on other services such as Google Gemini as a backup. To manage token limits during quiz generation, the system employs a selective text inclusion strategy. They implement a question caching and reuse strategy to further optimize system performance and reduce costs.
The authors provide a cost analysis for serverless functions using various providers (Azure, AWS, Cloudflare) and AI models. The analysis shows that Mixtral-8x7b and Gemini are the most cost-effective options. SocratiQ is designed for efficient scalability, utilizing a "local first" approach that minimizes back-end demands. This is combined with a serverless architecture hosted on Azure.
SocratiQ facilitates integration with course structures by allowing students to securely share their progress with instructors through a system of cryptographically hashed PDFs. The authors describe the cryptographic hashing process in Algorithm 3, which demonstrates how hashing is employed to create tamper-evident progress reports. The system is designed with a "local-first" architecture. User data is stored locally in the browser's IndexedDB, while selective centralized caching improves quiz quality and reduces operational costs.
The paper evaluates SocratiQ's effectiveness in supporting self-paced learning through an analysis of AI-generated questions and a limited case paper in a sprint-style machine learning course. They use Bloom's Taxonomy to evaluate the cognitive depth and relevance of AI-generated questions. The analysis reveals that lower-order cognitive skills dominate at the beginner level, with remembering at 42% and understanding at 28%. The paper presents examples of AI-generated questions categorized by chapters and difficulty levels. Student feedback highlights the system's effectiveness in providing interactive and engaging learning experiences while pointing to opportunities for enhancing user interface design and question diversity.