SimClass Framework: Classroom ASR & LLM Teaching

• SimClass Framework is a modular platform that simulates realistic classroom scenarios through synthetic speech data and LLM-powered interactive teaching.
• It generates large-scale, labeled datasets by combining clean classroom speech with simulated noise under varied SNR conditions for robust ASR benchmarking.
• Its multi-agent system orchestrates dynamic classroom interactions, enabling in-depth studies on dialogue, engagement, and pedagogical effectiveness.

The SimClass Framework encompasses two principal research-driven systems: a scalable methodology for generating large-scale simulated classroom speech and noise datasets for automatic speech recognition (ASR) research, and a LLM-empowered, multi-agent framework for simulating interactive classroom teaching. Both are established as open, modular platforms enabling robust experimentation, benchmarking, and exploration of speech-driven AI in educational contexts (Attia et al., 10 Jun 2025, Zhang et al., 2024).

1. Definitions and Scope

SimClass refers to:

1. A modular pipeline for constructing realistic, labeled classroom speech datasets by simulating both “clean” and noisy classroom audio through high-fidelity methods, tailored for ASR and speech enhancement model evaluation (Attia et al., 10 Jun 2025).
2. A multi-agent classroom simulation teaching framework, in which LLM-powered agents (teachers, assistants, classmates) and a single real student interact under the direction of an orchestrating controller, enabling research in dialog, engagement, and collaborative education dynamics (Zhang et al., 2024).

Both frameworks address the scarcity of in-situ, large-scale, and systematically varied classroom data—or interaction environments—necessary for robust AI model training and evaluation.

2. Architecture and Components

2.1 SimClass for ASR Dataset Generation

SimClass introduces a modular, end-to-end pipeline:

• Data Ingestion and Partitioning
  • Sources: MyST children’s speech (~210 h transcribed/183 h untranscribed), MIT-OCW lectures (~174 h, human transcripts), Khan Academy (~7.5 h, human transcripts).
  • Partitioning is speaker- and channel-disjoint (train/dev/test: 313 h/37 h/41 h).
• Clean Speech Base Construction
  • Tracks are normalized into classroom-style speaker roles (children, lecturer).
• Classroom Noise Synthesis
  • Realistic classroom “babble” noise is generated with 3D acoustic simulation using a game engine, yielding a synthetic classroom noise corpus.
• Noisy Speech Mixing
  • Clean classroom speech and simulated noise are combined under controlled signal-to-noise ratios (SNRs), producing fully labeled “noisy classroom speech” suitable for ASR and enhancement benchmarks.

2.2 SimClass for LLM-Empowered Teaching

SimClass is organized around three subsystems:

• Session Controller
  • Maintains classroom state, executes the “Manager Agent” (meta-agent), and dispatches which participant speaks and what function is invoked next.
• Class Role Agents
  • LLM-derived agents are instantiated by role-prompting. The agent taxonomy is

  $$A = \{ a_0\,(\text{Teacher}),\, a_1\,(\text{Assistant}),\, a_2\ldots a_5\,(\text{Classmates}),\, a_m\,(\text{Manager}) \}$$ - Each $a_i = \rho(\text{LLM}, P_{A_i})$, where $P_{A_i}$ is the system prompt for role, personality, and allowed functions.

• User

  • A real human student engages via chat.
• Function Library
  • Tutoring ($f_X$) and interaction ($f_Y$) functions implement slide display, question answering, encouragement, and discipline control.
• Manager Agent
  • Receives class state $S_t = \{C_t, H_t \mid \hat{R}\}$; outputs “control signals” $s_t = (a_t, f_t)$ specifying which agent and function act next:

  $$s_t = \mathcal{L}(S_t), \quad S_{t+1} = \text{Update}(S_t, s_t)$$

3. Methodologies and Control Mechanisms

3.1 ASR-Focused Data Generation

SimClass systematically constructs datasets:

• Corpus Construction

  • Clean “classroom speech base” is synthesized through cross-domain alignment of children’s and lecture corpora to represent classroom dialog.
• Noise Corpus Generation
  • Children’s babble and physical acoustic characteristics are modeled by simulating virtual classrooms in a 3D game engine, generating spatially robust, dynamically evolving noise backgrounds.
• Data Fusion
  • Multiple SNR conditions are applied to combine clean speech and noise, fully labeled for supervised learning.

3.2 LLM-Driven Dialog Control

A multi-agent collaborative simulation is orchestrated as follows:

• Dialogue State Tracking
  • Maintains history $H_t$ (utterances), covered materials $C_t$, role set $a_i = \rho(\text{LLM}, P_{A_i})$0.
• Role-Specific Policies
  • Teacher ($a_i = \rho(\text{LLM}, P_{A_i})$1): Delivers content, answers, or advances slides.
  • Assistant ($a_i = \rho(\text{LLM}, P_{A_i})$2): Provides clarifications, encouragement, minor discipline.
  • Classmates ($a_i = \rho(\text{LLM}, P_{A_i})$3): Generate peer dialog, commentary, questions; each prompted by archetype.
  • Manager ($a_i = \rho(\text{LLM}, P_{A_i})$4): Selects $a_i = \rho(\text{LLM}, P_{A_i})$5 given $a_i = \rho(\text{LLM}, P_{A_i})$6 via single LLM call.
• Centralized Turn-Taking
  • This protocol enforces coherent progression and active dialog, supporting emergent group dynamics by allowing agents to respond to user and peer outputs in real time.

4. Evaluation Frameworks and Metrics

4.1 Experimental ASR Validation

Downstream ASR and speech enhancement models are trained/evaluated on SimClass data. Systematic comparison to real classroom benchmarks demonstrates that simulated data approximates the recognition challenges and acoustical realities of live classrooms (Attia et al., 10 Jun 2025).

4.2 Behavioral and Pedagogical Analysis

• Flanders Interaction Analysis System (FIAS)
  • Each utterance is classified into ten interaction categories (Teacher Indirect 1–4, Teacher Direct 5–7, Student Response 8, Student Initiation 9, Silence 10).
  • Class transcripts become Markov sequences, aggregated in a $a_i = \rho(\text{LLM}, P_{A_i})$7 transition matrix $a_i = \rho(\text{LLM}, P_{A_i})$8.
  • Derived metrics:
  • Teacher Talk (TT), Student Talk (ST), Indirect/Direct Ratio (IDR), Student Initiation Ratio (SIR).
• Community of Inquiry (CoI)
  • Subjective assessment of cognitive presence, teaching presence, and social presence on 0–2 scales, averaged across participants.

5. Empirical Findings and Emergent Dynamics

5.1 Quantitative Outcomes

• Full SimClass achieves TT $a_i = \rho(\text{LLM}, P_{A_i})$9, ST $P_{A_i}$0—closely matching in situ classroom norms (TT $P_{A_i}$1–$P_{A_i}$2, ST $P_{A_i}$3–$P_{A_i}$4).
• Ablations (removal of classmates, disabling interactions) significantly decrease student talk (ST approaches zero) and lower cognitive/social presence (measured by CoI survey, $P_{A_i}$5 by standard error bars), while teaching presence remains high ($P_{A_i}$6) when scripts are used (Zhang et al., 2024).

5.2 Observed Group Behaviors

Emergent phenomena in agent interactions include:

• Collaborative Teaching/Discussion: Classmate agents follow up on user questions, deepening discourse.
• Emotional Companionship: Non-teacher agents dynamically encourage users, especially after errors or disengagement.
• Discipline Control: Classmate and assistant roles steer discussion back to topic as needed.

This suggests that multi-agent design, not just a single LLM tutor, is critical for simulating a robust and engaging classroom environment.

6. Limitations and Prospects

Key limitations identified:

• Dependence on a single LLM (GPT-4) and restricted agent diversity (four classmate types).
• Limited classroom function set $P_{A_i}$7 (no quizzes, polls, whiteboards at present).

Identified directions for future research include: integration of more varied and specialized roles (e.g., group leader, evaluator), expansion of the function/action library, exploration with open-source or multimodal LLM backbones, and longitudinal studies of learning outcomes and knowledge retention (Zhang et al., 2024). For dataset construction, expanding noise simulation to more challenging acoustic conditions and broader domains is considered a logical extension (Attia et al., 10 Jun 2025).