A Simulation-Based Method for Testing Collaborative Learning Scaffolds Using LLM-Based Multi-Agent Systems

Abstract: Background: Traditional research on collaborative learning scaffolding is often time-consuming and resource-heavy, which hinders the rapid iteration and optimization of instructional strategies. LLM-based multi-agent systems have recently emerged as a powerful tool to simulate complex social interactions and provide a novel paradigm for educational research. Objectives: This study proposes an LLM-based multi-agent simulation approach to investigate collaborative learning processes and the effectiveness of instructional scaffolds prior to actual classroom deployment. The research specifically examines the feasibility of simulating group discussions and the alignment of these simulations with established learning science theories. Methods: The simulation system was implemented using the MetaGPT framework and GPT-4o, comprising one teacher agent and five distinct student roles (Leader, Supporter, Expounder, Rebutter, and Summarizer). Two scaffolding strategies, "Deep Think before Speak" and "Direct Speak", were compared across ten classical Chinese poetry appreciation tasks. Evaluation was conducted through discourse analysis of quality and behavior. Results and Conclusions: The introduction of the "Deep Think before Speak" scaffold significantly improved the agents' discourse diversity and interaction depth while notably reducing content repetitiveness. Behavioral analysis showed that the scaffold encouraged more complex interaction patterns, such as reflecting, rebutting, and explaining. These findings align with the ICAP framework, as the scaffold prompted agents to move from simple "Active" participation to "Constructive" and "Interactive" knowledge co-construction. This study demonstrates the feasibility and ecological validity of using LLM-based multi-agent systems to simulate authentic collaborative learning dynamics.

• The paper presents a simulation-based approach using LLM-driven multi-agent systems to test collaborative learning scaffolds before real-world deployment.
• It compares ‘Deep Think before Speak’ and ‘Direct Speak’ conditions, showing significant improvements in discourse diversity and reduced repetitiveness.
• The study validates a rapid, cost-effective framework for evaluating instructional strategies, aligning simulation outcomes with established educational theories.

Simulation-Based Evaluation of Collaborative Learning Scaffolds with LLM-Based Multi-Agent Systems

Introduction

This paper presents a simulation-based approach for systematically testing collaborative learning scaffolds using LLM-driven multi-agent systems, instantiated through the MetaGPT framework and GPT-4o. Addressing the inefficiencies associated with traditional, human-centric empirical studies in collaborative learning, the authors propose an automated methodology enabling iterative, high-fidelity evaluation of instructional scaffolds prior to real-world deployment. The approach leverages the cognitive and generative capacities of state-of-the-art LLMs to simulate authentic group interactions, enriched with formalized agent roles. The comparative study centers on metacognitive scaffolds—specifically, "Deep Think before Speak" versus "Direct Speak"—within simulated student-teacher discourse aimed at classical Chinese poetry appreciation.

Methodological Framework

System Architecture

The collaborative simulation environment is comprised of three principal modules: Communication, Memory, and Agent. The Agent module distinguishes between a teacher agent (responsible for task orchestration and metacognitive feedback) and five predefined student agents assuming differentiated functional roles (Leader, Supporter, Expounder, Rebutter, Summarizer). Prompt engineering and Chain-of-Thought (CoT) protocols are applied to induce role-aligned, contextually grounded behavior, ensuring that agent discourse and cognitive dynamics model those observed in human collaborative learning settings.

Scaffolding Conditions

• Deep Think Before Speak: Agents are required to undergo structured reflective processes incorporating content analysis, instruction interpretation, multi-turn context tracking, and differentiated contribution before response generation. Prompt templates enforce the sequentialization of CoT steps.
• Direct Speak: Agents respond reactively to prompts from other agents or the teacher without enforced reflection or context integration.

Experimental Tasks and Evaluation

Ten tasks derived from a university-level poetry appreciation curriculum were operationalized, each with instructor-vetted scoring rubrics. Agent discourse was coded along five dimensions: fluency, repetitiveness, contradiction, relevance, and diversity. Behavioral analyses encompassed planning, monitoring, reflecting, elaboration, support, questioning, rebutting, and explanation. Coding reliability was ensured through substantial human-model agreement (Kappa ≥ 0.75), with automatic and manual verification conducted on all outputs.

Empirical Results

Discourse Quality

The "Deep Think before Speak" scaffold elicited statistically significant improvements in diversity of agent-generated viewpoints ($p_{adj}=0.016$), and a marked reduction in content repetitiveness ($p_{adj}=0.012$) relative to "Direct Speak." Indicators of contradiction and irrelevance were near zero for both conditions, signifying high logic and coherence intrinsic to the LLM simulation. Notably, mean discourse length and output volume did not differ significantly across scaffolding strategies, indicating that enhancements in quality were not achieved at the cost of brevity or information density.

A role-disaggregated analysis in the structured scaffold condition revealed:

• Leader and Rebutter agents generated the most novel and least redundant discourse.
• Supporter and Summarizer roles were associated with high repetitiveness, functionally consistent with consensus-building and synthesis.

Behavioral Transitions

The "Deep Think before Speak" condition activated richer, higher-order behavioral transitions:

• Substantial increases were observed in reflect, plan, rebut, and explain codes (all $p_{adj}<0.045$).
• The resulting discussion flow included extensive multi-step argumentation, with explain-support-question-rebut-reflect cycles, closely mirroring authentic collaborative discourse models in educational psychology.
• Teacher agents in the structured reflection condition reduced directive feedback, focusing instead on encouragement and affirmation, in line with increased student agent autonomy.

By contrast, "Direct Speak" yielded limited behavior diversity, primarily featuring linear utterance-support-question transitions with little evidence of integrative or reflective cycles.

Theoretical and Practical Implications

The induced agent interaction patterns are congruent with central learning science theories. In particular:

• The scaffolding prompts facilitate student agents' migration from "Active" toward "Constructive" and "Interactive" cognitive engagement per the ICAP taxonomy, operationalizing meaningful knowledge co-construction.
• The system realistically generates the full spectrum of collaborative learning behaviors—task initiation, consensus formation, creative reasoning, and critical opposition—validating its ecological and theoretical fidelity.

From a research methodology standpoint, the simulation enables rapid, large-scale, and cost-effective testing of collaborative scaffolds, circumventing logistical, ethical, and resource constraints of classroom-based intervention studies. This paradigm supports iterative instructional design, enabling cycle-to-cycle optimization informed by robust behavioral and linguistic analytics.

Future Perspectives

The scalability and agentic flexibility of LLM-based simulation platforms such as the one described provide a road map for future AI-supported educational research. Promising avenues include:

• Integrating real-time adaptation to stochastic or adversarial conditions (e.g., uncooperative agents, heterogeneous prior knowledge).
• Bridging simulation-to-practice by grounding scaffold recommendations in both synthetic and empirical evidence.
• Extending agent fidelity for multimodal tasks (e.g., image-annotation, math-rich dialogue), leveraging LLM multi-modal capacities.
• Evolving scaffold complexity beyond metacognitive prompts to dynamic orchestration contingent on group macro-behaviors.

From a theoretical perspective, such frameworks offer empirical means to test and refine sociocognitive models of group interaction, epistemic agency, and the emergence of collective intelligence within artificial societies.

Conclusion

This study advances a rigorously validated, simulation-based method for evaluating collaborative learning scaffolds using LLM-powered multi-agent systems (2604.11161). The results offer strong evidence that prompt-engineered metacognitive scaffolds, instantiated in agent-based environments, generate deep and diverse collaborative discourse aligned with established learning theories. The presented approach materially contributes to both the design research methodology in collaborative learning and the theoretical understanding of group-level knowledge construction in computational social science. Future work should emphasize transfer validity to human learners and real-world classroom practice, as well as the development of even more adaptive and context-sensitive educational agents.