Using Generative AI and Multi-Agents to Provide Automatic Feedback (2411.07407v1)

Abstract: This study investigates the use of generative AI and multi-agent systems to provide automatic feedback in educational contexts, particularly for student constructed responses in science assessments. The research addresses a key gap in the field by exploring how multi-agent systems, called AutoFeedback, can improve the quality of GenAI-generated feedback, overcoming known issues such as over-praise and over-inference that are common in single-agent LLMs. The study developed a multi-agent system consisting of two AI agents: one for generating feedback and another for validating and refining it. The system was tested on a dataset of 240 student responses, and its performance was compared to that of a single-agent LLM. Results showed that AutoFeedback significantly reduced the occurrence of over-praise and over-inference errors, providing more accurate and pedagogically sound feedback. The findings suggest that multi-agent systems can offer a more reliable solution for generating automated feedback in educational settings, highlighting their potential for scalable and personalized learning support. These results have important implications for educators and researchers seeking to leverage AI in formative assessments, offering a pathway to more effective feedback mechanisms that enhance student learning outcomes.

• The paper presents a dual-agent framework, AutoFeedback, that reduces over-praise from 15.42% to 1.25% and over-inference from 27.20% to 7.08%.
• It employs a state-of-the-art LLM for initial feedback generation, followed by a quality control agent to refine responses in science assessments.
• The study demonstrates a practical GenAI approach for delivering personalized, accurate feedback that enhances student learning in educational settings.

Analysis of Multi-Agent Systems for Enhancing Automated Feedback in Educational Assessments

The paper under review develops and evaluates an innovative approach to automatic feedback generation in educational settings using multi-agent systems. Specifically, it addresses two prevalent issues in Generative AI (GenAI)-generated feedback: over-praise and over-inference. The researchers introduce a novel multi-agent framework, named AutoFeedback, consisting of two distinct AI agents: one for generating feedback and another for validating and revising this initial output. This paper's focus on educational assessments, particularly within science contexts, highlights the system's potential for improving pedagogical practices by offering more nuanced and precise feedback to students’ responses.

Methodology and Implementation

In the implementation, the AutoFeedback system utilizes a two-agent architecture: the first agent produces preliminary feedback using state-of-the-art LLMs like GPT-4, while the second agent acts as a quality control measure, identifying and rectifying any instances of over-praise or over-inference. The system was tested against a dataset of 240 student responses to a scientifically-focussed assessment task. Notably, this task evaluates students' understanding of thermal energy's effect on particle motion, aligned with middle school physical science learning objectives. The performance of the multi-agent system was then compared to feedback generated by a single-agent LLM, revealing marked improvements through the use of the dual-agent system.

Key Findings and Performance

The results indicate that the AutoFeedback system significantly mitigates the over-praise and over-inference issues. Specifically, over-praise occurrences decreased from 15.42% to 1.25%, and over-inference from 27.20% to 7.08%. These findings demonstrate the efficacy of the multi-agent system over single-agent approaches, thus suggesting an advanced strategy for deploying GenAI in educational environments. Such reduction in error rates is critical as it ensures that feedback closely aligns with student performance, providing realistic and constructive guidance for improvement. This enhancement is pivotal in fostering authentic student learning and providing educators with a tool to support individualized student progress effectively.

Implications and Future Directions

The paper presents significant implications for educational technology, particularly in developing tools that augment the teacher's ability to deliver tailored and real-time feedback in large-scale educational scenarios. By embracing a multi-agent paradigm, the paper bridges a critical gap in the automatic feedback generation domain, offering a more dependable and context-sensitive alternative to traditional single-agent LLMs.

However, there is room for further research to explore the scalability and adaptability of this system across different subjects and educational levels. Investigating the system's applicability in real-world classroom settings would be a logical next step, as would considering the role of such automated systems in comprehensive educational policy frameworks. Additionally, extending research to incorporate more diverse datasets and feedback types could offer insights into optimizing multi-agent systems for maximal educational impact.

Overall, this paper demonstrates the potential of multi-agent systems to advance the field of automated educational feedback, creating pathways for more personalized and effective learning interventions. As AI continues to evolve, further exploration into multi-agent frameworks could pave the way for more sophisticated, flexible, and impactful educational technologies.