TeamVision: An AI-powered Learning Analytics System for Supporting Reflection in Team-based Healthcare Simulation

Abstract: Healthcare simulations help learners develop teamwork and clinical skills in a risk-free setting, promoting reflection on real-world practices through structured debriefs. However, despite video's potential, it is hard to use, leaving a gap in providing concise, data-driven summaries for supporting effective debriefing. Addressing this, we present TeamVision, an AI-powered multimodal learning analytics (MMLA) system that captures voice presence, automated transcriptions, body rotation, and positioning data, offering educators a dashboard to guide debriefs immediately after simulations. We conducted an in-the-wild study with 56 teams (221 students) and recorded debriefs led by six teachers using TeamVision. Follow-up interviews with 15 students and five teachers explored perceptions of its usefulness, accuracy, and trustworthiness. This paper examines: i) how TeamVision was used in debriefing, ii) what educators found valuable and challenging, and iii) perceptions of its effectiveness. Results suggest TeamVision enables flexible debriefing and highlights the challenges and implications of using AI-powered systems in healthcare simulation.

• The paper introduces TeamVision, an AI-powered system that automates multimodal data analysis to support reflective debriefing in team-based healthcare simulations.
• It employs advanced sensing technologies and visual analytics to provide actionable insights on team communication and coordination, validated by positive feedback from educators and students.
• User evaluation with 56 teams and 221 students demonstrated improved debriefing outcomes and flexibility, underscoring the system's potential to revolutionize reflective practices in medical education.

TeamVision: An AI-powered Learning Analytics System for Supporting Reflection in Team-based Healthcare Simulation

This essay presents an in-depth analysis of "TeamVision: An AI-powered Learning Analytics System for Supporting Reflection in Team-based Healthcare Simulation" (2501.09930). The paper introduces TeamVision, an AI-powered multimodal learning analytics (MMLA) system designed to enhance reflective practices in team-based healthcare simulations. Through the integration of advanced AI techniques, TeamVision aims to bridge the gap in providing concise, data-driven summaries that effectively support debriefing in healthcare education.

Introduction and Objectives

Simulation-based learning has gained traction in healthcare education as a safe yet realistic modality for developing clinical and teamwork skills. Structured debriefs post-simulation enable participants to reflect critically on their actions. However, there's a gap in leveraging video content effectively due to its cumbersome nature, limiting its utility for evidence-based debriefing. TeamVision addresses this gap by utilizing AI to automate the capture and processing of multimodal data—voice, positioning, and body metrics during high-fidelity simulations—and offering them through a user-friendly dashboard to educators guiding debriefs.

Methodology: AI-Powered Multimodal Learning Analytics (MMLA)

TeamVision incorporates advanced sensing technologies and AI to gather comprehensive multimodal data. The system tracks voice activity, spatial parameters, and body orientation. These are translated into actionable analytics using sophisticated algorithms to surface evidence of team dynamics, communication patterns, and task prioritization strategies.

Figure 1: Catalogue of visualisations and sources in the debrief control view: A) Priority chart, B) Speech and location ward map, C) Speech sociogram, D) Communication network, and E) Video snippet.

From the data, TeamVision produces visualizations and metrics—such as priority charts, sociograms, and speech-location maps—providing educators with detailed insights into team coordination and communication strategies.

In-the-Wild Study and Insights

The utility of TeamVision was validated through an in-the-wild study with 56 teams constituting 221 students, facilitated by six educators. The qualitative feedback and interaction data were analyzed to discern the system's impact on enabling flexible and iterative debriefs.

Key findings highlighted the following:

1. Enhanced Debriefing: TeamVision allowed for nuanced, data-driven discussions, significantly improving educators' ability to tailor debrief to specific learning outcomes, including examining communication efficacy and teamwork roles.
2. Perceived Accuracy and Trustworthiness: Educators and students provided positive feedback on TeamVision's accuracy and trustworthiness, citing enhanced reflective practices as a critical benefit. However, initial unfamiliarity with the system was noted as a barrier, though easily mitigated by training and iterative use.
3. Flexibility and Usability: The adaptable nature of TeamVision was positively received, enabling educators to customize visual analytics to match specific team performances during debriefing sessions seamlessly.

   Figure 2: Top: Strategies followed by educators while using the filtering option during their debrief sessions. Note how educators mostly used only the information of one particular phase to generate their corresponding visualisations and support their discussions. Bottom: Frequency of the use of visualisations per phase. The communication network and sociogram were frequently used together when visiting all phases.

Implications and Future Directions

TeamVision's integration in real-world educational settings reveals its potential impact on refining reflection practices in healthcare simulations. The findings underscore the promising role of AI-powered tools in education, stressing the importance of user-centered design to address adoption hurdles effectively. Moreover, the enhancement of reflective practices through TeamVision offers scalable insights into teamwork dynamics, promising to elevate the learning curve in medical education settings significantly.

Future work should expand on longitudinal studies to gauge the sustained impact of such AI-powered analytics tools on learning outcomes and practice integration. The adaptability of TeamVision holds potential beyond healthcare, suggesting its utility in diverse team-based educational contexts, necessitating further exploration into its cross-disciplinary applications.

Conclusion

TeamVision represents a significant advancement in the deployment of AI-driven analytics for educational enhancement. By facilitating structured, reflective learning through data-backed debriefs, TeamVision not only addresses current limitations in healthcare simulation debriefing but also pioneers an innovative approach for leveraging AI in professional education contexts. The research lays a foundation for future innovations aimed at integrating learning analytics more deeply into interactive and reflective educational settings.