HypomimiaCoach: An AU-based Digital Therapy System for Hypomimia Detection & Rehabilitation with Parkinson's Disease

Abstract: Hypomimia is a non-motor symptom of Parkinson's disease that manifests as delayed facial movements and expressions, along with challenges in articulation and emotion. Currently, subjective evaluation by neurologists is the primary method for hypomimia detection, and conventional rehabilitation approaches heavily rely on verbal prompts from rehabilitation physicians. There remains a deficiency in accessible, user-friendly and scientifically rigorous assistive tools for hypomimia treatments. To investigate this, we developed HypomimaCoach, an Action Unit (AU)-based digital therapy system for hypomimia detection and rehabilitation in Parkinson's disease. The HypomimaCoach system was designed to facilitate engagement through the incorporation of both relaxed and controlled rehabilitation exercises, while also stimulating initiative through the integration of digital therapies that incorporated traditional face training methods. We extract action unit(AU) features and their relationship for hypomimia detection. In order to facilitate rehabilitation, a series of training programmes have been devised based on the Action Units (AUs) and patients are provided with real-time feedback through an additional AU recognition model, which guides them through their training routines. A pilot study was conducted with seven participants in China, all of whom exhibited symptoms of Parkinson's disease hypomimia. The results of the pilot study demonstrated a positive impact on participants' self-efficacy, with favourable feedback received. Furthermore, physician evaluations validated the system's applicability in a therapeutic setting for patients with Parkinson's disease, as well as its potential value in clinical applications.

• The paper introduces a digital therapy system that uses facial action units (AUs) and AI to detect and rehabilitate hypomimia in Parkinson's patients.
• It employs advanced models like the Swin Transformer and Graph Convolutional Networks to extract and analyze facial features accurately.
• User studies indicate improved facial expressiveness and engagement, suggesting significant benefits for digital neurorehabilitation in Parkinson’s care.

The paper "HypomimiaCoach: An AU-based Digital Therapy System for Hypomimia Detection & Rehabilitation with Parkinson’s Disease" investigates a novel approach to address hypomimia, a non-motor symptom of Parkinson’s disease characterized by reduced facial expression. This condition can significantly affect communication and emotional expression in patients, thereby complicating social interactions and quality of life.

System Overview

The authors have developed HypomimiaCoach, a digital therapy system utilizing Action Units (AUs)—distinct facial muscle movements—to detect and rehabilitate hypomimia symptoms. The system is designed to provide a scientifically rigorous yet accessible solution for patients, integrating artificial intelligence technology with conventional therapeutic practices.

Detection and Rehabilitation Components

Detection Component:

1. Data Acquisition and Preprocessing: The system captures facial movement data via video recording. These recordings are processed using the MTCNN model, which aids in segmenting and normalizing facial frames for analysis.
2. Feature Extraction and Classification: Utilizes a Swin Transformer model for facial feature extraction, transforming facial expression data into actionable AU features. These features are further enhanced using Graph Convolutional Networks (GCNs) to understand interconnections among AUs, allowing the system to detect hypomimia accurately.

Rehabilitation Component:

1. Training Design: The system offers both basic and advanced training exercises. Basic training focuses on individual facial regions such as the eyebrows, eyes, lips, and articulation muscles, providing real-time feedback. Advanced training integrates these exercises with Chinese opera music to motivate and engage patients, emphasizing rhythm and cultural relevance.
2. Feedback Mechanism: Real-time analysis of facial movements gives users immediate feedback, categorized into levels such as "perfect" or "good." This interactive feedback loop is crucial for ensuring exercises are performed correctly, facilitating better rehabilitation outcomes.

User Study

To evaluate the system’s effectiveness, a user study was conducted with Parkinson’s patients and healthcare professionals. The study aimed to assess user engagement, system usability, and the potential impact on patients’ rehabilitation outcomes. Participants generally found the digital therapy format engaging, with feedback indicating improved facial expressiveness and emotional relief post-training.

Methodological Insights

The system leverages self-determination theory, aiming to increase patient autonomy, competence, and engagement through digital interventions. HypomimiaCoach addresses challenges inherent in traditional therapies, such as the need for extensive in-person interaction and subjective assessments by clinical staff.

Design Implications

The paper suggests several design improvements based on user feedback, including:

• Simplifying complex exercises and offering clearer guidance, especially for patients with cognitive impairments.
• Further enhancing patient autonomy through personalized music choices and adaptable training plans.
• Incorporating a range of music and visual filters to better accommodate diverse user preferences.

Conclusions and Future Directions

HypomimiaCoach represents an innovative step towards integrating digital therapies with traditional Parkinson’s treatments. Its use of AI could facilitate early diagnosis and intervention, particularly in resource-limited settings. Future work could expand upon the dataset to improve model accuracy and explore additional therapeutic areas, potentially improving the quality of life for Parkinson’s patients globally.