Motion as Emotion: Detecting Affect and Cognitive Load from Free-Hand Gestures in VR (2409.12921v1)

Abstract: Affect and cognitive load influence many user behaviors. In this paper, we propose Motion as Emotion, a novel method that utilizes fine differences in hand motion to recognise affect and cognitive load in virtual reality (VR). We conducted a study with 22 participants who used common free-hand gesture interactions to carry out tasks of varying difficulty in VR environments. We find that the affect and cognitive load induced by tasks are associated with significant differences in gesture features such as speed, distance and hand tension. Standard support vector classification (SVC) models could accurately predict two levels (low, high) of valence, arousal and cognitive load from these features. Our results demonstrate the potential of Motion as Emotion as an accurate and reliable method of inferring user affect and cognitive load from free-hand gestures, without needing any additional wearable sensors or modifications to a standard VR headset.

• The paper demonstrates a novel method for detecting affect and cognitive load from free-hand gestures in VR tasks.
• Researchers found that challenging conditions reduce gesture speed and distance while increasing hand tension and limiting head movement.
• The study’s support vector classification achieved up to 91% accuracy in within-user tests, though it revealed generalization challenges across users.

Motion as Emotion: Detecting Affect and Cognitive Load from Free-Hand Gestures in VR

The paper "Motion as Emotion: Detecting Affect and Cognitive Load from Free-Hand Gestures in VR" by Chua et al. presents a novel approach to recognizing affective states and cognitive load using hand and head motion data collected during free-hand gesture interactions in virtual reality (VR). This study is situated at the intersection of affective computing and user input systems in immersive environments.

Experimental Setup and Tasks

The study was designed around four distinct VR tasks—Slingshot, Card Sequence Memorization, Button Sequence Memorization, and UI Navigation—each intended to induce varying levels of affect and cognitive load. Data were recorded from 22 participants using a Meta Quest Pro VR headset and Polar Verity Sense optical heart rate monitor. Each task incorporated typical VR interactions such as selection, swiping, and object manipulation via gestures tracked by the VR headset.

The affective and cognitive dimensions were measured using subjective self-reports (Affective Slider and NASA-TLX) and physiological data (HRV measures). The tasks' efficacy in eliciting the intended mental states was validated through significant differences in self-reported arousal, valence, and cognitive load between easy and challenging conditions. However, HRV metrics did not show significant differences between conditions, aligning with prior findings on the limitations of short-term HRV measures for detecting affect and cognitive states.

Gesture and Motion Analysis

The study leveraged the wrist-relative motion of the index finger tip as a primary source of gesture data. Additional features, such as hand tension and head movement, were extracted to capture the nuances of gesture formation. Significant findings included:

1. Gesture Distance and Speed: Participants displayed reduced gesture speed and distance under challenging conditions. This finding contradicts the intuitive expectation from Fitts' and Hick-Hyman Laws, suggesting that increased muscle stiffness under cognitive load leads to shorter gesture distances due to reduced extension and flexion of finger joints.
2. Hand Tension: Higher tension was observed in the challenging conditions, especially in gestures that involved pinching or pressing, indicating increased muscle stiffness.
3. Head Movement: Challenging conditions were associated with significantly reduced head movement across most tasks, possibly due to cognitive resource allocation strategies that minimize proprioceptive and vestibular processing during high cognitive load.

Classification Model and Performance

Support vector classification (SVC) models were employed to evaluate the predictive power of the extracted motion features. The models achieved high accuracy in within-user cross-validation (up to 91% for UI navigation task condition) but showed decreased performance in leave-one-user-out cross-validation. This indicates the models' limited ability to generalize across different users, emphasizing the potential need for personalized models or more sophisticated machine learning approaches.

Implications and Future Directions

The research presents several practical and theoretical implications. For practical applications, systems that utilize gesture-based inputs in VR can incorporate affect and cognitive load detection to create adaptive interfaces. For instance, productivity applications in augmented reality (AR) could dynamically adjust task difficulty or provide guided assistance based on real-time cognitive load measures. Similarly, social VR environments could use affect recognition to enhance digital communication by conveying nonverbal cues.

Theoretically, this study advances the understanding of the interplay between cognitive load, affective states, and motor actions in immersive environments. It suggests that theories like Fitts' Law and the Hick-Hyman Law may need reconsideration or adaptation when applied to free-hand gesture inputs in 3D space.

Future work should explore more refined feature extraction methods to capture the subtleties of gesture and motion data and improve the models' generalizability. Further, tasks should be designed to disentangle the effects of various dimensions of mental states, allowing for a more granular analysis. The exploration of other joints and hand regions can also provide a more comprehensive understanding of the gesture formation process under varying mental states.

In conclusion, "Motion as Emotion" demonstrates a promising approach to emotion and cognitive load recognition from free-hand gestures in VR, opening avenues for adaptive and responsive systems in immersive environments. This work sets the stage for future research to deepen the interplay between affective computing and human-computer interaction in XR technologies.