Modeling Face Emotion Perception from Naturalistic Face Viewing: Insights from Fixational Events and Gaze Strategies

Abstract: Face Emotion Recognition (FER) is essential for social interactions and understanding others' mental states. Utilizing eye tracking to investigate FER has yielded insights into cognitive processes. In this study, we utilized an instructionless paradigm to collect eye movement data from 21 participants, examining two FER processes: free viewing and grounded FER. We analyzed fixational, pupillary, and microsaccadic events from eye movements, establishing their correlation with emotion perception and performance in the grounded task. By identifying regions of interest on the face, we explored the impact of eye-gaze strategies on face processing, their connection to emotions, and performance in emotion perception. During free viewing, participants displayed specific attention patterns for various emotions. In grounded tasks, where emotions were interpreted based on words, we assessed performance and contextual understanding. Notably, gaze patterns during free viewing predicted success in grounded FER tasks, underscoring the significance of initial gaze behavior. We also employed features from pre-trained deep-learning models for face recognition to enhance the scalability and comparability of attention analysis during free viewing across different datasets and populations. This method facilitated the prediction and modeling of individual emotion perception performance from minimal observations. Our findings advance the understanding of the link between eye movements and emotion perception, with implications for psychology, human-computer interaction, and affective computing, and pave the way for developing precise emotion recognition systems.