Rethinking Eye-blink: Assessing Task Difficulty through Physiological Representation of Spontaneous Blinking

Abstract: Continuous assessment of task difficulty and mental workload is essential in improving the usability and accessibility of interactive systems. Eye tracking data has often been investigated to achieve this ability, with reports on the limited role of standard blink metrics. Here, we propose a new approach to the analysis of eye-blink responses for automated estimation of task difficulty. The core module is a time-frequency representation of eye-blink, which aims to capture the richness of information reflected on blinking. In our first study, we show that this method significantly improves the sensitivity to task difficulty. We then demonstrate how to form a framework where the represented patterns are analyzed with multi-dimensional Long Short-Term Memory recurrent neural networks for their non-linear mapping onto difficulty-related parameters. This framework outperformed other methods that used hand-engineered features. This approach works with any built-in camera, without requiring specialized devices. We conclude by discussing how Rethinking Eye-blink can benefit real-world applications.

• The paper presents a novel blink analysis framework that converts blink timeseries data into spectrograms for assessing mental workload.
• It employs a Lomb-Scargle periodogram and a 2D LSTM network to capture dynamic blink patterns, achieving over 70% classification accuracy.
• Findings suggest practical applications in usability and accessibility, enabling real-time, contact-free cognitive state monitoring.

Assessing Task Difficulty through Spontaneous Blinking

Introduction

This paper introduces the "Rethinking Eye-blink" framework, a novel approach for assessing mental workload and task difficulty via the physiological representation of spontaneous blinking (2102.06690). Unlike conventional eye-tracking methods which predominantly focus on metrics such as Blink Rate (BR) and Blink Duration (BD), which have shown limited sensitivity to psychological states, this research explores a more complex time-frequency representation to extract richer, more informative patterns from spontaneous blinking. The framework employs a standard RGB camera to extract blink timeseries data, which is then converted into spectrograms for feature learning, ultimately feeding into a Long Short-Term Memory (LSTM) network for real-time task difficulty assessment.

Methodology and Implementation

Framework Overview: The proposed system operates through three main stages: automatic blink timeseries extraction, transformation into a time-frequency spectrogram, and the application of 2D LSTM networks for feature learning. The use of a built-in webcam for data collection emphasizes the framework's contact-free nature, enabling broad applicability without specialized hardware.

Automatic Blink Detection: Blink detection is achieved by analyzing the eye aspect ratio changes over time using facial landmarks from RGB images, while applying filtering to minimize baseline drift. This pre-processed data forms the basis of further spectrogram analysis.

Time-Frequency Representation: Inspired by respiratory variability analysis, the blink data is transformed into a spectrogram using the Lomb-Scargle periodogram, which is designed to handle irregularly sampled biological signals, enhancing robustness in noisy environments.

Feature Learning via 2D LSTM: The spectrograms feed into a multi-dimensional LSTM model trained for non-linear mapping of blink patterns to difficulty levels. LSTM networks are chosen for their ability to model temporal dynamics more effectively than traditional CNNs, thereby offering improved performance in classifying mental workload states.

Experimental Evaluation

The experimental work comprised two studies.

Study I: Focused on testing the sensitivity of the new blink metrics, specifically Blink Entropy (BE), in relation to task difficulty. Using a controlled experimental setup with a mathematical subtraction task at varying difficulty levels, the research demonstrated that BE significantly outperformed traditional BR and BD measures in correlational analyses with task difficulty indicators.

Study II: This study further validated the Rethinking Eye-blink framework's ability to classify task difficulty. Using 18-fold leave-one-subject-out cross-validation, the system achieved a mean accuracy exceeding 70% across multiple labeling strategies, demonstrating substantial improvements over baseline methods involving hand-engineered features and simpler models.

Applications and Implications

The findings suggest practical implications for usability and accessibility evaluations, allowing real-time and unobtrusive assessment in various interactive environments such as VR and mobile applications. Potential applications include augmenting educational settings by providing insights into collective learning difficulties during live sessions, enhancing the instructional design to better cater to learners' needs.

Moreover, in compliance-driven environments such as web accessibility assessments, this framework can automate evaluations, identifying barriers that may not be apparent through traditional assessment methods.

Discussion

The research underscores the importance of advanced feature representations in physiological computing, demonstrating how complex, dynamic patterns captured in time-frequency domains can significantly enhance the sensitivity of physiological metrics to psychological states. While traditional eye-blink metrics have been dismissed for their sensitivity limitations, this study showcases a paradigm shift by representing blinks more dynamically and integrating sophisticated ML models for superior inference capabilities.

Conclusion

The Rethinking Eye-blink framework marks a substantial step forward in leveraging physiological signals for cognitive state assessment, particularly in HCI contexts. The research illustrates that integrating time-frequency analysis with LSTM networks can revitalize traditional metrics, transforming spontaneous blinking into a powerful tool for continuous mental workload monitoring, with extensive potential for future AI developments and applications.