EffortNet: EEG Listening Effort Assessment

• EffortNet is a deep learning framework for EEG-based listening effort assessment that decodes individual alpha oscillations during speech comprehension.
• It integrates self-supervised, incremental, and transfer learning to robustly adapt to subject variability with minimal calibration.
• The model achieves 80.9% classification accuracy under varying conditions, outperforming traditional CNN-based approaches in noisy environments.

EffortNet is a deep learning framework designed for objective assessment of listening effort in speech comprehension by decoding individual EEG alpha oscillation patterns. It directly addresses the measurement of cognitive load during auditory processing in both typical and impaired populations, offering a probability-based metric for comparing speech enhancement technologies under ecologically valid conditions.

1. Architectural Overview and Learning Paradigms

EffortNet comprises a convolutional neural network (CNN) configured to process multichannel EEG time-series data. Its architecture integrates a three-phase training procedure which combines self-supervised learning (SSL), incremental learning (IL), and transfer learning (fine-tuning):

• Phase 1: Self-Supervised Learning (SSL) The framework utilizes large-scale unlabeled EEG data to pretrain an encoder-decoder pair on a mask-based reconstruction objective. The SSL loss function is the mean squared error between the original and reconstructed signals:

$$L_{SSL}(\theta, \phi) = L( g_{\phi}(f_{\theta}(\tilde{X})), X )$$

The encoder ($f_\theta$) and decoder ($g_\phi$) are optimized to learn robust, input-invariant representations:

$$(\theta^*, \phi^*) = \arg\min_{\theta, \phi} L_{SSL}(\theta, \phi)$$

• Phase 2: Incremental Learning (IL) IL enables adaptation to subject-to-subject variability in EEG responses. Training is partitioned per subject, with “replay” mechanisms to retain previously acquired subject-specific knowledge and mitigate catastrophic forgetting. Loss is defined as:

$$L_{IL}^t(\theta, \psi) = L_{current}(h_\psi(f_\theta(X^t)), Y^t) + \lambda \cdot L_{replay}(h_\psi(f_\theta(X')), Y')$$

with $\lambda = 1$.

$$(\theta^{t}, \psi^{t}) = \arg\min_{\theta, \psi} L_{IL}^t(\theta, \psi)$$

• Phase 3: Fine-Tuning (Transfer Learning) The pretrained and incrementally learned model is further fine-tuned on a small set of labeled target data, using a supervised cross-entropy objective for adaptation to new individuals:

$$L_{target}(\theta, \psi) = L( h_\psi(f_\theta(X_{tgt})), Y_{tgt} )$$

Final parameters are:

$$(\theta^{(F-T)}, \psi^{(F-T)}) = \arg\min_{\theta, \psi} L_{target}(\theta, \psi)$$

The CNN contains four convolutional layers tailored for EEG spatial-temporal pattern extraction (post bandpass filtering and wavelet transform in the 8–13 Hz alpha band), reshaping, dense layers, and sigmoid output for binary classification (low vs. high listening effort).

2. EEG Data Acquisition and Preprocessing

EffortNet is trained and evaluated on EEG data from 122 participants recorded via a 64-channel QuickCap system during speech comprehension tasks under four acoustic conditions: clean, noisy, MMSE-enhanced, and Transformer-enhanced speech. Data preprocessing involves artifact removal, re-referencing, bandpass filtering, and discrete wavelet transform to extract time–frequency representations specific to the alpha band (8–13 Hz).

Statistical analysis demonstrates that alpha power is significantly elevated in noisy speech scenarios, validating alpha-band oscillations’ role as objective biomarkers of increased cognitive effort. This neurophysiological response is interpreted as reflecting mechanisms for suppressing irrelevant stimuli and managing increased auditory cognitive load.

3. Strategies to Overcome Inter-Individual Variability

EffortNet is specifically designed to address the substantial variability in EEG responses across individuals—a major obstacle in developing scalable neural metrics for cognitive states. The framework’s integration of SSL and IL enables:

• Learning robust features from unlabeled data, generalizable across subjects without full annotation dependency.
• Progressive assimilation of subject-specific EEG characteristics using replay-based IL, which maintains performance across a growing population without catastrophic forgetting.
• Rapid individualization via transfer learning, requiring only a small fraction of labeled data for target participants. Collectively, these paradigms facilitate accurate generalization while minimizing experimental calibration and annotation overhead for new users.

4. Comparative Evaluation and Performance Metrics

EffortNet achieves an average EEG-based listening effort classification accuracy of 80.9% with only 40% training data from new subjects, surpassing baseline CNN architectures (62.3%) and STAnet models (61.1%). This performance is robust even as training data is reduced, with ablation studies confirming that all three training phases (SSL, IL, fine-tuning) are essential for optimal cross-subject generalization.

The model employs a probability output metric for “low listening effort” (LLE) classification. In evaluation:

Speech Condition	Probability of LLE (%)
Clean	80
Transformer-Enhanced	62.3
MMSE-Enhanced	40
Noisy	15.7

Notably, the probability-based neural metric aligns with objective measures such as STOI and PESQ, but diverges from subjective intelligibility ratings—highlighting a discrepancy between conscious perception and underlying neural effort biomarkers.

5. Assessment of Speech Enhancement Technologies

EffortNet enables quantitative comparison of speech enhancement algorithms through EEG-derived listening effort metrics. Transformer-based enhancement produces neural responses more similar to clean speech than classical MMSE enhancement, as revealed by higher LLE probabilities. In contrast, subjective ratings do not consistently reflect these neural differences, suggesting that objective EEG metrics may access aspects of cognitive processing not captured by behavioral reporting.

A plausible implication is that neural-based metrics could address limitations of subjective assessments in the evaluation of cognitive load imposed by speech enhancement systems and auditory prostheses, with relevance for both research and clinical translation.

6. Practical Applications and Significance

EffortNet facilitates personalized, objective assessment of listening effort for clinical and consumer hearing technologies. The architecture’s capability to adapt quickly to individual EEG profiles with minimal calibration data supports large-scale deployment in clinics, research, and device evaluation settings. Applications include:

• Quantitative benchmarking of hearing aids and enhancement algorithms in diverse listener populations.
• Cognitive-aware optimization of speech processing technologies for aging and hearing-impaired users.
• Potential reduction of long-term cognitive load impacts associated with sustained listening effort. This flexible, probability-based neural metric offers a complement or alternative to standard behavioral and intelligibility ratings, potentially advancing the evaluation and design of cognitive-adaptive auditory technologies.

7. Relationship to Prior Work and Broader Context

EffortNet builds on neural approaches for modeling cognitive state from electrophysiological signals in challenging listening environments (Sung et al., 21 Aug 2025). The framework’s methodological rigor—multi-phase learning, detailed EEG preprocessing, and comparative metric analysis—differentiates it from prior single-paradigm CNN methods. Its focus on alpha-band oscillatory biomarkers is consistent with established findings in auditory neuroscience, while the integration of SSL, IL, and transfer learning reflects current best practices in bio-signal deep learning frameworks. As objective EEG-based effort metrics gain clinical acceptance, EffortNet’s design and performance characteristics provide a benchmark for scalable, individualized assessment in cognitive and auditory interfacing.