Neural Tracking of Sustained Attention, Attention Switching, and Natural Conversation in Audiovisual Environments using Mobile EEG

Abstract: Everyday communication is dynamic and multisensory, often involving shifting attention, overlapping speech and visual cues. Yet, most neural attention tracking studies are still limited to highly controlled lab settings, using clean, often audio-only stimuli and requiring sustained attention to a single talker. This work addresses that gap by introducing a novel dataset from 24 normal-hearing participants. We used a mobile electroencephalography (EEG) system (44 scalp electrodes and 20 cEEGrid electrodes) in an audiovisual (AV) paradigm with three conditions: sustained attention to a single talker in a two-talker environment, attention switching between two talkers, and unscripted two-talker conversations with a competing single talker. Analysis included temporal response functions (TRFs) modeling, optimal lag analysis, selective attention classification with decision windows ranging from 1.1s to 35s, and comparisons of TRFs for attention to AV conversations versus side audio-only talkers. Key findings show significant differences in the attention-related P2-peak between attended and ignored speech across conditions for scalp EEG. No significant change in performance between switching and sustained attention suggests robustness for attention switches. Optimal lag analysis revealed narrower peak for conversation compared to single-talker AV stimuli, reflecting the additional complexity of multi-talker processing. Classification of selective attention was consistently above chance (55-70% accuracy) for scalp EEG, while cEEGrid data yielded lower correlations, highlighting the need for further methodological improvements. These results demonstrate that mobile EEG can reliably track selective attention in dynamic, multisensory listening scenarios and provide guidance for designing future AV paradigms and real-world attention tracking applications.

• The paper demonstrates that mobile EEG effectively tracks sustained, switching, and conversational attention in complex audiovisual settings.
• Using full-cap and cEEGrid arrays alongside TRF analysis, the study reveals distinct neural markers, such as the P2 component, that differentiate attention conditions.
• The results show robust model generalization across tasks while highlighting limitations of cEEGrid performance for real-world auditory attention decoding.

Neural Tracking of Sustained, Switching, and Conversational Attention in Audiovisual Environments Using Mobile EEG

Introduction

The paper "Neural Tracking of Sustained Attention, Attention Switching, and Natural Conversation in Audiovisual Environments using Mobile EEG" (2601.15097) addresses key challenges in real-world auditory attention decoding (AAD) by leveraging mobile EEG in complex, multisensory conditions. Existing AAD approaches largely remain constrained to audio-only, highly controlled laboratory environments that lack ecological validity. This work introduces a mobile EEG dataset with 24 normal-hearing Danish speakers exposed to sustained attention, attention switching, and natural conversation scenarios within audiovisual (AV) paradigms. Both full-cap scalp EEG and cEEGrid around-the-ear arrays were employed, providing critical evidence regarding system robustness and modality-specific limitations.

Experimental Paradigm and Methods

The experimental design incorporated three ecologically valid conditions: sustained attention (SustAC), switching attention (SwitAC), and conversation attention (ConvAC). Stimuli were delivered via calibrated loudspeakers at fixed azimuths, with synchronized AV content ensuring naturalistic sensory environments. Critically, conditions required participants to maintain, switch, or flexibly allocate attention between real AV talkers, simulating real-world conversational listening.

Behavioral performance (question accuracy, self-rated listening and understanding difficulty) was tracked, and neural correlates of attention were assessed using temporal response functions (TRFs), forward/backward modeling, correlation-based metrics, and selective attention classification at multiple temporal resolutions.

Figure 1: Experimental EEG setup, preprocessing workflows for both stimuli and neural data, and subsequent TRF-based analysis pipelines.

The EEG setup combined 44 scalp electrodes and dual 10-channel cEEGrid arrays. Data preprocessing included ICA-based artifact rejection, referencing, bandpass filtering, and normalization. Speech features comprised acoustic envelopes and acoustic onsets, extracted through Gammatone filterbanks and edge detection, band-limited, and aligned temporally with EEG. TRFs were estimated using boosting regression and evaluated via cross-validation, adopting both forward (stimulus-to-EEG) and backward (EEG-to-stimulus) approaches.

Behavioral Results

Behavioral accuracy confirmed task engagement (response accuracy: 85–89% across conditions). Both listening and understanding difficulty ratings escalated in dynamic (SwitAC, ConvAC) conditions relative to SustAC, reflecting higher cognitive load with increased task complexity. These self-reported measures validate the ecological difficulty inherent in the experimental design.

Figure 2: Behavioral performance and self-rated listening/understanding difficulties for SustAC, SwitAC, and ConvAC.

Neural Attention Tracking: TRF and Correlation Analyses

TRF Morphology and Statistical Differentiation

TRF analyses revealed marked differences between attended and ignored speech across all conditions, with the most prominent modulations detected at the P2 (envelope-based, ~180–200 ms) and corresponding N1/P1 peaks (onset-based), classically implicated in selective attention and target discrimination.

Figure 3: Grand-average TRFs for acoustic envelope and onset features, for attended and ignored streams under all conditions, showing channel-averaged and spatial topographies.

Statistical cluster analysis confirmed significant attended-vs-ignored differences at these latencies (notably at P2 and N400), with broader, less distinct TRF morphology observed during conversational attention relative to controlled tasks. Peak latencies for SwitAC and ConvAC consistently lagged SustAC by ~20 ms, indicating that dynamic allocation and higher scene complexity delay the emergence of neural attention signatures.

Figure 4: Cluster-based permutation statistics isolating significant differences between attended and ignored TRFs for each condition.

Decoding Performance and Optimal Lag Structure

Backward TRF modeling produced significantly higher stimulus-reconstruction correlations for attended than ignored speech using scalp EEG (typically $p < 0.01$ for SustAC and SwitAC; ConvAC sometimes showed increased ignored correlations due to informational masking and scene complexity). cEEGrid arrays yielded lower, less stable correlations, often not exceeding statistical significance except for select features and conditions.

Figure 5: Distribution of attended, ignored, and control speech correlations across conditions/features for both electrode types.

Optimal lag analysis demonstrated that maximum attention effects clustered around 100–250 ms post-stimulus for attended speech. The shape and precision of these peaks varied with task complexity: conversational attention produced broader, less sharply tuned optimal lags, reflecting increased computational demand and possibly simultaneous multisource encoding.

Figure 6: Lag-resolved correlation maps showing peak separations between attended and ignored conditions for each paradigm and electrode modality.

Attention classification accuracy with scalp EEG exceeded 80% for full-trial windows; even for window sizes down to ~1–2 s, accuracy remained above chance (55–70%) in all conditions. The SwitAC and ConvAC tasks demonstrated comparable performance to SustAC, highlighting model robustness to dynamic attention and natural conversation. cEEGrid classification dropped below 60% for all but the longest windows, evidencing current limitations for unobtrusive mobile AAD deployment.

Figure 7: Attention decoding accuracy as a function of classification window size and condition, for both scalp and cEEGrid modalities.

Special Considerations: Conversation Trials and Model Generalization

Analysis of TRFs for conversation-focused trials (attending to AV dyad versus side talker) showed that conversation-evoked TRFs exhibited extended, less focal lag structure compared to single-talker trials. This temporal broadening and reduction in sparsity are likely a function of temporally overlapping, dynamically competing sources and increased visual context in conversational AV scenes.

Figure 8: Differential TRF structure for trials with attention on conversation versus side single-talker, highlighting specific electrode locations of interest.

Backward models trained on sustained attention data generalized well to both attention-switching and conversational trials, demonstrating a considerable degree of invariant neural encoding across different ecologically relevant tasks (SwitAC: mean accuracy 0.70, ConvAC: 0.66 using models trained in SustAC). This cross-condition generalizability supports unified modeling approaches for practical real-world AAD systems.

Practical and Theoretical Implications

The findings establish that mobile EEG using standard cap and (to a lesser extent) around-the-ear arrays, combined with linear TRF-based AAD, is robust to both sustained and dynamically shifting attention in AV multispeaker environments. The attention-related P2 difference is a reliable neural marker even in unconstrained conversational contexts. However, cEEGrid performance trails scalp EEG, requiring further method refinement, including spatial filtering, referencing, and subject-specific training.

The generalized performance of linear modeling across tasks and visual contexts indicates that bottom-up auditory attention signatures are preserved across diverse real-world conditions. There is also convergence with reports that transient features (acoustic onsets) yield sharper, sparser neural tracking than continuous amplitude cues (envelope), especially under informational masking.

Methodologically, extensive controls—including cross-validation avoiding within-trial dependence—address concerns about bias from eye-gaze or idiosyncratic stimulus events, highlighted by recent bias-detection studies.

Theoretically, the temporal expansion and latency shift of attention-related peaks in AV conversation suggest an increased neural integration window and resource allocation during complex multisensory scene analysis. This finding provides constraints for computational models of attention and may inform neuro-engineered steering of next-generation brain-controlled hearing assistive devices.

Conclusion

This work provides comprehensive evidence that mobile EEG—especially full-cap but also, with caveats, cEEGrid—enables reliable neural decoding of attention in highly dynamic, multi-talker, multimodal environments. Attention-related neural markers, notably the P2 component, persist across conditions requiring rapid attention shifts and during natural AV conversation. Models are robust to task and context, supporting future translation to practical assistive technologies for hearing attention and neuro-adaptive AV scene analysis.

Further research is necessary to:

• Optimize cEEGrid and other unobtrusive methods for parity with full-cap systems
• Refine attention decoding approaches for split/flexible attention and more complex naturalistic scenarios
• Integrate neural tracking with real-time closed-loop auditory scene manipulation for hearing augmentation

These results represent a substantive advance toward ecologically valid, portable neuro-steered communication technology.