Sp-EEG-Deformer: Dual-Modal Auditory Decoding

• The paper demonstrates that fusing EEG signals with beamformed audio spatial spectrum via a transformer encoder significantly boosts multi-class attended speaker direction decoding.
• The architecture employs tokenization, self-attention, and a fusion block, achieving up to 61.8% accuracy in 15-class paradigms compared to CNN-based benchmarks.
• This dual-modal approach highlights the potential for neuro-steered beamforming in realistic auditory environments, paving the way for advanced hearing-assistive technologies.

Sp-EEG-Deformer refers to a dual-modal neural decoding architecture that integrates electroencephalogram (EEG) time series and audio spatial spectrum for multi-class attended speaker direction decoding in auditory scenes with competing sources. This approach is motivated by the limitations of EEG-only attention decoding, especially in fine-grained (e.g., 14- or 15-way) directional tasks, and leverages spatial cues derived from beamformed audio to substantially improve classification accuracy over previous baselines. The technical foundation is a transformer-style (“Deformer”) encoder for EEG featuring token-level attention and a fusion block for embedding audio spatial information at the intermediate feature level. Sp-EEG-Deformer achieves statistically significant improvements in attended direction decoding, with reported 14-class leave-one-subject-out accuracy of 55.4% and up to 61.8% in 15-class leave-one-trial-out paradigms with 10-second EEG windows, outperforming both CNN- and LSM-CNN-based architectures (Zhang et al., 2024).

1. Decoding Objective and Dual-Modal Motivation

Sp-EEG-Deformer addresses the multi-class attended speaker direction decoding problem: given a short EEG segment recorded while a subject attends to one of multiple spatially separated concurrent speakers, the system predicts the exact attended azimuth, discretized as 15 candidate angles (±135°, ±120°, ±90°, ±60°, ±45°, ±30°, ±15°, 0°). Traditional EEG-only models yield low accuracy (6–10% in 15-way tests), attributed to limited spatial encoding in scalp EEG. Microphone arrays, conversely, isolate spatial cues but cannot identify which speaker is attended without neural input. By synchronizing each EEG window with a spatial spectrum reflecting beamformed audio energy at each candidate direction, Sp-EEG-Deformer grounding attention inference in both neurophysiological and environmental signals produces a robust gain in decoding performance (Zhang et al., 2024).

2. Audio Spatial Spectrum Feature: Computation and Synchronization

The spatial spectrum feature is computed via a Minimum-Variance Distortionless Response (MVDR) beamformer applied to two-channel audio from in-ear microphones. For each trial $j$ with STFTs $Y_i(f, n)$, the spatial covariance at frequency $f$ is

$$\mathbf{R}(f) = \frac{1}{N} \sum_{n=0}^{N-1} \mathbf{y}(f, n) \mathbf{y}(f, n)^H,$$

where $\mathbf{y}(f, n) = [Y_1(f, n); Y_2(f, n)]$. The MVDR spatial spectrum at direction $\theta$ is

$$P(f, \theta) = \frac{1}{\mathbf{g}(f, \theta)^H \mathbf{R}(f)^{-1} \mathbf{g}(f, \theta)},$$

with $\mathbf{g}(f, \theta)$ the array steering vector. The final spatial spectrum for trial $j$ and attended direction $\theta_j$ is the frequency average: $Y_i(f, n)$0 All EEG decision windows within the trial inherit $Y_i(f, n)$1. The spectrum vector is normalized (zero mean, unit variance) over training data for compatibility with the neural encoder (Zhang et al., 2024).

3. Sp-EEG-Deformer Architecture

The Sp-EEG-Deformer consists of the following key architectural blocks:

• EEG Preprocessing: EEG is downsampled to 128 Hz and band-pass filtered (1–32 Hz), with ICA for artifact removal and interpolation for bad channels.
• Tokenization: Each EEG window ($Y_i(f, n)$2; e.g. $Y_i(f, n)$3 for $Y_i(f, n)$4s) is transformed into $Y_i(f, n)$5 tokens by a 1D convolutional embedding ($Y_i(f, n)$6 tokens for $Y_i(f, n)$7s), each with $Y_i(f, n)$8 features. Learnable positional encodings are added.
• Encoder Layers: $Y_i(f, n)$9 repeated transformer encoder blocks, each with pre-layer normalization, multi-head self-attention (MHSA), and a two-layer feed-forward network (FFN) with GELU nonlinearity.
• Fusion Block: After $f$0 (typically 1) encoder blocks, the spatial spectrum vector $f$1 is mapped to the token space via a fully connected layer, concatenated to EEG token features, then projected back to $f$2. This forms a dual-modal representation.
• Deep Encoding and Classification: The fused stream passes through the remaining encoder blocks, followed by global average pooling and a final classification head yielding 15 logits for softmax output.

Architectural Parameterization

With $f$3s: input EEG $f$4 transforms into $f$5 tokens; spatial spectrum $f$6 bins. The fusion block shape is $f$7 before and after fusion. The final feature before the classification head is $f$8 (Zhang et al., 2024).

4. Training Protocol and Loss

Training uses standard 15-way cross-entropy loss: $f$9 with Adam optimizer (β₁=0.9, β₂=0.999), initial learning rate $$\mathbf{R}(f) = \frac{1}{N} \sum_{n=0}^{N-1} \mathbf{y}(f, n) \mathbf{y}(f, n)^H,$$0, weight decay $$\mathbf{R}(f) = \frac{1}{N} \sum_{n=0}^{N-1} \mathbf{y}(f, n) \mathbf{y}(f, n)^H,$$1, and learning rate reduction on plateau. Early stopping is employed on validation performance, batch size 64, and maximum epochs up to 100. Dropout (0.1) is used within MHSA and FFN submodules (Zhang et al., 2024).

Cross-validation protocols include leave-one-trial-out (LOTO), leave-one-subject-out (LOSO), and strict variants (excluding adjacent/same-class/same-audio trials) for robust generalization assessment.

5. Experimental Results and Comparative Analysis

The efficacy of Sp-EEG-Deformer is demonstrated via direct comparison with CNN and LSM-CNN architectures, with and without audio spatial spectrum (denoted “Sp-Aux”). In 15-way decoding (chance ≈6.7%), key scores are:

Model	LOSO (1 s)	LOSO (10 s)	LOTO (1 s)	LOTO (10 s)
Sp-Aux-CNN	49.9±7.1	39.5±10.9	56.3±2.5	61.4±1.4
Sp-Aux-LSM-CNN	55.4±2.3	58.4±3.6	56.1±2.3	61.4±1.4
Sp-Aux-EEG-Deformer (proposed)	57.5±2.1	56.3±1.0	60.0±1.7	61.8±2.5

Reducing the number of alternative classes (2, 4, 8, 15) increases accuracy, but even in 4- or 8-way tasks, EEG-only accuracy drops to 20–30% while the fused Sp-Aux Deformer remains above 60% (Zhang et al., 2024).

Ablation studies confirm both the necessity and the robustness of multi-modal fusion: models degrade in stringent cross-validation but remain significantly above chance, suggesting some residual reliance on correlated audio and class content but demonstrating substantial information encoding in dual-modal cues.

6. Architectural and Empirical Insights

Self-attention-based Deformer encoders outperform CNNs and LSM-CNNs for attended direction decoding due to their capacity for long-range temporal and channelwise dependency modeling and flexible cross-token interaction. The intermediate fusion block grounds noisy neural attention traces in the structured spatial peaks of the audio spectrum. Without the spatial spectrum, deep models collapse to near-random performance in fine-grained directional decoding. The persistence of high accuracy even under strict cross-validation highlights the extractor’s robustness and the utility of synchronizing environmental and endogenous signals for neural decoding.

A plausible implication is that multi-class auditory attention can be decoded in realistic listening scenarios with moderate window lengths (1–10 s), enabling brain-driven spatial filtering for advanced hearing-assistive technologies (Zhang et al., 2024).

7. Significance and Potential Applications

Sp-EEG-Deformer establishes a reference paradigm for multi-modal, high-class-count neural decoding in complex listening environments. Its methodology and demonstrated accuracy provide a concrete foundation for future research in neuro-steered beamforming and brain-computer interfaces targeting hearing restoration, where precise directional attention decoding is critical. The architecture and findings reinforce the principle that combined use of neurophysiological and environmental features surpasses the limits of unimodal approaches in neural interface engineering (Zhang et al., 2024).