---
title: 'Sp-EEG-Deformer: Dual-Modal Auditory Decoding'
url: https://www.emergentmind.com/topics/sp-eeg-deformer
type: topic
---

# Sp-EEG-Deformer: Dual-Modal Auditory Decoding

Sp-EEG-Deformer refers to a dual-modal neural decoding architecture that integrates electroencephalogram (EEG) time series and audio spatial spectrum for multi-class attended speaker direction decoding in auditory scenes with competing sources. This approach is motivated by the limitations of EEG-only attention decoding, especially in fine-grained (e.g., 14- or 15-way) directional tasks, and leverages spatial cues derived from beamformed audio to substantially improve classification accuracy over previous baselines. The technical foundation is a transformer-style (“Deformer”) encoder for EEG featuring token-level attention and a fusion block for embedding audio spatial information at the intermediate feature level. Sp-EEG-Deformer achieves statistically significant improvements in attended direction decoding, with reported 14-class leave-one-subject-out accuracy of 55.4% and up to 61.8% in 15-class leave-one-trial-out paradigms with 10-second EEG windows, outperforming both CNN- and LSM-CNN-based architectures [2411.06928].

## 1. Decoding Objective and Dual-Modal Motivation

Sp-EEG-Deformer addresses the multi-class attended speaker direction decoding problem: given a short EEG segment recorded while a subject attends to one of multiple spatially separated concurrent speakers, the system predicts the exact attended azimuth, discretized as 15 candidate angles (±135°, ±120°, ±90°, ±60°, ±45°, ±30°, ±15°, 0°). Traditional EEG-only models yield low accuracy (6–10% in 15-way tests), attributed to limited spatial encoding in scalp EEG. Microphone arrays, conversely, isolate spatial cues but cannot identify which speaker is attended without neural input. By synchronizing each EEG window with a spatial spectrum reflecting beamformed audio energy at each candidate direction, Sp-EEG-Deformer grounding attention inference in both neurophysiological and environmental signals produces a robust gain in decoding performance [2411.06928].

## 2. Audio Spatial Spectrum Feature: Computation and Synchronization

The spatial spectrum feature is computed via a Minimum-Variance Distortionless Response (MVDR) beamformer applied to two-channel audio from in-ear microphones. For each trial $j$ with STFTs $Y_i(f, n)$, the spatial covariance at frequency $f$ is
\[
\mathbf{R}(f) = \frac{1}{N} \sum_{n=0}^{N-1} \mathbf{y}(f, n) \mathbf{y}(f, n)^H,
\]
where $\mathbf{y}(f, n) = [Y_1(f, n); Y_2(f, n)]$. The MVDR spatial spectrum at direction $\theta$ is
\[
P(f, \theta) = \frac{1}{\mathbf{g}(f, \theta)^H \mathbf{R}(f)^{-1} \mathbf{g}(f, \theta)},
\]
with $\mathbf{g}(f, \theta)$ the array steering vector. The final spatial spectrum for trial $j$ and attended direction $\theta_j$ is the frequency average:
\[
\mathbf{p}_j = \frac{1}{F} \sum_{f=1}^F P(f, \theta_j) \in \mathbb{R}^{F}.
\]
All EEG decision windows within the trial inherit $\mathbf{p}_j$. The spectrum vector is normalized (zero mean, unit variance) over training data for compatibility with the neural encoder [2411.06928].

## 3. Sp-EEG-Deformer Architecture

The Sp-EEG-Deformer consists of the following key architectural blocks:

- **EEG Preprocessing:** EEG is downsampled to 128 Hz and band-pass filtered (1–32 Hz), with ICA for artifact removal and interpolation for bad channels.
- **Tokenization:** Each EEG window ($C \times L$; e.g. $30 \times 128$ for $T=1\,$s) is transformed into $M$ tokens by a 1D convolutional embedding ($M=15$ tokens for $T=1\,$s), each with $d_e=64$ features. Learnable positional encodings are added.
- **Encoder Layers:** $N_e$ repeated transformer encoder blocks, each with pre-layer normalization, multi-head self-attention (MHSA), and a two-layer feed-forward network (FFN) with GELU nonlinearity.
- **Fusion Block:** After $L_f$ (typically 1) encoder blocks, the spatial spectrum vector $\mathbf{p}$ is mapped to the token space via a fully connected layer, concatenated to EEG token features, then projected back to $d_e$. This forms a dual-modal representation.
- **Deep Encoding and Classification:** The fused stream passes through the remaining encoder blocks, followed by global average pooling and a final classification head yielding 15 logits for softmax output.

#### Architectural Parameterization

With $T=1\,$s: input EEG $(30 \times 128)$ transforms into $M=15$ tokens; spatial spectrum $F \approx 256$ bins. The fusion block shape is $(15 \times 64)$ before and after fusion. The final feature before the classification head is $\mathbb{R}^{64}$ [2411.06928].

## 4. Training Protocol and Loss

Training uses standard 15-way cross-entropy loss:
\[
\mathcal{L} = -\sum_{c=1}^{15} y^{\text{true}}_c \log \hat{p}_c,
\]
with Adam optimizer (β₁=0.9, β₂=0.999), initial learning rate $1 \times 10^{-3}$, weight decay $1 \times 10^{-4}$, and learning rate reduction on plateau. Early stopping is employed on validation performance, batch size 64, and maximum epochs up to 100. Dropout (0.1) is used within MHSA and FFN submodules [2411.06928].

Cross-validation protocols include leave-one-trial-out (LOTO), leave-one-subject-out (LOSO), and strict variants (excluding adjacent/same-class/same-audio trials) for robust generalization assessment.

## 5. Experimental Results and Comparative Analysis

The efficacy of Sp-EEG-Deformer is demonstrated via direct comparison with CNN and LSM-CNN architectures, with and without audio spatial spectrum (denoted “Sp-Aux”). In 15-way decoding (chance ≈6.7%), key scores are:

| Model                          | LOSO (1 s) | LOSO (10 s) | LOTO (1 s) | LOTO (10 s) |
|---------------------------------|------------|-------------|------------|-------------|
| Sp-Aux-CNN                     | 49.9±7.1   | 39.5±10.9   | 56.3±2.5   | 61.4±1.4    |
| Sp-Aux-LSM-CNN                 | 55.4±2.3   | 58.4±3.6    | 56.1±2.3   | 61.4±1.4    |
| Sp-Aux-EEG-Deformer (proposed) | 57.5±2.1   | 56.3±1.0    | 60.0±1.7   | 61.8±2.5    |

Reducing the number of alternative classes (2, 4, 8, 15) increases accuracy, but even in 4- or 8-way tasks, EEG-only accuracy drops to 20–30% while the fused Sp-Aux Deformer remains above 60% [2411.06928].

Ablation studies confirm both the necessity and the robustness of multi-modal fusion: models degrade in stringent cross-validation but remain significantly above chance, suggesting some residual reliance on correlated audio and class content but demonstrating substantial information encoding in dual-modal cues.

## 6. Architectural and Empirical Insights

Self-attention-based Deformer encoders outperform CNNs and LSM-CNNs for attended direction decoding due to their capacity for long-range temporal and channelwise dependency modeling and flexible cross-token interaction. The intermediate fusion block grounds noisy neural attention traces in the structured spatial peaks of the audio spectrum. Without the spatial spectrum, deep models collapse to near-random performance in fine-grained directional decoding. The persistence of high accuracy even under strict cross-validation highlights the extractor’s robustness and the utility of synchronizing environmental and endogenous signals for neural decoding.

A plausible implication is that multi-class auditory attention can be decoded in realistic listening scenarios with moderate window lengths (1–10 s), enabling brain-driven spatial filtering for advanced hearing-assistive technologies [2411.06928].

## 7. Significance and Potential Applications

Sp-EEG-Deformer establishes a reference paradigm for multi-modal, high-class-count neural decoding in complex listening environments. Its methodology and demonstrated accuracy provide a concrete foundation for future research in neuro-steered beamforming and brain-computer interfaces targeting hearing restoration, where precise directional attention decoding is critical. The architecture and findings reinforce the principle that combined use of neurophysiological and environmental features surpasses the limits of unimodal approaches in neural interface engineering [2411.06928].

Source: https://www.emergentmind.com/topics/sp-eeg-deformer