FAConformer: Frequency-Aware Convolutional Transformer for Auditory Attention Decoding

Abstract: Auditory attention decoding (AAD) aims to infer the attended speaker from neural responses in multi-speaker acoustic environments and is a key problem for neuro-steered hearing systems. Although recent studies have achieved encouraging progress, existing AAD models still do not fully exploit frequency domain electroencephalography (EEG) information. In particular, most approaches introduce multi-band information through handcrafted feature extraction or direct cross-band feature concatenation, which mainly exploit frequency information at a shallow level and may overlook band-specific patterns and cross-band interactions. To address these limitations, this paper proposes FAConformer, a frequency-aware CNN-Transformer framework for AAD that explicitly integrates band-specific encoding and adaptive cross-band interaction. Specifically, FAConformer first decomposes EEG signals into multiple frequency bands and assigns each band to an independent CNN-Transformer encoder for band-specific modeling. The resulting band-wise features are then adaptively fused by a carefully designed frequency-aware attention (FAA) module that models cross-band dependencies by treating band-wise features as tokens. Further, band-wise auxiliary supervision (BAS) is introduced to prevent weakly contributing branches from being under-optimized during joint training. In this way, FAConformer performs frequency-aware modeling that more effectively exploits frequency domain information. Extensive experiments on two public AAD datasets with three decision-window lengths demonstrated that FAConformer consistently outperformed 12 competitive baselines, surpassing the current state-of-the-art model by 4.9%. Further analyses of band importance, ablation, and parameter sensitivity verify the effectiveness, robustness, and interpretability of the proposed framework. Code is available at https://github.com/wzwvv/FAConformer.

• The paper introduces FAConformer, a frequency-aware CNN-Transformer that explicitly decomposes EEG signals into multiple bands for robust auditory attention decoding.
• It utilizes dedicated CNN-Transformer encoders for each frequency band and a Frequency-Aware Attention module to adaptively fuse cross-band information.
• Experimental results demonstrate state-of-the-art accuracy gains, with a 4.9% improvement on DTU and 3.0% on KUL, alongside interpretable attention maps.

Frequency-Aware Modeling for Auditory Attention Decoding: An Expert Summary of FAConformer

Introduction and Motivation

Auditory attention decoding (AAD) seeks to discern which speaker a listener is attending to from neural recordings in multi-speaker environments, facilitating neuro-steered hearing technologies. Prevailing approaches leverage EEG but typically integrate frequency information at only a shallow level through handcrafted features or direct feature concatenation, neglecting the complex, hierarchical structure of band-specific patterns and cross-band interactions.

FAConformer addresses these limitations by introducing a frequency-aware CNN-Transformer framework that (1) performs explicit multi-band decomposition, (2) encodes each frequency band with an independent CNN-Transformer, and (3) applies a Frequency-Aware Attention (FAA) module for adaptive, data-driven cross-band fusion. To optimize all branches during training, a Band-wise Auxiliary Supervision (BAS) protocol prevents under-utilization of weaker but potentially informative frequency bands.

Model Architecture and Methodology

FAConformer is structured as a hierarchical pipeline comprising three core stages: multi-band decomposition, within-band encoding, and cross-band hierarchical fusion.

• Multi-band Decomposition: Input EEG data are decomposed into eight canonical frequency bands per trial, using separate band definitions tailored to the DTU and KUL datasets. This operation is performed in the frequency domain via FFT and inverse FFT to isolate band-limited signals, retaining time-domain information for each frequency range.
• Within-Band Encoding (WBE): Each band-limited signal is processed independently by a dedicated CNN-Transformer encoder. CNN layers learn local spatio-temporal structure, and Transformer layers capture long-range temporal dependencies within each frequency band. Band-specific features are extracted in a manner that ensures the preservation of unique neural oscillatory signatures relevant to auditory attention.
• Cross-Band Hierarchical Fusion (FAA): The sequence of band-wise features is aggregated as tokens and input into a two-layer, two-head Transformer module (FAA), which computes adaptive self-attention across bands. This design allows the model to assign non-uniform contributions from each frequency band informed by the data, facilitating the discovery and exploitation of cross-band dependencies critical to accurate AAD.
• Band-Wise Auxiliary Supervision (BAS): To ensure all bands are adequately trained—even those with low FAA-assigned weights—each band token branches to its own auxiliary classifier. A weighted sum of auxiliary and main losses, controlled by hyperparameter $\lambda$, guarantees robust and broadly optimized representations across all frequency channels.
• Inference: Only the global classifier is active at test time, streamlining deployment.

Experimental Results

Experiments were performed on two benchmark datasets—DTU and KUL—each comprising 64-channel EEG recordings in dual-speaker paradigms, with multiple decision-window lengths (2s, 1s, 0.1s). FAConformer was rigorously compared against twelve strong baselines, encompassing CNN-only, AAD-specific, and hybrid CNN-Transformer models.

Quantitative Performance

• FAConformer surpassed all baselines, achieving state-of-the-art average accuracy gains of 4.9% (DTU) and 3.0% (KUL) over previous bests, and showing especially marked advantage at shorter (more challenging) window lengths. For example, on the 0.1s setting on KUL, the performance edge over the second-best (IFNet) was 3.44%.
• Strong robustness was demonstrated across subjects, with FAConformer outperforming on both strong and weakly performing individuals, as reflected in the per-subject breakdown.

(Figure 1)

Figure 1: Subject-wise classification accuracy on (a) DTU and (b) KUL. Each bar denotes the mean accuracy of a model, and the error bar indicates the standard deviation.

Band Importance and Attention Patterns

FAA learned interpretable, subject-specific cross-band attention maps with clear dominant columns, indicating that only select bands (often in the higher frequency ranges for KUL) were emphasized for global decision-making. This substantiates FAA's capacity for adaptive, non-uniform aggregation.

(Figure 2)

Figure 2: Subject-wise FAA cross-band attention maps on (a) DTU and (b) KUL. Brighter values indicate stronger attention assigned from the query band to the key band.

Feature Discriminability

$t$-SNE analysis of latent features illustrated that FAConformer produces more tightly grouped and better-separated clusters for attention classes than strong CNN and CNN-Transformer baselines, confirming that the frequency-aware architectural enhancements offer qualitatively superior representation learning.

(Figure 3)

Figure 3: $t$-SNE visualizations of features extracted by EEGNet, IFNet, DBConformer, DBPNet, and FAConformer across datasets and window lengths.

Ablation and Sensitivity

Ablation confirmed the complementary value of WBE, BAS, and FAA. WBE offered statistically significant baseline gains, BAS stabilized optimization, and FAA afforded notable improvements for shorter time windows. Parameter sensitivity analysis indicated stable performance across a broad hyperparameter range, attesting to practical robustness.

Figure 4: Parameter sensitivity analysis of FAConformer regarding $\lambda$, FAA layers $L_f$, and FAA heads $H_f$, highlighting insensitivity to these key hyperparameters.

Model Complexity

While FAConformer introduces greater parameter counts and training times versus the most efficient CNN-only models (e.g., IFNet), inference times remain within acceptable bounds for offline or batch deployment scenarios, and the trade-off delivers unmatched decoding accuracy.

Theoretical and Practical Implications

FAConformer substantiates the theoretical proposition that hierarchical, frequency-aware modeling can surpass monolithic or shallow fusion approaches for EEG-based AAD. By treating frequency bands not as mere feature sources but as separate modeling branches whose outputs are adaptively fused, the architecture enforces a more neurobiologically plausible and technically powerful inductive bias. The explicit modeling of cross-band dependencies extends the representational capacity while BAS counters the risk of optimization stagnation in low-weight branches, a pitfall for multi-branch neural designs.

Practically, these advances position FAConformer as a preferable backbone for real-time or assistive brain-computer interfaces where robust, interpretable, and generalizable auditory attention detection is required. The demonstrated subject-level interpretability (FAA attention maps) aligns with clinical and neuroscientific needs for model transparency and diagnosis.

Future Directions

Potential trajectories emanate from this work:

• Lightweight frequency-aware modeling: Further reduction of parameterization and computational footprint, perhaps exploiting band selection or pruning guided by the FAA maps.
• Cross-domain generalization: Direct extension to cross-subject, cross-dataset, and multimodal (e.g., audio-EEG, MEG) scenarios.
• Integration with low-power neuromorphic or online AAD systems: Application to settings where latency, power efficiency, or hardware constraints are dominant.

Conclusion

FAConformer presents a frequency-aware CNN-Transformer architecture for auditory attention decoding that decisively improves upon existing paradigms by jointly optimizing within-band encoding, adaptive cross-band fusion, and robust auxiliary supervision. The architecture achieves statistically significant accuracy improvements across subjects and temporal resolutions, yields interpretable attention patterns, and possesses robust hyperparameter characteristics. These attributes recommend FAConformer as a new reference standard for EEG-based AAD and as a blueprint for future advances in frequency-aware brain-computer interface modeling.