Channel-Oriented Design for EEG-to-Music Reconstruction

Abstract: Brain-computer interfaces aim to decode naturalistic stimuli from neural signals, yet most progress to date has focused on vision and language. In this article, we study a more challenging but far less explored setting, EEG-to-music reconstruction, where signals are weak, distributed, and highly susceptible to noise and channel variability. Our central finding is that early channel mixing destroys weak but discriminative EEG signals. To address this, we propose a channel-oriented design with three key components. Specifically, channel-wise tokenization treats each electrode as an explicit token to retain spatially localized neural evidence, channel-wise multi-view self-distillation enforces consistency across temporal crops and random channel subsets to learn robust and distributed representations, and channel-wise data augmentation introduces structured channel dropout to improve invariance to noise, artifacts, and missing electrodes. Together, these components preserve weak yet informative signals across channels and enable stable alignment to a semantic music representation space. We integrate this channel-oriented design within an encoding-alignment-decoding pipeline for EEG-to-music reconstruction. Theoretically, we characterize when preserving channel-level structure leads to improved alignment. Empirically, we compare with a range of state-of-the-art baselines and demonstrate consistent and significant performance gains.

• The paper introduces a channel-oriented design that preserves channel-specific EEG signals, avoiding premature mixing and enhancing semantic alignment with music.
• It employs channel-wise tokenization, multi-view self-distillation, and structured dropout to robustly map EEG representations to semantic audio embeddings using transformer and diffusion models.
• Empirical results demonstrate improved CLAP scores and identification accuracy, while the approach offers enhanced interpretability and practical implications for EEG stimulus reconstruction.

Channel-Oriented Representation Design for EEG-to-Music Reconstruction

Motivation and Problem Setting

Electroencephalography (EEG)-based brain-computer interfaces (BCIs) have achieved notable advances in decoding vision and language stimuli; however, EEG-driven music reconstruction remains underexplored. The challenge stems from the distributed, weak, and artifact-prone nature of EEG signals in response to semantically-rich and temporally-complex music. Music-related neural signatures are spatially dispersed across electrodes, with individual channels carrying partial, noisy evidence. The paper identifies premature channel mixing as the primary bottleneck in current pipelines, arguing that early spatial pooling destroys localized discriminative signal before semantic alignment and generation.

Channel-Oriented Representation Framework

The proposed channel-oriented framework is instantiated via three principal components:

Channel-wise tokenization: Each electrode is treated as an explicit token in the transformer-based encoder, preserving spatial origins and avoiding early mixing. Temporal patches per channel are embedded and passed through global attention, enabling adaptive cross-channel aggregation and retention of channel identity.

Channel-wise multi-view self-distillation: The encoder is pretrained using a DINO-style self-distillation objective. Multiple temporal views (global and local crops) are generated, with random channel subsets dropped for the student branch. Consistency between student and teacher outputs under these augmentations enforces robustness to missing channels and stable identification of distributed neural patterns.

Channel-wise data augmentation: During alignment, structured channel dropout is applied—channels are randomly dropped, rather than arbitrary feature dimensions, to simulate noise, artifacts, or missing electrodes. This compels the model to form invariant representations under partial observations, enhancing robustness across montages and subjects.

Pipeline Architecture and Reconstruction

The full pipeline encompasses three stages: EEG encoding via the channel-oriented transformer; joint alignment to semantic music embeddings using contrastive learning; and music waveform reconstruction through a pretrained diffusion model.

On the music side, the CLAP audio encoder provides semantic embeddings, and AudioLDM conditions audio generation. A linear ridge regression adapter projects EEG embeddings into the CLAP space, decoupling representation learning from generative capacity.

Figure 1: The reconstruction pipeline. After alignment, EEG representations are mapped to the CLAP audio embedding space using ridge regression. The transformed embedding is then used to condition AudioLDM for music reconstruction.

Theoretical Analysis

Theoretical analysis is conducted to substantiate the channel-oriented design. Masking-induced augmentation graphs are formalized, and normalized cross-cluster overlap metrics are defined. Through a covariance-based argument, the paper proves that channel-level masking yields strictly smaller overlap with music-induced clusters compared to block-wise masking, under a biologically plausible covariance condition. This analysis demonstrates that per-channel structure preservation enables improved alignment to semantic targets, justifying the representation-centric approach.

Numerical Results

Systematic comparisons are performed against EEG2Mel and transformer-based EEG foundation models (LaBraM, EEGPT, CBraMod). All methods use the same alignment protocol and AudioLDM decoder, isolating effects of representation design.

Notable empirical findings include:

• Best-in-class alignment and reconstruction: The channel-oriented model achieves a CLAP score of 0.683 and 50-way identification accuracy of 0.487, improving by 0.085 over the strongest foundation baseline and 0.228 over EEG2Mel.
• Channel-wise tokenization is critical: Block tokenization degrades 50-way accuracy from 0.487 to 0.141, confirming that explicit electrode tokens substantially improve fine-grained semantic alignment.
• Self-distillation and channel dropout are essential: Removing multi-view pretraining or channel dropout drops performance markedly, validating the robustness and distributed representation learned by these components.
• Spectrogram metrics (SSIM, PSNR) misrepresent perceptual quality: Direct spectrogram regression (EEG2Mel) achieves high SSIM/PSNR but fails on semantic metrics, underscoring the superiority of semantic alignment for perceptual targets.
• Interpretability: Attention maps over channel tokens reveal anatomically and stimulus-specific structures, with posterior temporal electrodes strongly attended during rhythmic music, and inter-song clustering reflecting cultural familiarity.

Practical and Theoretical Implications

Preservation of channel-level structure in EEG transformers is shown to be vital for decoding continuous, semantically-rich stimuli from weak neural data. The empirical gains are achieved without reliance on decoder complexity; instead, they result from robust, interpretable EEG representations. The theoretical framework suggests further directions in graph-based augmentation, channel group modeling, and cross-modal embedding regularization. The BCI field should prioritize representation-centric pipelines for music, speech, and environmental auditory decoding, leveraging channel-level masking, distillation, and augmentation.

From a neuroengineering perspective, the approach supports more reliable stimulus decoding with limited or noisy electrode coverage, and enables anatomically-resolved interpretability. In AI research, it points toward foundation models for neural time-series focused on stimulus reconstruction, rather than classification, and motivates joint benchmarking across modalities.

Future directions include scaling to larger EEG-audio datasets, extension to non-music auditory inputs (speech, environmental sound), and incorporating more expressive alignment adapters. Integration with source-localized EEG modeling and unsupervised electrode selection may further improve generalization and interpretability.

Conclusion

This paper rigorously demonstrates that channel-wise tokenization, temporal self-distillation, and channel-oriented augmentation collectively enable superior EEG-to-music semantic alignment and reconstruction (2606.04040). The representation-centric principle—preserving per-channel structure in neural decoding pipelines—should be broadly adopted in EEG modeling for naturalistic stimulus retrieval and reconstruction. The empirical and theoretical results strongly advocate for deferred channel mixing and distributed robustness as key elements in future neural stimulus decoding architectures.