EEGFormer: Towards Transferable and Interpretable Large-Scale EEG Foundation Model (2401.10278v1)
Abstract: Self-supervised learning has emerged as a highly effective approach in the fields of natural language processing and computer vision. It is also applicable to brain signals such as electroencephalography (EEG) data, given the abundance of available unlabeled data that exist in a wide spectrum of real-world medical applications ranging from seizure detection to wave analysis. The existing works leveraging self-supervised learning on EEG modeling mainly focus on pretraining upon each individual dataset corresponding to a single downstream task, which cannot leverage the power of abundant data, and they may derive sub-optimal solutions with a lack of generalization. Moreover, these methods rely on end-to-end model learning which is not easy for humans to understand. In this paper, we present a novel EEG foundation model, namely EEGFormer, pretrained on large-scale compound EEG data. The pretrained model cannot only learn universal representations on EEG signals with adaptable performance on various downstream tasks but also provide interpretable outcomes of the useful patterns within the data. To validate the effectiveness of our model, we extensively evaluate it on various downstream tasks and assess the performance under different transfer settings. Furthermore, we demonstrate how the learned model exhibits transferable anomaly detection performance and provides valuable interpretability of the acquired patterns via self-supervised learning.
- XAI for transformers: Better explanations through conservative propagation. In International Conference on Machine Learning, 435–451. PMLR.
- An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv:1803.01271.
- A simple framework for contrastive learning of visual representations. In International conference on machine learning, 1597–1607. PMLR.
- Amplifying Pathological Detection in EEG Signaling Pathways through Cross-Dataset Transfer Learning. arXiv preprint arXiv:2309.10910.
- Time-series representation learning via temporal and contextual contrasting. arXiv preprint arXiv:2106.14112.
- Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 12873–12883.
- Som-vae: Interpretable discrete representation learning on time series. arXiv preprint arXiv:1806.02199.
- Unsupervised scalable representation learning for multivariate time series. Advances in neural information processing systems, 32.
- Large language models are zero-shot time series forecasters. arXiv preprint arXiv:2310.07820.
- Clinically Relevant Unsupervised Online Representation Learning of ICU Waveforms. In ICLR 2023 Workshop on Time Series Representation Learning for Health.
- The TUH EEG CORPUS: A big data resource for automated EEG interpretation. In 2014 IEEE signal processing in medicine and biology symposium (SPMB), 1–5. IEEE.
- Self-supervised contrastive learning for EEG-based sleep staging. In 2021 International Joint Conference on Neural Networks (IJCNN), 1–8. IEEE.
- BENDR: using transformers and a contrastive self-supervised learning task to learn from massive amounts of EEG data. Frontiers in Human Neuroscience, 15: 653659.
- EEGNet: a compact convolutional neural network for EEG-based brain–computer interfaces. Journal of neural engineering, 15(5): 056013.
- Temporal dependencies in feature importance for time series prediction. In The Eleventh International Conference on Learning Representations.
- A multi-domain adaptive graph convolutional network for EEG-based emotion recognition. In Proceedings of the 29th ACM International Conference on Multimedia, 5565–5573.
- A Multi-view Spectral-Spatial-Temporal Masked Autoencoder for Decoding Emotions with Self-supervised Learning. In Proceedings of the 30th ACM International Conference on Multimedia, 6–14.
- Protecting the Future: Neonatal Seizure Detection with Spatial-Temporal Modeling. arXiv preprint arXiv:2307.05382.
- Labeling EEG Components with a Bag of Waveforms from Learned Dictionaries. In ICLR 2023 Workshop on Time Series Representation Learning for Health.
- Contrastive representation learning for electroencephalogram classification. In Machine Learning for Health, 238–253. PMLR.
- A time series is worth 64 words: Long-term forecasting with transformers. arXiv preprint arXiv:2211.14730.
- Beit v2: Masked image modeling with vector-quantized visual tokenizers. arXiv preprint arXiv:2208.06366.
- Variational instance-adaptive graph for EEG emotion recognition. IEEE Transactions on Affective Computing.
- A dataset of neonatal EEG recordings with seizure annotations. Scientific data, 6(1): 1–8.
- Modeling Multivariate Biosignals With Graph Neural Networks and Structured State Space Models. In Conference on Health, Inference, and Learning, 50–71. PMLR.
- Self-supervised graph neural networks for improved electroencephalographic seizure analysis. arXiv preprint arXiv:2104.08336.
- Unsupervised representation learning for time series with temporal neighborhood coding. arXiv preprint arXiv:2106.00750.
- Neural discrete representation learning. Advances in neural information processing systems, 30.
- Attention Is All You Need. arXiv:1706.03762.
- BrainBERT: Self-supervised representation learning for intracranial recordings. arXiv preprint arXiv:2302.14367.
- Hierarchical dynamic graph convolutional network with interpretability for EEG-based emotion recognition. IEEE Transactions on Neural Networks and Learning Systems.
- Learning Topology-Agnostic EEG Representations with Geometry-Aware Modeling. In Thirty-seventh Conference on Neural Information Processing Systems.
- Ts2vec: Towards universal representation of time series. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, 8980–8987.
- One Fits All: Power General Time Series Analysis by Pretrained LM. arXiv preprint arXiv:2302.11939.