Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
139 tokens/sec
GPT-4o
47 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

SI-SD: Sleep Interpreter through awake-guided cross-subject Semantic Decoding (2309.16457v3)

Published 28 Sep 2023 in cs.LG, eess.SP, and q-bio.NC

Abstract: Understanding semantic content from brain activity during sleep represents a major goal in neuroscience. While studies in rodents have shown spontaneous neural reactivation of memories during sleep, capturing the semantic content of human sleep poses a significant challenge due to the absence of well-annotated sleep datasets and the substantial differences in neural patterns between wakefulness and sleep. To address these challenges, we designed a novel cognitive neuroscience experiment and collected a comprehensive, well-annotated electroencephalography (EEG) dataset from 134 subjects during both wakefulness and sleep. Leveraging this benchmark dataset, we developed SI-SD that enhances sleep semantic decoding through the position-wise alignment of neural latent sequence between wakefulness and sleep. In the 15-way classification task, our model achieves 24.12% and 21.39% top-1 accuracy on unseen subjects for NREM 2/3 and REM sleep, respectively, surpassing all other baselines. With additional fine-tuning, decoding performance improves to 30.32% and 31.65%, respectively. Besides, inspired by previous neuroscientific findings, we systematically analyze how the "Slow Oscillation" event impacts decoding performance in NREM 2/3 sleep -- decoding performance on unseen subjects further improves to 40.02%. Together, our findings and methodologies contribute to a promising neuro-AI framework for decoding brain activity during sleep.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (37)
  1. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in neural information processing systems, 33:12449–12460, 2020.
  2. Sleep—a brain-state serving systems memory consolidation. Neuron, 111(7):1050–1075, 2023.
  3. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597–1607. PMLR, 2020.
  4. Neural latent aligner: cross-trial alignment for learning representations of complex, naturalistic neural data. In International Conference on Machine Learning, pages 5661–5676. PMLR, 2023.
  5. Decoding speech perception from non-invasive brain recordings. Nature Machine Intelligence, 5(10):1097–1107, 2023.
  6. Decoding visual neural representations by multimodal learning of brain-visual-linguistic features. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
  7. Dewave: Discrete eeg waves encoding for brain dynamics to text translation. arXiv preprint arXiv:2309.14030, 2023.
  8. Neural decoding of visual imagery during sleep. Science, 340(6132):639–642, 2013.
  9. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pages 448–456. pmlr, 2015.
  10. Altered neural responses to sounds in primate primary auditory cortex during slow-wave sleep. Journal of Neuroscience, 31(8):2965–2973, 2011.
  11. Large brain model for learning generic representations with tremendous EEG data in BCI. In The Twelfth International Conference on Learning Representations, 2024.
  12. Supervised contrastive learning. Advances in neural information processing systems, 33:18661–18673, 2020.
  13. Mechanisms of systems memory consolidation during sleep. Nature neuroscience, 22(10):1598–1610, 2019.
  14. Inducing task-relevant responses to speech in the sleeping brain. Current Biology, 24(18):2208–2214, 2014.
  15. Eegnet: a compact convolutional neural network for eeg-based brain–computer interfaces. Journal of neural engineering, 15(5):056013, 2018.
  16. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461, 2019.
  17. Decoding cognition from spontaneous neural activity. Nature Reviews Neuroscience, 23(4):204–214, 2022.
  18. Shaping overnight consolidation via slow-oscillation closed-loop targeted memory reactivation. Proceedings of the National Academy of Sciences, 119(44):e2123428119, 2022.
  19. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PMLR, 2021.
  20. Odor cues during slow-wave sleep prompt declarative memory consolidation. Science, 2007.
  21. A deep learning framework for neuroscience. Nature neuroscience, 22(11):1761–1770, 2019.
  22. If deep learning is the answer, what is the question? Nature Reviews Neuroscience, 22(1):55–67, 2021.
  23. Decoding material-specific memory reprocessing during sleep in humans. Nature Communications, 8(1):15404, 2017.
  24. Endogenous memory reactivation during sleep in humans is clocked by slow oscillation-spindle complexes. Nature communications, 12(1):3112, 2021.
  25. Respiration shapes sleep-oscillations and memory reactivation in humans. bioRxiv, pages 2023–03, 2023.
  26. The neural correlates of dreaming. Nature neuroscience, 20(6):872–878, 2017.
  27. Replay of neuronal firing sequences in rat hippocampus during sleep following spatial experience. Science, 271(5257):1870–1873, 1996.
  28. Decoding natural images from eeg for object recognition. arXiv preprint arXiv:2308.13234, 2023.
  29. Eeg conformer: Convolutional transformer for eeg decoding and visualization. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 31:710–719, 2022.
  30. Hierarchical nesting of slow oscillations, spindles and ripples in the human hippocampus during sleep. Nature Neuroscience, 18(11):1679–1686, 2015.
  31. Disruption of hierarchical predictive coding during sleep. Proceedings of the National Academy of Sciences, 112(11):E1353–E1362, 2015.
  32. Behavioral and brain responses to verbal stimuli reveal transient periods of cognitive integration of external world in all sleep stages. bioRxiv, pages 2022–05, 2022.
  33. An open-source, high-performance tool for automated sleep staging. Elife, 10:e70092, 2021.
  34. Attention is all you need. Advances in neural information processing systems, 30, 2017.
  35. Reactivation of hippocampal ensemble memories during sleep. Science, 265(5172):676–679, 1994.
  36. Biot: Biosignal transformer for cross-data learning in the wild. Advances in Neural Information Processing Systems, 36, 2024.
  37. Which reference should we use for eeg and erp practice? Brain topography, 32:530–549, 2019.
Citations (2)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now