Pretraining Large Brain Language Model for Active BCI: Silent Speech (2504.21214v2)

Abstract: This paper explores silent speech decoding in active brain-computer interface (BCI) systems, which offer more natural and flexible communication than traditional BCI applications. We collected a new silent speech dataset of over 120 hours of electroencephalogram (EEG) recordings from 12 subjects, capturing 24 commonly used English words for LLM pretraining and decoding. Following the recent success of pretraining large models with self-supervised paradigms to enhance EEG classification performance, we propose Large Brain LLM (LBLM) pretrained to decode silent speech for active BCI. To pretrain LBLM, we propose Future Spectro-Temporal Prediction (FSTP) pretraining paradigm to learn effective representations from unlabeled EEG data. Unlike existing EEG pretraining methods that mainly follow a masked-reconstruction paradigm, our proposed FSTP method employs autoregressive modeling in temporal and frequency domains to capture both temporal and spectral dependencies from EEG signals. After pretraining, we finetune our LBLM on downstream tasks, including word-level and semantic-level classification. Extensive experiments demonstrate significant performance gains of the LBLM over fully-supervised and pretrained baseline models. For instance, in the difficult cross-session setting, our model achieves 47.0\% accuracy on semantic-level classification and 39.6\% in word-level classification, outperforming baseline methods by 5.4\% and 7.3\%, respectively. Our research advances silent speech decoding in active BCI systems, offering an innovative solution for EEG LLM pretraining and a new dataset for fundamental research.

• The paper proposes a Large Brain Language Model (LBLM) and a novel Future Spectro-Temporal Prediction (FSTP) pretraining paradigm for decoding silent speech from EEG, utilizing a dataset of over 120 hours.
• Experimental results show LBLM significantly outperforms baselines, achieving 47.0% semantic-level and 39.6% word-level accuracy in challenging cross-session silent speech decoding.
• This research marks a significant step for EEG-based silent speech BCI, promising enhanced real-world applications, particularly for individuals with speech impairments.

Silent Speech Decoding Using Electroencephalogram in Brain-Computer Interfaces

The paper presented focuses on enhancing the silent speech decoding capabilities in active brain-computer interface (BCI) systems by leveraging innovative pretraining techniques on electroencephalogram (EEG) data. The authors collected a significant dataset, amounting to over 120 hours of EEG recordings across 12 subjects, capturing silent speech attempts with 24 commonly used English words. This dataset serves as a foundation for developing their proposed Large Brain LLM (LBLM), which aims to decode silent speech, providing an essential edge over traditional BCI applications.

Methodology

The research introduces the Large Brain LLM (LBLM), a sophisticated approach for decoding silent speech from EEG signals. The core innovation lies in the Future Spectro-Temporal Prediction (FSTP) pretraining paradigm, which consists of two critical stages aimed at extracting comprehensive EEG signal representations:

• Masked Spectro-Temporal Prediction (MSTP): This stage involves reconstructing masked EEG data in terms of waveforms, amplitude, and phase, warming up the model's capacity to interpret raw and spectral signal characteristics.
• Autoregressive Spectro-Temporal Prediction (ASTP): In advancing the model's prediction capabilities, ASTP requires the model to predict future EEG signals and spectral components based purely on past data, emphasizing temporal dependencies vital for language decoding.

The paper employs a conformer backbone integrated with a layer-gating mechanism, ensuring stable training and efficient extraction of both short- and long-term dependencies in EEG data.

Results and Implications

The experimental results highlight that the LBLM significantly exceeds the performance of established baselines in both word-level and semantic-level classification tasks. Notably, in the challenging cross-session setting, the LBLM reaches 47.0% accuracy in semantic-level classification and 39.6% accuracy in word-level classification, outperforming traditional fully-supervised and pretrained models by notable margins.

These results underscore the efficacy of the FSTP pretraining paradigm in enabling the model to learn intricate, contextually meaningful EEG representations. This advancement promises enhancements in real-world BCI systems, particularly for individuals with speech impairments. The ability to predict short-term future EEG signals adds another dimension to its utility, potentially improving the robustness and adaptability of active BCI systems in dynamic environments.

Future Directions

The research opens several avenues for further exploration. Enhancements in silent speech decoding can benefit from incorporating additional modalities or expanding vocabulary datasets to refine model training and applications. Furthermore, exploring the integration with other neural modalities, such as fMRI or MEG, could diversify model input and enhance decoding accuracy. The robust framework laid out by this paper holds considerable promise for driving advancements in more intuitive and efficient BCI systems, extending applications beyond current paradigms into areas such as assistive technologies and silent communication systems.

In conclusion, the proposed Large Brain LLM, supported by an extensive dataset and a novel pretraining strategy, marks a significant step forward in EEG-based silent speech decoding, with substantive implications for the future of brain-computer interface technologies.