SilentSpeech-EEG: EEG Speech Decoding

• SilentSpeech-EEG is a family of EEG-based paradigms that convert brain signals into acoustics, text, or class labels across varied task settings.
• Experimental paradigms range from controlled direct synthesis of MFCC features to large-scale benchmarks using transformer and Conformer-based models.
• Methodologies leverage end-to-end learning, region-specific feature extraction, and advanced tokenizers to address subject variability and improve decoding robustness.

Searching arXiv for recent SS-EEG and related silent-speech EEG papers to ground the article. arxiv_search(query="SilentSpeech-EEG OR EEG silent speech decoding OR speech synthesis using EEG", max_results=10, sort_by="submittedDate") SilentSpeech-EEG (SS-EEG) denotes a family of EEG-based speech and language decoding paradigms rather than a single standardized system. In the arXiv literature, the label has been used for direct EEG-to-speech synthesis, silent word classification, semantic decoding, continuous EEG-to-text recognition, and a framework for explicitly modeling non-speech cognitive states during speech imagination. Across these usages, the common premise is that noninvasive scalp EEG contains task-relevant structure for speech perception, speech production, covert articulation, or imagined speech, and that this structure can be exploited by sequence models, structured decoders, or large pretrained backbones to generate acoustics, text, or class labels (Krishna et al., 2020, Krishna et al., 2020, Sharon et al., 2020, Zhou et al., 29 Apr 2025, Zhao et al., 29 Jan 2026).

1. Terminological scope and conceptual boundaries

The earliest works using the SS-EEG label define materially different targets. In "Speech Synthesis using EEG" (Krishna et al., 2020), SS-EEG is a direct EEG-to-speech synthesis system that maps EEG feature sequences to 13-dimensional MFCC acoustic features and then reconstructs waveforms with Griffin-Lim. In "The 'Sound of Silence' in EEG -- Cognitive voice activity detection" (Sharon et al., 2020), SS-EEG is a two-stage imagined/heard speech decoding framework that first detects cognitive speech activity versus non-speech states and then constrains unit recognition. In "Continuous Silent Speech Recognition using EEG" (Krishna et al., 2020), the task is sentence-level EEG-to-text translation with a CTC ASR model. In later work, SS-EEG becomes the name of a large-scale benchmark for word-level silent speech decoding from EEG, and also the substrate for models such as LBLM and BrainStack (Zhou et al., 29 Apr 2025, Zhao et al., 29 Jan 2026).

This multiplicity matters for interpretation. Some SS-EEG studies are passive or partially stimulus-driven, such as listen EEG recorded while a subject hears an utterance or sentence presentation during silent reading; others are framed as active BCI settings in which a subject voluntarily attempts to articulate a word silently without relying on external listening inputs (Krishna et al., 2020, Krishna et al., 2020, Zhou et al., 29 Apr 2025). A further boundary is introduced by the cognitive VAD work: its non-speech EEG state is explicitly defined as a state that does not correspond to brain cognitive speech activity, but it is not resting-state EEG; it is a task-embedded silence state occurring within a conscious speech-related task (Sharon et al., 2020).

A common misconception is therefore to treat SS-EEG as either a single dataset or a single decoding problem. The literature instead uses the term for several related formulations positioned at different points along the pipeline from neural activity to acoustics, words, semantic groups, or text.

2. Experimental paradigms and data regimes

A compact way to organize SS-EEG is by output target and experimental paradigm.

SS-EEG usage	Representative setup	Primary output
Direct synthesis from EEG	Listen EEG and spoken EEG, 4 subjects, 4 commands	MFCCs or waveform
EEG-only or EEG-assisted recognition	Limited-vocabulary words or vowels, noisy or absent speech	Class label
Continuous silent recognition	Silent reading of USC-TIMIT sentences	Text
Cognitive VAD	Heard EEG and imagined EEG with silence-state structure	Activity labels and constrained unit decoding
Large-scale benchmark decoding	24 silent English words, 16 sessions per subject	Word or semantic class
Heterogeneous EEG/EMG decoding	Multi-subject, multi-configuration silent speech	Word or mora sequence

The early synthesis and recognition studies are small and tightly controlled. The 2020 synthesis paper reports four subjects, all UT Austin undergraduates in their early twenties, with 3 female and 1 male, each listening to four natural utterances and then speaking the same utterances aloud; the commands were “Hi Bixby,” “Call Mom,” “Open Camera,” and “What’s the weather,” with 70 speech-EEG recordings collected for each subject and sentence using a Brain Vision EEG recording system with 32 wet electrodes and 10–20 placement (Krishna et al., 2020). The related waveform paper reuses a four-subject listen/spoken dataset collected with a Brain Products ActiChamp EEG amplifier, 32 wet electrodes, and simultaneous audio, while the 2019 recognition paper uses four male undergraduate students, five vowels and four words, background-noise and no-noise conditions, and parallel speech-EEG recordings with a BrainVision EEG system and 32 wet EEG electrodes (Krishna et al., 2020, Krishna et al., 2019).

The continuous sentence-level study is likewise small but shifts the target from classification to transcription. It records four male subjects in their early to mid 20s, with each subject silently reading the first 30 English sentences from USC-TIMIT three times, giving 90 EEG recordings per subject, acquired with Brain Products hardware and 32 wet electrodes following the standard 10–20 system (Krishna et al., 2020).

By contrast, the recent SS-EEG benchmark papers scale the problem to word-level covert articulation. One paper introduces a new silent-speech EEG dataset with 12 subjects, about 120 total hours of recording, 16 sessions per subject, 6000 trials per subject, 24 English words, and 6 semantic groups, collected from 12 right-handed native English speakers from Australia using a 128-channel Neuroscan Quik-Cap, SynAmps RT 128-channel amplifier, and CURRY 9 (Zhou et al., 29 Apr 2025). The BrainStack paper describes SS-EEG as over 120 hours of EEG recordings from 12 subjects performing 24 silent words, 60,000 trials total, about 250 repeats per class per subject, and a structured four-stage trial protocol consisting of 5 seconds of rest including a 1.5-second fixation cross, 1 second of word presentation, a 0.2-second auditory cue, and 1.5 seconds of silent repetition; it further notes that 2 subjects were excluded due to severe signal contamination, leaving 10 usable participants in the released benchmark (Zhao et al., 29 Jan 2026).

A different branch of the literature studies silence structure rather than direct lexical decoding. The cognitive VAD paper uses two datasets: a 128-channel EGI Geodesic net dataset with 16 healthy subjects in an anechoic chamber, and a 4-channel InteraXon Muse dataset with 8 subjects, both centered on passive listening followed by speech imagination, with only 5 phrases common to both datasets used in experiments (Sharon et al., 2020).

Another recent extension broadens SS-EEG to heterogeneous biosignal configurations. The EEG/EMG decoding paper trains across multiple datasets totaling around 220 hours, including 8 healthy participants and 1 patient in the main eego sports dataset, with silent and vocalized Japanese word and sentence tasks, a 64-word vocabulary, 153 sentences, and deliberately heterogeneous electrode layouts across datasets and users (Inoue et al., 16 Jun 2025).

3. Signal acquisition, preprocessing, and feature representations

The preprocessing pipelines vary substantially across SS-EEG formulations, but several recurrent patterns are visible. Early BrainVision/Brain Products studies use 1000 Hz EEG, a 4th-order IIR band-pass filter from 0.1 Hz to 70 Hz, a 60 Hz notch filter, and ICA artifact removal through EEGLAB or related tooling to suppress ECG, EMG, EOG, and related artifacts (Krishna et al., 2020, Krishna et al., 2020, Krishna et al., 2020, Krishna et al., 2019). The cognitive VAD paper instead uses a 0.1–60 Hz band-pass filter, a 50 Hz notch filter, baseline correction, bad-channel detection and interpolation for EGI, and ICA artifact removal with removal of the top 1 component for Muse and the top 3 for EGI (Sharon et al., 2020).

The large-benchmark papers adopt higher-density acquisitions and more explicit session control. BrainStack uses a 128-channel Neuroscan EEG system, 1000 Hz sampling, 1–45 Hz band-pass filtering, segmentation into clean trial epochs, per-channel z-score normalization, and artifact rejection, with 8 EXG channels that include EOG, EMG, and reference electrodes (Zhao et al., 29 Jan 2026). The LBLM paper reports raw recordings with 133 channels in total—122 EEG channels, 4 eye-movement electrodes, 6 reference/noise electrodes, and 1 trigger channel—followed by 1–75 Hz band-pass filtering, 50 Hz notch filtering, rereferencing by average of the 122 EEG channels, ICA artifact removal, segmentation into 2 s epochs with 0.5 s overlap, and downsampling to 250 Hz for training (Zhou et al., 29 Apr 2025). The heterogeneous EEG/EMG system uses a 50 Hz notch, common average reference for EEG, 2–120 Hz band-pass filtering, and 240 Hz resampling, with an additional 27 Hz notch for patient data to remove medical device interference (Inoue et al., 16 Jun 2025).

Representation choice has evolved from handcrafted features toward raw-signal and patch-based encoders. The 2019 and 2020 small-data studies repeatedly use five statistical EEG features per channel—root mean square, zero crossing rate, moving window average, kurtosis, and power spectral entropy—typically extracted at 100 Hz and then compressed with KPCA or autoencoders (Krishna et al., 2019, Krishna et al., 2020, Krishna et al., 2020). In the direct synthesis paper, 31 channels yield 155 EEG features, which are reduced to 30 dimensions by KPCA with a polynomial kernel of degree 3 (Krishna et al., 2020). In the continuous CTC paper, 155-dimensional EEG features are reduced to 20 dimensions by KPCA (Krishna et al., 2020). In the 2019 ASR paper, 155 features are reduced to 39 dimensions for most datasets and then expanded with derivatives to 117 final EEG features, while vowels without noise use an autoencoder reduction to 6 dimensions and then 18 final features with derivatives (Krishna et al., 2019).

The synthesis paper based on Krishna et al. (2019) instead relies on three previously introduced EEG feature sets, extracted at 100 Hz per channel and reduced by KPCA for two of the three sets, yielding 30, 50, and 93 dimensions respectively (Krishna et al., 2020). The cognitive VAD work uses Short-Term Energy with a Hamming window of length 125 samples as its main feature (Sharon et al., 2020).

Recent benchmark-scale models largely abandon handcrafted statistics. BrainStack partitions the scalp into 7 anatomically motivated regions and learns expert representations over those regions plus a global expert (Zhao et al., 29 Jan 2026). LBLM uses overlapping patches from each EEG channel with patch length $P=25$, stride $S=6$, sinusoidal positional embedding, and subject embedding, while the heterogeneous EEG/EMG system uses tokenizer modules that map variable electrode layouts into a fixed latent space (Zhou et al., 29 Apr 2025, Inoue et al., 16 Jun 2025). This suggests a transition from low-rate engineered features toward end-to-end spatial-temporal representation learning.

4. Modeling strategies and objective functions

The earliest SS-EEG synthesis systems are regression models from EEG to acoustics. "Speech Synthesis using EEG" (Krishna et al., 2020) uses a two-layer GRU regression network with 256 hidden units in the first GRU layer, dropout 0.2, 128 hidden units in the second GRU layer, dropout 0.2, and a time-distributed dense layer with 13 output units. The model predicts 13-dimensional MFCC acoustic features at each 100 Hz time step, is trained with MSE using Adam at learning rate 0.01, batch size 100, and 250 epochs in Keras, and reconstructs waveforms with Griffin-Lim.

The companion waveform paper moves one step closer to direct neural speech synthesis. Its main path is Raw EEG waveform (31 channels) → temporal convolutional network (TCN) → upsampling → time-distributed dense layer → predicted audio waveform, with a first TCN layer of 256 filters, upsampling by a factor of 15, dropout 0.2, a second TCN layer of 32 filters, final upsampling, and a time-distributed dense layer with linear activation and 1 hidden unit, trained with Adam and MSE for 5000 epochs (Krishna et al., 2020). The same paper also keeps a second path in which EEG features are mapped to 16 different acoustic features with GRU regression.

Recognition-oriented SS-EEG systems use classification and sequence-decoding losses instead of acoustic regression. The 2019 ASR paper employs a GRU-based recurrent neural network in TensorFlow consisting of one GRU layer, an average pooling layer, a dense layer, and an output classification layer, trained with Adam, learning rate 0.001, dropout 0.2 on the dense layer, batch size 1, and cross-entropy, under three input modes: MFCC only, MFCC + EEG concatenated, and EEG only (Krishna et al., 2019). It also introduces generalized distillation, with a teacher trained on EEG + MFCC, soft targets from the teacher, and a student trained on MFCC + soft targets.

The continuous silent reading study uses a CTC ASR pipeline with 2 GRU layers of 128 and 64 hidden units, dropout 0.1 in each GRU layer, a Temporal Convolutional Network with 32 filters, a time-distributed dense layer with softmax activation over a character-based vocabulary, and beam-search decoding with an external 4-gram LLM via shallow fusion (Krishna et al., 2020). The objective is CTC loss.

The cognitive VAD formulation is architecturally distinct. It uses a GMM-HMM sequence model implemented in Kaldi, with around 3 Gaussian mixtures per state on average, HMM states tuned experimentally, 40 update iterations, Viterbi decoding with lattice beam width 1 to 4, and a first-stage activity detector using a boost silence probability of 1.28 (Sharon et al., 2020). The central idea is hierarchical classification: activity detection first labels frames as speech or non-speech and refines non-speech into NS$_b$, NS$_i$, and NS$_e$, after which unit decoding is restricted to candidates consistent with the inferred silence structure.

Modern benchmark-scale SS-EEG models incorporate explicit neuroanatomical or self-supervised inductive biases. BrainStack is a functionally guided Neuro-MoE framework with 7 anatomically motivated regional experts—Prefrontal, Frontal, Central, Left-Temporal, Right-Temporal, Parietal, and Occipital—plus a transformer-based global expert, adaptive expert routing, and global-to-local distillation trained with a hierarchical multi-objective loss (Zhao et al., 29 Jan 2026). LBLM is a 22.61M-parameter Conformer-based Large Brain LLM with a layer-gating mechanism and a pretraining scheme called Future Spectro-Temporal Prediction, consisting of Masked Spectro-Temporal Prediction followed by Autoregressive Spectro-Temporal Prediction over raw waveform, Fourier amplitude, and Fourier phase targets (Zhou et al., 29 Apr 2025). The heterogeneous EEG/EMG system uses a tokenizer, temporal positional encoding, a CLS token, an 11-layer Conformer encoder, and task-specific heads, with four tokenizer variants—GAP, ES, SS, and OTFK—to cope with variable electrode placements (Inoue et al., 16 Jun 2025).

5. Empirical results across formulations

The headline numerical results differ by task and metric. In the acoustic-regression synthesis setting, the 2020 paper reports per-subject MCD, RMSE, and normalized RMSE for listen EEG → listen MFCC and spoken EEG → spoken MFCC. Its best highlighted values are a spoken EEG MCD as low as 0.433 and a listen EEG MCD as low as 0.471, compared with 5.737 and 1.34 in Krishna et al. (2019), while also observing that all three EEG feature sets perform similarly when enough training data are available (Krishna et al., 2020). The direct waveform paper reports the lowest spoken-condition RMSE as 0.583 for subject 2 and the lowest listen-condition RMSE as 0.489 for subject 2, while noting that the predicted waveform captures broad characteristics of the true waveform, is less noisy than earlier acoustic-feature reconstruction, but is not intelligible (Krishna et al., 2020).

In limited-vocabulary recognition, the 2019 EEG-assisted ASR paper reports very high small-closed-set accuracies. On the test set, EEG-only recognition reaches 99.38% for words with noise, compared with 93.00% for MFCC-only and 97.50% for MFCC+EEG; for words without noise, EEG-only reaches 96.87%. The same paper reports that distillation improves the MFCC-only student model, with the best no-noise word result at $T=2, \lambda=0.2$, yielding 98.61% test accuracy (Krishna et al., 2019).

In continuous sentence-level EEG-to-text decoding, results remain much weaker. The CTC paper reports test WERs of 74.86% for the 12-total-sentence / 5-unique-sentence setting and 83.34% for the 72-total-sentence / 30-unique-sentence setting, with 92.55% WER when training on the first 3 subjects and testing on the 4th subject (Krishna et al., 2020). The cognitive VAD work reports that hierarchical classification gives an absolute average improvement of 7.8% accuracy over baseline in the inter-session case, with activity detection accuracy about 76%, and that HC outperforms BL, DNS, and DNS3 across Muse and EGI settings (Sharon et al., 2020).

Benchmark-scale word-level decoding on the large SS-EEG corpus produces stronger but still far-from-saturated accuracies. The LBLM paper reports average cross-session accuracies of 39.6% in word-level classification and 47.0% in semantic-level classification for the full LBLM + MSTP + ASTP model, outperforming the best baseline methods by 7.3% and 5.4% absolute respectively (Zhou et al., 29 Apr 2025). BrainStack reports average within-subject word classification accuracies of 28.78% for EEGNet, 29.50% for TCNet, 23.89% for EEGConformer, 28.28% for STTransformer, 18.29% for LaBraM, 32.78% for BrainStack_Homo, 37.19% for BrainStack_RoI5, and 41.87% for full BrainStack, corresponding to a +12.37 percentage-point gain over TCNet; its best subject-wise result is 88.05% on S01 (Zhao et al., 29 Jan 2026).

In the heterogeneous EEG/EMG setting, larger multi-configuration pretraining produces the strongest closed-set word results among the surveyed works, but under a different modality regime. The subject-specific tokenizer trained on the full all dataset yields $95.3 \pm 2.6\%$ word classification accuracy for healthy participants and 54.5% for a speech-impaired patient, compared with single-subject baselines of $70.1 \pm 17.1\%$ and 13.2% (Inoue et al., 16 Jun 2025).

These figures are not directly comparable across the full literature because they evaluate different targets—MFCC regression, waveform prediction, word classification, semantic classification, and sentence transcription—with different metrics including MCD, RMSE, normalized RMSE, accuracy, WER, F1-score, and PER. The comparative question is therefore better framed within each task family than across the entire SS-EEG label.

6. Limitations, misconceptions, and open problems

Subject dependence is a persistent limitation. The 2020 synthesis paper reports large subject variability, with Subject 4 consistently much worse than Subject 1, and concludes that each person’s brain activity patterns are unique even for the same stimuli and tasks (Krishna et al., 2020). The continuous CTC paper shows the same issue at the sequence level, with 92.55% WER in the cross-subject transfer setting (Krishna et al., 2020). BrainStack likewise reports substantial subject variability, with S01 at 88.05% and subjects such as S05 and S06 much harder, while its routing analysis suggests that Occipital expert weights correlate positively with subject accuracy and that Temporal experts are consistently important (Zhao et al., 29 Jan 2026).

Another limitation is that high headline accuracies often arise in small, controlled, closed-vocabulary regimes. The 2019 EEG-only recognition results are based on four subjects, five vowels and four words, controlled timing, and a fixed 60 dB background music noise source; the paper explicitly states that no large-vocabulary ASR is demonstrated and that broad generalization remains unproven (Krishna et al., 2019). The continuous sentence work extends the output space but still uses only 30 unique sentences and reports high WERs (Krishna et al., 2020). The direct waveform synthesis paper is explicit that the generated speech is not intelligible (Krishna et al., 2020).

The literature also distinguishes between active and passive decoding, a distinction that is sometimes blurred. The LBLM paper argues that active BCI silent speech decoding should infer intended silent speech directly from EEG without relying on external stimuli such as listening or reading inputs, whereas earlier SS-EEG systems frequently use listened utterances, displayed words, or silently read sentences as the task structure (Zhou et al., 29 Apr 2025). A plausible implication is that some improvements in large benchmarks may reflect better handling of covert articulation under controlled cueing rather than resolution of fully spontaneous internal speech decoding.

Silence modeling introduces a separate conceptual correction. The cognitive VAD paper shows that non-speech states in EEG should not be conflated with resting EEG; instead, beginning silence, inter-unit silence, and ending silence may have distinct temporal and topographic structure within speech-related tasks (Sharon et al., 2020). This directly challenges any formulation that assumes every frame in an imagined-speech segment should be labeled as speech content.

Data scale, leakage control, and heterogeneity remain central open problems. BrainStack adopts a session-wise split with 14 training, 1 validation, and 1 test session per subject specifically to avoid leakage from adjacent trials in the same recording block, and the LBLM paper emphasizes cross-session evaluation with held-out sessions as a harder and more realistic setting (Zhao et al., 29 Jan 2026, Zhou et al., 29 Apr 2025). The heterogeneous EEG/EMG study argues that montage variability is itself a deployment bottleneck and shows that preserving spatial structure through subject-specific tokenization or an on-the-fly kernel is superior to overly lossy pooling, while also noting that patient generalization remains uncertain because only one patient was studied and that cross-language transfer is limited (Inoue et al., 16 Jun 2025).

Future directions are therefore relatively consistent across the literature: larger datasets, broader vocabularies, better cross-subject and cross-session robustness, stronger LLMs for sentence-level decoding, improved handling of variable electrode layouts, more patient data, and clinically relevant deployment. The 2020 synthesis paper states that the datasets will be published to support further research (Krishna et al., 2020). The direct waveform paper frames improved intelligibility as necessary for accessible speech prosthetics for people with aphasia, stuttering, severe stroke, ALS, and other speech-impairing conditions (Krishna et al., 2020). The heterogeneous EEG/EMG study adds post-laryngectomy and broader assistive communication settings, while the LBLM and BrainStack papers position large-scale SS-EEG as a benchmark substrate for scalable and interpretable brain-language decoding (Inoue et al., 16 Jun 2025, Zhou et al., 29 Apr 2025, Zhao et al., 29 Jan 2026).