EEG2TEXT: Decoding Brain Signals to Text

• EEG2TEXT is the process of translating non-invasive EEG signals into natural language text using deep sequence-to-sequence models and cross-modal alignment.
• Key challenges include low signal-to-noise ratios, high inter-subject variability, and risks of language model hallucination.
• Recent approaches combine transformer-based EEG encoders with pretrained language decoders to achieve improved semantic accuracy and robust benchmarking.

Electroencephalography-to-Text (EEG2TEXT) refers to the open-vocabulary generation of natural language directly from non-invasive brain signals measured via electroencephalography. It is a core challenge in brain-computer interface (BCI) research, driven by both clinical communicative applications and cognitive science. The canonical EEG2TEXT task is: given a variable-length sequence of feature vectors extracted from raw EEG during natural reading or target presentation, generate the corresponding spoken or written text, potentially over the entire language vocabulary. This decoding task faces unique obstacles due to low signal-to-noise ratio, cross-subject idiosyncrasies, information-capacity mismatch between EEG and language, and the risk of text hallucinations from powerful LLMs. Recent research has focused on deep sequence-to-sequence architectures, representation alignment, and rigorous benchmarking to establish reliable, semantically grounded brain-to-text interfaces.

1. Problem Formulation and Core Challenges

EEG2TEXT aims to learn a mapping $f_\theta: X \rightarrow S$, where $$X = [x_1, x_2, ..., x_L] \in \mathbb{R}^{L\times d}$$ is a sequence of $d$-dimensional EEG feature vectors (typically word-epoch aligned), and $S = [s_1, ..., s_{|S|}]$ is the tokenized natural language target—often an unconstrained sequence over a large vocabulary. The main technical impediments are:

• Subject and Session Variability: EEG encodings are highly subject-dependent, making cross-subject generalization difficult (Feng et al., 2023).
• Domain Gap: Semantic representations of language differ greatly from bioelectric features captured by EEG.
• Low SNR and High Noise: Non-invasive scalp EEG is orders of magnitude weaker than neural signals relevant for semantics, further challenged by artifacts and environmental noise (Jo et al., 10 May 2024).
• Information Bottleneck and Posterior Collapse: The information bandwidth of EEG is insufficient for verbatim language decoding, driving generative decoders to fall back on generic hypotheses rather than signal-inferred content (Liu et al., 21 May 2025).
• Overstated Benchmarking: Widespread use of teacher-forcing at inference and lack of noise baselines has led to overestimation of true EEG-driven performance (Jo et al., 10 May 2024).

2. Model Architectures and Representation Learning

EEG2TEXT models universally adopt an encoder–decoder paradigm, but implementations vary in EEG representation, semantic alignment, and integration with LLMs:

• EEG Encoder: Transformer-based deep encoders now dominate, with 6–12 self-attention layers receiving either frequency-band aggregated vectors (e.g., d=840; 105 channels × 8 bands), spatially-compressed region tokens, or 1D convolutions over time and/or channels (Feng et al., 2023, Liu et al., 3 May 2024). Subject-adaptive modules (e.g., learned per-subject vectors) are often included to mitigate cross-individual variability (Amrani et al., 2023, Gedawy et al., 11 Feb 2025).
• Cross-Modal Alignment: Several models employ explicit contrastive losses (semantically similar EEG–text pairs pulled together, dissimilar pairs pushed apart) (Feng et al., 2023, Wang et al., 27 Feb 2024, Liu et al., 21 May 2025), or cross-modal codebooks and InfoNCE losses for fine-grained semantic matching (Tao et al., 14 Sep 2024).
• Multi-View and Spatial Modules: EEG2TEXT architectures such as multi-view transformers process grouped electrode subsets (e.g., Broca’s, Wernicke’s, occipital) through separate channels, allowing the model to learn region-specific linguistic features and fuse them via global attention (Liu et al., 3 May 2024).
• Language Decoders: Pretrained LMs such as BART, T5, PEGASUS, MiniLM, and Flan-T5 serve as autoregressive decoders, typically with cross-attention over EEG-derived embeddings (Feng et al., 2023, Amrani et al., 2023, Khushiyant, 8 Sep 2025). Clean-up passes via GPT-4 can further improve grammaticality and sentence fluency (Amrani et al., 2023, Gedawy et al., 11 Feb 2025).
• Instruction-Tuned and Conversational Models: Foundation models such as WaveMind integrate contrastive-aligned EEG encodings with vision-language LLMs (Vicuna-1.5-7B), leveraging large-scale instruction tuning to support open-ended generation and analysis across multiple cognitive and clinical tasks (Zeng et al., 26 Sep 2025).

3. Training Strategies and Objectives

EEG2TEXT is trained via a combination of generative, discriminative, and self-supervised objectives:

• Cross-Entropy Sequence Loss: The dominant objective is next-token prediction over the text decoder output, optimized via standard cross-entropy.
• Contrastive Objectives: InfoNCE or CLIP-style contrastive losses are used to directly align EEG and text embedding spaces (Feng et al., 2023, Wang et al., 27 Feb 2024, Tao et al., 14 Sep 2024, Liu et al., 21 May 2025). Hard-positive/negative mining via curriculum learning further improves semantic calibration (Feng et al., 2023).
• Masked Modeling and Autoencoding: Pretext tasks such as masked EEG reconstruction (masked autoencoders/MAE), masked language modeling, or hybrid multi-stream architectures (CET-MAE) encourage transferable cross-modal features (Wang et al., 27 Feb 2024, Liu et al., 3 May 2024).
• Adversarial and Regularization Components: In self-supervised and domain-adversarial settings, discriminators or auxiliary classifiers further enforce subject-invariance or prevent collapse (Liu et al., 3 May 2024).

The training schedule often involves stagewise procedures: initial self-supervised or contrastive pretraining (with EEG and/or paired text), followed by supervised sequence-to-sequence fine-tuning. Typical datasets include the ZuCo corpus (word- and sentence-aligned multi-subject EEG), with careful exclusion of test sentences for generalization assessment.

4. Evaluation Protocols, Metrics, and Benchmarking Issues

A variety of metrics quantify EEG2TEXT performance:

• BLEU-N: $N$-gram precision and brevity, reported at up to BLEU-4; typical values for state-of-the-art range from 6.8%–44% on ZuCo depending on the architecture and evaluation protocol (Wang et al., 27 Feb 2024, Murad et al., 20 May 2025, Amrani et al., 2023).
• ROUGE-N/F/L: Overlap-based metrics for recall, precision, and longest common subsequence.
• BERTScore: Semantic similarity in embedding space, capturing fluency and human comprehensibility (Amrani et al., 2023, Gedawy et al., 11 Feb 2025).
• Word/Character Error Rate (WER/CER): Edit distance normalized over reference length; high for EEG2TEXT (typical WER/CER ≈ 0.68–1.10) (Murad et al., 20 May 2025, Lévy et al., 18 Feb 2025).
• Retrieval Accuracy and Semantic Classification: For semantically faithful generation, retrieval of ground-truth sentences among distractors via EEG–text embedding similarity, or zero-shot category inference from latent EEG vectors (Liu et al., 21 May 2025).

Recent critical analyses have identified severe benchmarking flaws due to widespread use of teacher-forcing at test time, which artificially inflates reported BLEU/ROUGE/WER by feeding the ground-truth previous tokens instead of the model’s own predictions. Autoregressive decoding yields a three-fold reduction in reported BLEU compared to teacher-forced evaluation, exposing genuine model limitations (Jo et al., 10 May 2024). Additionally, noise baselines—training and testing on input-matched Gaussian noise—often result in comparable metrics to real EEG, highlighting the importance of strict benchmarks separating language priors from actual EEG-driven content.

5. Empirical Results and Comparative Performance

A range of architectures and alignment strategies have been evaluated, with performance summarized as follows (generation metrics on ZuCo and related benchmarks):

Model (Year)	BLEU-1 (%)	BLEU-4 (%)	ROUGE-1 F1 (%)	WER
Baseline BART [Wang & Ji]	~40	~6.8	~22–30	0.78–1.10
R1 Translator (BART)	44.44	—	34.47	0.728
CET-MAE / E2T-PTR	—	8.99	32.61	—
C-SCL + BrainBART	39.14	—	—	0.6848
SEE (cross-modal codebook)	—	7.70	31.1	—
GLIM (semantic, no Tf)	26.0*	10.6*	—	—
EEG2Text (Multi-View)	45.2	14.1	34.2	—

(*denotes BLEU-1@multi-target variant; WER values are in (Feng et al., 2023, Murad et al., 20 May 2025, Amrani et al., 2023, Wang et al., 27 Feb 2024, Tao et al., 14 Sep 2024, Liu et al., 21 May 2025, Liu et al., 3 May 2024))

Empirical ablations consistently demonstrate that cross-modal contrastive alignment, region-based multi-view encoding, and subject-conditioning yield the largest improvements over vanilla EEG→Text sequence training. Pretraining with masked reconstruction, semantic-aware contrastive learning with curriculum, and false-negative mitigation further improve robustness and transferability. Instruction-tuned and dialog-capable LLM variants enable interpretable open-ended output in medical and visual reasoning scenarios (Zeng et al., 26 Sep 2025). Multilingual frameworks have also been introduced, albeit with lower BLEU-1 on non-phonetic languages (e.g., EEG2TEXT-CN for Chinese, BLEU-1=6.38%) (Lu et al., 1 Jun 2025).

6. Interpretability, Hallucination, and Reliability

EEG2TEXT models are especially susceptible to posterior collapse: powerful LMs can ignore the input and produce plausible outputs solely from language priors, even when provided pure noise as input (Liu et al., 21 May 2025, Jo et al., 10 May 2024). Direct alignment of EEG and text embeddings (via contrastive objectives) and rigorous baseline controls (noise input) can partially mitigate this phenomenon, but meaningful progress requires:

• Semantically grounded evaluation: Generation should be assessed for content faithfulness, not just surface similarity.
• Latent retrieval and classification tasks: Zero-shot accuracy on sentiment or relation categories, as well as EEG–text retrieval, serve as robust checks for EEG content usage.
• Noise baselining: All studies should report performance with random/non-informative EEG input as reference.
• Instruction/few-shot prompt evaluation: Incorporating domain labeling and dynamic querying can further probe neural–semantic dependencies (Zeng et al., 26 Sep 2025).

Visualization techniques—such as t-SNE on EEG–text latent embeddings and saliency maps—confirm the emergence of subject-invariant, semantically clustered representations after proper alignment (Feng et al., 2023, Rezvani et al., 9 Jul 2025).

7. Future Directions and Open Problems

Major open challenges include:

• Scaling to diverse and larger datasets: Datasets such as ZuCo remain limited in both linguistic and subject variety, restricting generalization and domain adaptation potential (Wang et al., 27 Feb 2024, Amrani et al., 2023).
• Robust semantic alignment: Improving cross-modal representation learning by leveraging self-supervised EEG pretraining, multimodal (e.g., EEG–eye-tracking) integration, and graph-based encoding of channel topology (Masry et al., 26 May 2025).
• Mitigating hallucination and LLM bias: Combining frozen LMs, contrastive objectives, real-signal benchmarks, and domain adversarial training.
• Real-time, low-latency communication: Adapting foundation models for streaming inference and inner-speech decoding remains a technical barrier (Zeng et al., 26 Sep 2025).
• Generalization across languages and modalities: Multilingual EEG2TEXT (e.g., Chinese, logographic scripts) and multimodal models are under development but require more data and task-adaptive architectures (Lu et al., 1 Jun 2025).
• Clinical deployment and ethical safeguards: Privacy, data security, and subject-aware customization underpin all translation to real-world assistive applications (Murad et al., 20 May 2025, Amrani et al., 2023).

In summary, EEG2TEXT constitutes a frontier of open-vocabulary brain signal decoding, where the interplay of deep neural architectures, cross-modal alignment, rigorous benchmarking, and careful handling of semantic/subject variability is essential for scientifically valid and clinically usable neurosemantic interfaces.