EEG2TEXT: Translating Brain Signals to Text

• EEG2TEXT is a technique that decodes scalp EEG signals into free-form text, integrating neuroscience, signal processing, and deep learning.
• Modern systems employ encoder–decoder pipelines with transformer architectures and pretrained models like BART and T5 for cross-modal alignment.
• Evaluations highlight challenges in signal fidelity and subject variability, prompting innovations in contrastive learning and personalized adaptations.

Electroencephalography-to-Text (EEG2TEXT) refers to the task of decoding non-invasive scalp EEG signals directly into free-form, open-vocabulary natural language. This area sits at the intersection of neuroscience, signal processing, and natural language generation, aiming to enable real-time brain-computer interfaces capable of supporting communication for individuals with severe motor or speech impairments. Recent advances in deep learning and LLMs have driven rapid innovation in EEG2TEXT, yet major challenges in signal fidelity, semantic grounding, and reliable evaluation remain.

1. Model Architectures and Training Objectives

Modern EEG2TEXT systems predominantly employ encoder–decoder pipelines, with raw or preprocessed EEG fed into a deep encoder whose output is mapped to the latent space of a pretrained LLM decoder such as BART, PEGASUS, or T5. The core architectural elements are:

• EEG Encoder: Typically a stack of L=6 Transformer layers with multi-head self-attention (e.g., 8 heads per layer) processes word-level EEG feature vectors in ℝ^840, generated by Hilbert transforms in multiple frequency bands (Jo et al., 10 May 2024). Alternative encoders include bidirectional LSTM, CNN-RNN hybrids, and multi-view convolutional transformers partitioning electrodes by anatomical region (Liu et al., 3 May 2024, Murad et al., 20 May 2025).
• Projection and Alignment: The encoder output is linearly projected to the embedding space required by the LLM, supporting cross-modal alignment of EEG representations to NLP tokens (Amrani et al., 2023, Gedawy et al., 11 Feb 2025).
• LLM Decoder: Pretrained autoregressive decoders (e.g., BART, T5) generate open-vocabulary tokens, attending to the encoded EEG context (Wang et al., 2021).
• Training Objective: Supervised sequence-to-sequence learning with cross-entropy reconstruction loss is standard:

$$L(E, S; \theta) = -\sum_{t=1}^N \log p_\theta(y_t \mid y_{<t}, E)$$

where teacher-forcing is used during training, i.e., the ground-truth previous token is always fed as input to the decoder.

Several works have investigated auxiliary objectives, including mean-squared error alignment between EEG and pretrained LM embeddings (Amrani et al., 2023, Gedawy et al., 11 Feb 2025), multi-task learning, and strongly regularized contrastive alignment to promote cross-modal semantic fidelity (Feng et al., 2023, Wang et al., 27 Feb 2024, Tao et al., 14 Sep 2024, Liu et al., 21 May 2025).

2. Evaluation Protocols, Benchmarks, and Identified Limitations

Assessment of EEG2TEXT models is highly sensitive to evaluation methodology. The critical findings of Jo et al. demonstrate that:

• Implicit Teacher Forcing: Use of teacher-forcing at inference time (i.e., feeding ground-truth tokens during evaluation) leads to artificial inflation of text-generation metrics. Specifically, BLEU-1 increases by more than threefold compared to free-running inference, where the model must autoregressively generate each token based only on its own previous outputs (Jo et al., 10 May 2024).
• Noise Baseline Benchmark: Evaluating models on pure white Gaussian noise (e_t ∼ ℵ(0, I_840)) as EEG input yields metrics indistinguishable from those achieved with real EEG signals. This exposes models that actually ignore the input modality and merely parrot label distributions learned during training.
• Recommended Best Practices: Always report full metric breakdowns (BLEU-N, ROUGE-1, WER) for both EEG and noise baselines, avoid teacher-forcing at test time, and release full code/configuration for reproducibility.

The field has not yet standardized on challenge benchmarks, but the vast majority of work is conducted on the ZuCo corpus (versions 1.0, 2.0), which provides sentence-aligned, multi-subject, word-level EEG and eye-tracking data (Jo et al., 10 May 2024, Amrani et al., 2023, Gedawy et al., 11 Feb 2025, Murad et al., 20 May 2025).

3. Subject Variability and Personalization

Inter-individual differences in EEG patterns create substantial domain gaps that degrade generalization. State-of-the-art models address this by incorporating:

• Subject-Specific Layers: A per-subject learnable vector or transformation is added at the encoding stage, shifting the EEG latent representation closer to subject-independent, semantic structure. Ablation studies show 1–1.5 point BLEU/ROUGE improvement via this mechanism (Amrani et al., 2023, Gedawy et al., 11 Feb 2025).
• Mixed-Subject and Leave-One-Subject-Out Evaluation: Curriculum semantic-aware contrastive learning (C-SCL) demonstrates robust generalization even in zero-shot settings, transforming the encoder space from subject-driven clusters to semantic-driven clusters (Feng et al., 2023).
• Best Practices: Collect large, heterogeneous subject pools, apply meta-learning for rapid adaptation, and consider unsupervised pre-training on subject-agnostic EEG features (Amrani et al., 2023, Gedawy et al., 11 Feb 2025).

4. Integration of Contrastive and Self-Supervised Learning

Given EEG–text alignment is confounded by small datasets and modality mismatch, leading models employ contrastive and/or self-supervised pre-training regimes:

• Contrastive Objectives: InfoNCE or margin-based losses are used to pull together EEG representations of the same sentence across subjects and push apart unrelated sentences (Feng et al., 2023, Wang et al., 27 Feb 2024, Tao et al., 14 Sep 2024, Liu et al., 21 May 2025). This yields improvements of 1–2 points in BLEU/ROUGE and ∼1–2% absolute reduction in WER.
• Multi-Modal Masked Autoencoding: Frameworks such as CET-MAE combine intra-modality masked token reconstruction with inter-modality contrastive alignment, resulting in robust, transferable EEG embeddings with notable BLEU/ROUGE gains over baselines (+8.34% ROUGE-1 F1, +32.2% BLEU-4) (Wang et al., 27 Feb 2024).
• Semantic Matching and Codebooks: Cross-modal codebooks and semantic-matching losses, as in the SEE model, mitigate the domain gap between continuous EEG and discrete text, improving statistical coverage and semantic robustness (Tao et al., 14 Sep 2024).

5. Performance Metrics, Empirical Results, and Hallucination Analysis

Model evaluation employs several complementary metrics:

• BLEU-N: Geometric mean of clipped n-gram precisions (BLEU-1 through BLEU-4), subject to brevity penalty.
• ROUGE-1 and ROUGE-L: Unigram overlap and longest common subsequence F1, quantifying recall and fluency.
• Word Error Rate (WER): $(S+D+I)/N$ quantifies cumulative word-level substitutions, deletions, insertions.
• BERTScore: Fuzzy, embedding-based measure of semantic overlap, shown to be better aligned with human intuition than n-gram metrics (Amrani et al., 2023).

Key empirical findings include:

Model	BLEU-1	BLEU-4	ROUGE-1 F	WER	Dataset
BART (w/tf, noise)	46%	21%	n/a	n/a	ZuCo v1.0/v2.0
EEG2Text (free running)	13–17%	<7%	~30%	>68%	ZuCo v1.0/v2.0
CET-MAE + E2T-PTR	~44%	8.99%	32.6%	±0.5%	ZuCo (3/5-task)
ETS (BART)	43.4%	20.2%	36.7%	—	ZuCo v1.0/v2.0
R1 Translator	—	—	34.5–38%	0.7280	ZuCo
C-SCL (BrainBART-Large)	35.9%	18.9%	39.1%	68.5%	ZuCo (test)
GLIM	26.0%	10.6%	12.3%	—	ZuCo, paraphrase

With correct free-running evaluation, modern LLM-based decoders such as BART or T5 exhibit only marginal discriminability between real EEG and noise unless bolstered by contrastive learning and signal-specific ablations (Jo et al., 10 May 2024). Models without such regularization tend to hallucinate plausible language independent of EEG input (posterior collapse). For example, GLIM imposes tight information bottlenecks and joint contrastive–generative objectives, reducing but not eliminating hallucination as measured by large BLEU/WER drop when inputting noise (Liu et al., 21 May 2025).

6. Key Challenges, Limitations, and Future Directions

Major unresolved issues in EEG2TEXT research include:

• Signal-to-Noise Limitation: Non-invasive EEG is fundamentally limited by poor SNR and spatial/temporal mixing, yielding far lower scores than speech-based or invasive BCI approaches (Lamprou et al., 10 Jan 2025).
• Dataset Scale and Diversity: Available corpora are orders of magnitude smaller than standard NLP datasets. This restricts model capacity, limits generalization, and impedes downstream evaluation (Gedawy et al., 11 Feb 2025).
• Evaluation Rigor: Teacher-forcing and lack of noise baselines have historically led to overestimation of model ability. Adoption of stricter evaluation protocols and standardized metrics is needed.
• Interpretability and Semantic Fidelity: Recent architectures such as GLIM and SEE seek to align EEG–text spaces for improved semantic interpretability and retrieval performance, but robust, evidence-based semantic grounding is not guaranteed (Liu et al., 21 May 2025, Tao et al., 14 Sep 2024).
• Personalization and Calibration: Domain adaptation, meta-learning, and prompt-based personalization strategies offer a path toward rapidly adaptable BCI systems (Amrani et al., 2023, Gedawy et al., 11 Feb 2025).
• Integration with Other Modalities: Eye-tracking, EMG, or multimodal fusion may yield substantial improvements in both fluency and semantic accuracy (Masry et al., 26 May 2025).

Recommendations for future research include prioritizing signal-specific encoder pre-training, integrating strong language priors via frozen/fine-tuned LLMs, robust contrastive/semi-supervised training, open dataset release, and transparent, reproducible code sharing (Jo et al., 10 May 2024, Amrani et al., 2023, Wang et al., 27 Feb 2024, Liu et al., 3 May 2024, Tao et al., 14 Sep 2024, Liu et al., 21 May 2025).

7. Impact, Application Domains, and Ethical Considerations

EEG2TEXT research holds promise for next-generation assistive communication technology, especially for users with locked-in syndrome or ALS. Despite technical challenges, current systems already demonstrate modular, real-time pipelines for text entry via scalably deployed EEG hardware (Murad et al., 26 Apr 2024, Omeiza et al., 2018, Gedawy et al., 11 Feb 2025). Advances in model efficiency, hardware miniaturization, and signal fidelity are likely to drive clinical and commercial impact.

However, privacy and interpretability concerns must be paramount. The potential for downstream misuse (e.g., unauthorized "mind-reading") necessitates embedded encryption, federated learning, and clear ethical guidelines for data acquisition, storage, and inference (Murad et al., 26 Apr 2024). Rigorous, open evaluation protocols, and human-in-the-loop validation, are essential to ensure these technologies are deployed fairly and safely.