EEG-to-Text Translation Models

• EEG-to-Text translation models are defined as sequence-to-sequence pipelines that convert multichannel EEG signals into open-vocabulary text through modular stages.
• They integrate advanced signal preprocessing, multimodal feature fusion, and deep learning techniques like CNNs and Transformers to enhance language decoding accuracy.
• Emerging frameworks leverage pretraining, contrastive alignment, and autoregressive decoding to overcome challenges such as low signal-to-noise ratios and inter-subject variability.

Electroencephalography-to-Text (EEG-to-Text) translation models are a class of algorithms and neural architectures designed to decode natural language content directly from noninvasive scalp EEG signals. This domain has emerged at the intersection of brain–computer interfaces (BCIs), computational neuroscience, and natural language processing, enabling the synthesis of open-vocabulary text from brain activity recorded during reading, perception, or mental imagery. Recent advances leverage deep multimodal networks, task-specific fusion pipelines, and large pretrained LLMs (PLMs), shifting the frontier from closed-vocabulary and letter recognition to robust, subject-general natural language decoding.

1. Core Architectural Principles

EEG-to-Text frameworks operate as sequence-to-sequence pipelines that map a multichannel EEG time series, typically extracted during natural sentence reading and temporally segmented at the word or phrase level, into token sequences in an open or unbounded vocabulary.

A representative example, the ETS framework, encapsulates three distinct architectural stages (Masry et al., 26 May 2025):

• Multimodal feature encoding: Raw EEG (band-pass filtered into Δ, α, β, γ bands) and eye-tracking features (FFD, TRT, GD) are concatenated per word, processed by CNNs per band/metric, and mapped into a shared d-dimensional Transformer embedding space via learnable adapters.
• Contextual sequence modeling: The entire word/fixation sequence enters a deep Transformer encoder, yielding contextualized EEG representations sensitive to long-range semantic dependencies—critical for maintaining coherence across multi-word spans.
• Sequence-to-sequence generation: These neural embeddings are projected into the token embedding space of a pretrained autoregressive decoder (e.g., BART or T5), which is then fine-tuned to maximize conditional likelihood over open vocabulary target text.

The majority of frameworks implement variants of this modular approach, often with modifications in the transistorization of the input features, pretraining regimens, and the scope of decoder fine-tuning (Wang et al., 27 Feb 2024, Liu et al., 3 May 2024, Tao et al., 14 Sep 2024).

2. Signal Processing and Feature Extraction

Signal preprocessing is foundational in controlling for the extremely low SNR of scalp EEG. The typical sequence (Murad et al., 26 Apr 2024, Shukla et al., 17 Feb 2025):

• Artifact removal: ICA, regression against EOG/reference, or hybrid combinations to remove ocular/muscle/line noise.
• Band-pass filtering: Often 0.5–40 Hz (sometimes up to 100 Hz if high γ is of interest); notch filtering at 50/60 Hz.
• Segmentation: Word-aligned epoching based on eye-tracking fixations or fixed sliding windows.

EEG feature construction methods include:

• Frequency-domain statistics: Hilbert amplitude per band, spectral power densities, discrete wavelet coefficients.
• Spatial patterns: Channel concatenation, spatial filtering (sometimes via CSP).
• Learned spatial–temporal filters: 1D/2D CNNs over time and channel axes, often followed by downsampling or PCA.

Fusion with eye-tracking data temporally anchors neural features to denotable language events, mitigating temporal jitter and enhancing neural-text alignment (Masry et al., 26 May 2025). Multiband and multi-view architectures partition channels by putative linguistic function (e.g., Broca’s/Wernicke’s areas), allowing Transformer heads to specialize across functional topology (Liu et al., 3 May 2024).

3. Learning Paradigms: Pretraining, Alignment, and Decoding

A wide spectrum of learning objectives and pretraining strategies has evolved:

• Self-supervised masked autoencoding: EEG-only or multimodal masked autoencoders (MAE, CET-MAE) reconstruct missing epochs/spans, injecting context modeling priors and stabilizing representations across subjects/tasks (Wang et al., 27 Feb 2024, Liu et al., 3 May 2024).
• Contrastive alignment: InfoNCE-style losses match EEG and text representations in a shared latent space, either globally (sequence-paired summary embeddings) or locally (per-token), often using codebooks or cross-modal prototypes to address the semantic gap (Tao et al., 14 Sep 2024, Duan et al., 2023).
• Discrete bottlenecking: Vector quantization (VQ-VAE) discretizes EEG temporal dynamics into codex embeddings, bridging EEG’s continuous geometry with token-discrete LLMs and mitigating inter-individual neural variability (Duan et al., 2023).
• Autoregressive generation and fine-tuning: The final stage involves conditioning a PLM (BART, T5, LLaMA, MiniLM) decoder on the neural embedding sequence, optimizing cross-entropy against the surface text. Beam search is employed for text sampling at inference; teacher-forcing during training improves stability but can obscure actual decoding ability if improperly used during evaluation (Masry et al., 26 May 2025, Jo et al., 10 May 2024).

4. Evaluation Metrics, Protocols, and Benchmarks

Quantifying model performance relies on standard NLG metrics, with unique considerations for neural decoding:

Metric	Description	Purpose
BLEU-n	n-gram precision with brevity penalty	Fluency, overlap
ROUGE-N/L/F1	Recall and F1 for word/sequence overlap	Informativeness
WER/CER	Word/Character error rate	Error quantification
BERTScore	Token-embedding similarity (semantics)	Semantic accuracy
F1 (classification)	Macro-averaged precision and recall (sentiment tasks)	Discriminative tasks

Robust evaluation requires discarding teacher-forcing at inference and benchmarking against random noise inputs (randomized EEG feature matrices with identical first/second moments) to detect spurious decoding or model memorization. Empirical results demonstrate that models evaluated with teacher-forcing can achieve BLEU/ROUGE scores on noise inputs nearly equal to those achieved on real EEG, underscoring the necessity for autoregressive, no-teacher-forcing protocols and noise baselines (Jo et al., 10 May 2024).

5. Empirical Results and System Comparisons

Mainstream EEG-to-Text models consistently employ the ZuCo corpus benchmark and are increasingly evaluated in multi-modal, cross-subject settings. Key recent metrics (BLEU-4, ROUGE-1-F1):

Framework	BLEU-4	ROUGE-1-F	Salient Features
ETS (Masry et al., 26 May 2025)	20.22	36.66	CNN+Transformer fusion, eye-tracking, SOTA sentiment pipeline
DeWave (Duan et al., 2023)	8.22	30.69	VQ-VAE codex, markerless translation
E2T-PTR (Wang et al., 27 Feb 2024)	8.99	32.61	Contrastive MAE, BART interface
EEG2TEXT (Liu et al., 3 May 2024)	14.10	34.20	Multi-view transformer, self-supervised pretraining
SEE (Tao et al., 14 Sep 2024)	7.7	31.1	Cross-modal codebook, semantic matching
C-SCL (Feng et al., 2023)	18.9	39.1	Curriculum contrastive learning, subject-independence
R1 Translator (Murad et al., 20 May 2025)	N/A	34.47	BiLSTM encoder + BART decoder

ETS demonstrates consistent boosts on higher-order BLEU and F1 over transformer or BiLSTM-based pipelines. Models such as WaveMind (Zeng et al., 26 Sep 2025) leverage even larger multi-modal pretraining and instruction tuning, supporting flexible conversational generation and object/event/affect interpretation.

Zero-shot sentiment classification using a generated text pipeline markedly outperforms direct EEG classification (F1: 68.18% vs. 37.1%), illuminating the value of modular, intermediate language representations (Masry et al., 26 May 2025).

6. Open Challenges and Methodological Rigor

Despite rapid progress, several obstacles persist:

• Information bottleneck: The low intrinsic capacity of scalp EEG and nonstationarity constrain fine-grained, verbatim reconstruction of text; semantically faithful summarization is an emerging consensus on what is achievable (Liu et al., 21 May 2025).
• Inter-subject and session variability: Addressed via contrastive alignment, discrete quantization, subject-specific adaptation layers, and curriculum sampling, but not fully solved (Feng et al., 2023, Tao et al., 14 Sep 2024).
• Data scarcity: Most current datasets remain in the 1k–20k sample regime, limiting the utility of very large models; augmentation, multimodal, and pretext tasks partially compensate but large-scale open corpora remain a bottleneck (Shukla et al., 17 Feb 2025).
• Benchmarks and evaluation: Without noise-reference and autoregressive-only protocols, comparisons are confounded by inflation from teacher-forcing and memorization artifacts (Jo et al., 10 May 2024).
• Interpretability and hallucination: Models may hallucinate plausible but EEG-agnostic output if the decoder is too powerful—a phenomenon formalized as posterior collapse. This is mitigated through contrastive losses and information-regularized training (Liu et al., 21 May 2025).

7. Future Directions and Applications

Prioritized future directions include:

• Multilingual and cross-modal expansion: Emerging work, such as EEG2TEXT-CN (Lu et al., 1 Jun 2025), explores Chinese EEG-to-text models via masked/contrastive pretraining, indicating feasibility for non-English brain–text decoding.
• Instruction tuning and foundation models: New models like WaveMind (Zeng et al., 26 Sep 2025) align EEG with CLIP-like representations and train on instruction-annotated datasets supporting open-ended Q&A, object/event/affect queries, and domain adaptation.
• Real-time and clinical deployment: Architectures with efficient inference paths, adapter-based fine-tuning, and edge-suitable hardware pave the way for BCIs usable by locked-in and aphasic populations (Khushiyant, 8 Sep 2025).
• Semantic evaluation metrics: Retrieval accuracy, zero-shot property classification, and semantic completeness via LLMs are increasingly preferred over raw n-gram overlap.
• Subject/person-specific and cross-subject modeling: Curriculum semantic-aware contrastive learning, subject tokenization, and vector quantization codices offer robust handling of neural idiosyncrasies (Feng et al., 2023, Wang et al., 27 Feb 2024, Duan et al., 2023).

In summary, EEG-to-Text translation has matured from proof-of-concept word/letter recognition to state-of-the-art multilingual, multimodal, and conversational-generation frameworks explicitly grounded in neural activity. The field is converging on robust, semantically meaningful, and evaluation-sound approaches, with ongoing work targeting scale, cross-modality, and high-fidelity subject-agnostic brain-to-language interfaces (Masry et al., 26 May 2025, Liu et al., 3 May 2024, Zeng et al., 26 Sep 2025, Liu et al., 21 May 2025).