Dual Transformer-Based Encoders

• Dual transformer-based encoders are advanced neural architectures that use separate transformer branches to model temporal and frequency data for robust signal representation.
• They integrate multi-task objectives, including classification, contrastive, and reconstruction losses, to improve alignment between EEG and text modalities.
• Empirical results show that these models enhance EEG-to-text generation and clinical monitoring, outperforming conventional single-stream approaches.

Dual transformer-based encoders constitute advanced neural architectures designed to jointly model heterogeneous or complementary data modalities—commonly temporal and spectral components—by parallel transformer branches that encode distinct dependency structures, and then fuse them via an adaptive mechanism. These frameworks have recently demonstrated substantial empirical and representational gains in structured EEG-text tasks and clinical neurocritical care applications, facilitating more faithful and descriptive alignments than conventional classification-based pipelines (Samanta et al., 2 Jan 2026).

1. Architectural Overview

A dual transformer-based encoder instantiates two independent transformer branches, each receiving a modality-specific input derived from shared raw data. For example, Wave2Word (Samanta et al., 2 Jan 2026) processes continuous EEG by projecting, post-STFT, into two representations: temporal (sequence of time slices) and frequency-centric (sequence of frequency slices). Each view is linearly projected and tokenized, then passed through a dedicated stack of transformer encoders, thereby modelling:

• Temporal dependencies: dynamic, sequential transitions and nonstationarities in EEG.
• Frequency dependencies: stationary oscillatory bands, rhythmic and periodic phenomena.

Outputs from each branch are mean-pooled to obtain summary vectors $h_t$ (temporal) and $h_f$ (frequency), which are then concatenated. An adaptive gating layer computes normalized weights $(\mu_t, \mu_f)$ by softmax over the concatenated vector, generating the final fused EEG embedding $$h_{\mathrm{eeg}} = [\mu_t \, h_t ; \mu_f \, h_f] \in \mathbb{R}^{2d}$$.

Branch	Input Representation	Output Summary
Temporal	Time-slice tokens (post-STFT)	$h_t$
Frequency	Freq-slice tokens (post-STFT)	$h_f$
Adaptive gating	$[h_t; h_f]$ (concatenation)	$h_{\mathrm{eeg}}$

This structure allows precise modelling of modality-specific structure while enabling shared downstream supervision and constraints.

2. Model Integration and Multi-Task Objectives

Dual transformer-based encoders are typically embedded within multi-task learning frameworks involving:

• Classification loss ($\mathcal{L}_\mathrm{cls}$): e.g., softmax-MLP applied to $h_{\mathrm{eeg}}$ for supervised event or pattern detection.
• Contrastive alignment loss ($\mathcal{L}_\mathrm{con}$): normalized CLIP-style objective aligning $h_{\mathrm{eeg}}$ and a text encoder output $h_{\mathrm{text}}$ (e.g., Bio_ClinicalBERT over ground-truth templates).
• EEG-conditioned text reconstruction loss ($\mathcal{L}_\mathrm{rec}$): negative log-likelihood computed by an autoregressive transformer decoder that predicts tokenized clinical language or unconstrained text, conditioned on $h_{\mathrm{eeg}}$ as a global prefix.

The complete loss function is expressed as:

$$\mathcal{L} = \lambda_1 \mathcal{L}_\mathrm{cls} + \lambda_2 \mathcal{L}_\mathrm{con} + \lambda_3 \mathcal{L}_\mathrm{rec}$$

where weights $\lambda_i > 0$ are determined via uncertainty-based joint optimization (Samanta et al., 2 Jan 2026).

In CET-MAE and E2T-PTR (Wang et al., 2024), a structurally related multi-stream encoder is applied for cross-modality self-supervised learning and intra-modality feature reconstruction, with the EEG-conditioned text loss directly driving token-level prediction from EEG embeddings through a transformer decoder (BART).

3. Representational Properties and Fusion Dynamics

Dual transformer-based encoders are designed to preserve complementary statistical regularities and semantic distinctions between views. The gating mechanism ensures that the fused embedding $h_{\mathrm{eeg}}$ adaptively prioritizes either temporal or frequency information according to input salience, potentially capturing subtle signal characteristics or clinical patterns missed by single-branch solutions.

In practice, $h_{\mathrm{eeg}}$ is sent in parallel to:

• Classification heads for prediction tasks.
• Contrastive heads for embedding alignment.
• Autoregressive text decoders for sequence generation.

This separation allows downstream decoders to access a condensed, modality-aware representation, improving cross-modal retrieval and descriptive fidelity (Samanta et al., 2 Jan 2026).

4. Training Procedures and Optimization Strategies

Wave2Word and related models employ AdamW optimization (fixed weight decay), automatic mixed precision (e.g., NVIDIA H100), and fixed schedules without warm restarts (Samanta et al., 2 Jan 2026). Loss weights $\{\lambda_i\}$ are initialized and jointly updated using uncertainty-based weighting (as in Kendall et al., 2018). Spectrogram augmentations—time and frequency masking, noise injection, random magnitude scaling—are applied during training to regularize representations jointly with all objectives.

No curriculum learning or staged unfreezing is reported: all encoder and decoder modules are updated concurrently during training and fine-tuning (Wang et al., 2024).

5. Empirical Impact and Ablation Findings

Experimental ablation demonstrates the crucial role of dual encoding and the EEG-conditioned text reconstruction objective:

• In Wave2Word, removing $\mathcal{L}_\mathrm{rec}$ reduces six-way accuracy by 0.54 points and cross-modal Recall@10 drops from 0.3390 to 0.3210; parameter count halves but representation quality sharply degrades, and convergence becomes unstable (Samanta et al., 2 Jan 2026).
• In CET-MAE/E2T-PTR, omitting the masked text objective or pre-training yields a drop in BLEU-4 by 0.5–1.0 points. The full protocol achieves an 8.3% relative gain in ROUGE-1 F1 and 32.2% in BLEU-4 over baseline (Wang et al., 2024).

A plausible implication is that classification objectives alone cannot ensure that the fused representations encode the semantic content required for high-fidelity text generation or cross-modal retrieval. The reconstruction loss acts as a consistency regulator and content-preserving constraint, producing embeddings that are both discriminative and descriptively faithful.

6. Applications and Extensions

Dual transformer-based encoders have enabled state-of-the-art performance in:

• Non-invasive EEG-to-language BCI applications: continuous and naturalistic text generation from scalp EEG (Wang et al., 2024).
• Neurocritical care monitoring: automated, clinically structured summarization and retrieval of EEG events with expert-level descriptive precision (Samanta et al., 2 Jan 2026).
• Semantic speech decoding from iEEG: transfer learning and unconstrained language generation, as in Neuro2Semantic (Shams et al., 31 May 2025), though Neuro2Semantic employs LSTM–transformer adapters rather than dual encoding.

Their design supports flexible integration of multiple modalities, self-supervised and supervised objectives, and robust representation learning, with empirical validation across controlled benchmarks (e.g., ZuCo, clinical neurocritical care datasets) (Wang et al., 2024, Samanta et al., 2 Jan 2026). Dual transformer architectures can be extended with advanced gating, fusion, and regularization strategies to further improve generalization and descriptive power.

7. Contextual Significance and Future Directions

The adoption of dual transformer-based encoders marks a shift from accuracy-centric pipelines towards representation learning frameworks that tightly integrate cross-modal alignment and descriptive generation. The consistent empirical evidence indicates that such architectures substantially outperform single-stream and classification-only models in both retrieval and language reconstruction tasks.

Future research may focus on:

• End-to-end trainable fusion mechanisms that generalize adaptive gating.
• Self-supervised pre-training protocols scaling to larger, multi-modal clinical corpora.
• Generalization to other biomedical signals or multi-sensor arrays, leveraging dual-stream approaches for richly annotated data.
• Integration with LLMs and external knowledge for context-aware generation (Wang et al., 2024).

This suggests dual transformer-based encoding presents a principled pathway for bridging raw sensory data and structured semantic understanding in clinical and neurotechnological domains.