Enhanced Transformer-CNN-BiLSTM Architecture

• Enhanced Transformer-CNN-BiLSTM is a hybrid model that combines CNNs, BiLSTMs, and Transformers to extract local, temporal, and global features from complex signals.
• It employs convolutional layers for hierarchical pattern extraction, BiLSTM for bidirectional temporal modeling, and Transformer self-attention for adaptive integration across features and time.
• Empirical studies show significant accuracy improvements and reduced overfitting in tasks such as EEG-based emotion classification and sleep-stage labeling.

The Enhanced Transformer-CNN-BiLSTM architecture is a hybrid deep neural model integrating convolutional neural networks (CNNs), bidirectional long short-term memory (BiLSTM) networks, and Transformer-based self-attention mechanisms. Developed for complex sequence and multivariate signal domains, this architecture supports feature extraction, bidirectional temporal modeling, and adaptive attention across time and feature channels, yielding state-of-the-art performance in biomedical pattern recognition tasks such as EEG-based emotion classification and sleep-stage labeling (Karim et al., 6 Feb 2026, Sadik et al., 2023).

1. Model Definition and Theoretical Motivation

The Enhanced Transformer-CNN-BiLSTM framework is engineered to address high-dimensional, temporally structured, and highly correlated input signals. It combines:

• Deep convolutional layers to capture local spatial and frequency patterns,
• Stacked BiLSTM layers for modeling bidirectional temporal dependencies,
• Multi-head self-attention mechanisms to adaptively integrate information across features and time.

Such architectural integration exploits complementary inductive biases: CNNs are efficient at extracting hierarchical local patterns, BiLSTMs model dependencies bidirectionally over sequences (crucial for time series and signals), and Transformers provide global context and selective weighting via attention.

This approach has proven effective on high-dimensional biological inputs (e.g., EEG, ECG, histopathological images), delivering marked improvements over unihybrid (CNN or LSTM-only) and shallow pipeline baselines (Karim et al., 6 Feb 2026, Sadik et al., 2023, Dubey et al., 2019).

2. Architectural Overview and Layer Composition

The canonical Enhanced Transformer-CNN-BiLSTM architecture consists of the following sequential components:

1. Input preprocessing and normalization Input feature vectors or signals undergo z-score or min-max normalization to stabilize training.
2. Convolutional feature extractor
   • Stacks of 1D or 2D convolutional layers, often organized in residual or DenseNet-style blocks.
   • E.g., for EEG, three residual blocks (each with two Conv1D layers, kernel size 3, 64 channels), yielding an intermediate feature map of shape $[F \times T]$ (Karim et al., 6 Feb 2026).
   • In histopathological imaging, the first 100 layers of Inception-V3 extract a 2048-dimensional embedding (Dubey et al., 2019).
3. Bidirectional LSTM
   • The CNN output is reshaped to a sequence fed into one or more stacked BiLSTM layers.
   • Each BiLSTM with hidden size $h$ per direction, producing hidden states $h_t = [\overrightarrow{h}_t ; \overleftarrow{h}_t]$.
   • For EEG and sleep-stage classification, two BiLSTM layers with $h=128$ per direction are typical (Karim et al., 6 Feb 2026, Sadik et al., 2023).
4. Transformer-based self-attention module
   • One or more (often dual) multi-head attention layers.
   • Computes $Q$, $K$, $V$ projections per head, attending over the sequence dimension.
   • For EEG, primary block uses 16 heads, followed by a secondary block with 8 heads (Karim et al., 6 Feb 2026).
   • In sleep-stage classification, three Transformer encoder layers are interleaved prior to BiLSTM (Sadik et al., 2023).
5. Pooling and classification head
   • Dual pooling (global avg $\oplus$ global max) to aggregate sequence/feature outputs.
   • Two dense layers (with dropout, ReLU or tanh), final softmax for target class probabilities.
   • Losses include cross-entropy with label smoothing ($\varepsilon=0.1$), AdamW/Adam optimization, and L2 or weight decay (Karim et al., 6 Feb 2026, Dubey et al., 2019, Sadik et al., 2023).

3. Mathematical Formulation

CNN Block: Each residual convolutional block applies: $$\mathbf{H}^{(b,\ell)} = \mathrm{ReLU}(\mathbf{W}^{(b,\ell)} * \mathbf{H}^{(b,\ell-1)} + \mathbf{b}^{(b,\ell)})$$ with residual connection after two sublayers: $$\mathbf{H}^{(b,\mathrm{out})} = \mathbf{H}^{(b,0)} + \mathbf{H}^{(b,2)}$$ as in (Karim et al., 6 Feb 2026).

BiLSTM Block: For input sequence $\{x_t\}$, LSTM cell update per direction: $$\begin{aligned} i_t &= \sigma(W_i x_t + U_i h_{t-1} + b_i) \ f_t &= \sigma(W_f x_t + U_f h_{t-1} + b_f) \ o_t &= \sigma(W_o x_t + U_o h_{t-1} + b_o) \ \tilde c_t &= \tanh(W_c x_t + U_c h_{t-1} + b_c) \ c_t &= f_t \odot c_{t-1} + i_t \odot \tilde c_t \ h_t &= o_t \odot \tanh(c_t) \end{aligned}$$ Stacked BiLSTM outputs $$h_t^\mathrm{bi} = [\overrightarrow{h}_t ; \overleftarrow{h}_t]$$.

Transformer Self-Attention:

For $H \in \mathbb{R}^{T \times d}$: $$\begin{aligned} Q &= HW^Q, \quad K = HW^K, \quad V = HW^V \ \text{Scores} &= QK^\top / \sqrt{d_k} \ A &= \mathrm{softmax}(\text{Scores}) \ \text{Output} &= AV \end{aligned}$$ In multi-head attention, outputs are concatenated then projected.

Pooling and Classifier:

After sequence aggregation: $$\mathbf{u} = \mathrm{ReLU}(W_1 \mathbf{h} + b_1), \quad \mathbf{u}' = \mathrm{Dropout}_{p}(\mathbf{u})$$

$$\mathbf{z} = W_2 \mathbf{u}' + b_2, \quad \hat{y}_c = \frac{\exp(z_c)}{\sum_k \exp(z_k)}$$

4. Training, Regularization, and Loss Schemes

Training protocols employ several strategies:

• Optimization: Adam or AdamW optimizers, with initial learning rates (e.g., $1 \times 10^{-3}$), decayed via cosine annealing or fixed schedule (Karim et al., 6 Feb 2026, Sadik et al., 2023, Dubey et al., 2019).
• Regularization: Dropout layers (typ. $p=0.3$–$0.5$ in fully connected/LSTM), L2 weight decay ($\lambda=1\text{e}{-4}$), and label smoothing ($\epsilon=0.1$) on cross-entropy loss.
• Early Stopping: Training halts if validation loss stagnates over a set epoch window (e.g., 30 epochs).
• Data Augmentation: For EEG, additive Gaussian noise and random scaling; for images, flips, rotations, and zooming (Karim et al., 6 Feb 2026, Dubey et al., 2019).
• Multi-loss Supervision: Some variants (e.g., for sleep-stage recognition) combine cross-entropy, contrastive, and KL divergence losses: $$\mathcal{L}_\text{total} = \mathcal{L}_\text{CE} + \alpha \mathcal{L}_\text{cont} + \beta \mathcal{L}_\text{KL}$$ with $\alpha$ and $\beta$ set as in (Sadik et al., 2023).

5. Empirical Performance and Comparative Results

Results across EEG and medical imaging domains show the architecture delivers state-of-the-art accuracy with minimal overfitting:

Model	Application	Validation Acc. (%)	Overfitting Gap (%)	Notable Features	Study
Enhanced Transformer-CNN-BiLSTM	EEG Emotion	99.19 ± 0.6	0.56	SHAP, dual attention, feature ablation	(Karim et al., 6 Feb 2026)
DenseRTSleep-II (Transformer-CNN-BiLSTM)	Sleep Stage	79.16	—	Multi-loss, DenseNet blocks, 3-head attention	(Sadik et al., 2023)
Inception-V3 + BiLSTM + Self-Attn.	Cardiac Image	93.10 test	—	Tanh activations, seq. flattening, single attn.	(Dubey et al., 2019)

Across all settings, hybrid models integrating multi-head attention mechanisms with convolutional and bidirectional temporal modeling consistently outperform non-attentional or unidirectional recurrent baselines, as confirmed by statistical significance tests (Wilcoxon, Friedman, $p < 0.01$) (Karim et al., 6 Feb 2026).

6. Interpretability, Feature Analysis, and Applicability

Feature attribution and ablation studies in EEG applications reveal:

• Covariance-based features (inter-channel EEG relationships) drive the largest gains in discriminative performance.
• Ablating covariance reduces accuracy by $15.3\%$; SHAP and attention weights confirm their primacy (Karim et al., 6 Feb 2026).
• Attention weight matrices concentrate on indices corresponding to inter-channel relationships, indicating the model adaptively focuses on feature subsets most relevant for the task.

The implication is that the architecture is particularly well suited to domains where complex inter-feature relationships demand both local and global adaptive weighting, such as multichannel biosignals, structured medical images, and temporally resolved sensor data.

7. Extensions, Variants, and Future Directions

Variants include:

• Increased attention depth (dual or triple attention, or Transformer-encoder stacks) (Karim et al., 6 Feb 2026, Sadik et al., 2023).
• Weighted multi-loss schemes for improved generalization, as in DenseRTSleep-II for sleep scoring (Sadik et al., 2023).
• Integration with DenseNet-style convolutional blocks and auxiliary positional encoding for domain-specific enhancements (Sadik et al., 2023).
• Application-specific regularization, such as EEG-appropriate data augmentation and advanced early stopping.

A plausible implication is that hybrid Transformer-CNN-BiLSTM architectures will continue to dominate high-dimensional time series and structured data analysis, especially as interpretability and generalization are prioritized in clinical and scientific applications. Further, the empirical success across modalities suggests the architecture's inductive biases are well matched to heterogeneous, complex data environments.