Stacked Bidirectional LSTM (BiLSTM)

• Stacked BiLSTM networks are deep recurrent architectures that use paired forward and backward LSTM layers to extract both past and future contextual information.
• Dense variants enhance gradient propagation by concatenating outputs from all previous layers, which mitigates vanishing gradients and improves training of deeper models.
• These models are applied in diverse areas including NLP, time-series forecasting, and meteorological prediction, delivering competitive performance with efficient parameter usage.

A stacked Bidirectional Long Short-Term Memory (BiLSTM) network is a deep recurrent neural architecture that arranges multiple BiLSTM layers in sequence, enabling hierarchical extraction of temporal features from sequential data. Each BiLSTM layer comprises a forward- and backward-propagating LSTM, concatenating their hidden states at each timestep to model both past and future contexts. Stacked BiLSTMs are utilized across NLP, time-series forecasting, and scientific prediction tasks, leveraging increased depth for modeling complex, multi-scale dependencies. Dense variations further extend connectivity to address vanishing-gradient challenges common in deep recurrent networks.

1. Mathematical Formulation and Architecture

A standard stacked BiLSTM with $L$ layers processes an input sequence $e_1,\dots,e_T$ as follows. For each layer $l=1,\dots,L$ and time step $t=1,\dots,T$:

$$\begin{aligned} \overrightarrow{h}^{(l)}_t &= \mathrm{LSTM}\left(\overrightarrow{h}^{(l)}_{t-1},\,x^{(l)}_t\right) \ \overleftarrow{h}^{(l)}_t &= \mathrm{LSTM}\left(\overleftarrow{h}^{(l)}_{t+1},\,x^{(l)}_t\right) \ h^{(l)}_t &= \left[\overrightarrow{h}^{(l)}_t ; \overleftarrow{h}^{(l)}_t\right] \ x_t^{(l)} &= h_t^{(l-1)}\quad (l>1), \qquad x_t^{(1)} = e_t \end{aligned}$$

Each LSTM cell uses gated recurrence: $$\begin{aligned} i_t &= \sigma(W_i x_t + U_i h_{t-1} + b_i) \ f_t &= \sigma(W_f x_t + U_f h_{t-1} + b_f) \ o_t &= \sigma(W_o x_t + U_o h_{t-1} + b_o) \ \tilde{c}_t &= \tanh(W_c x_t + U_c h_{t-1} + b_c) \ c_t &= f_t \odot c_{t-1} + i_t \odot \tilde{c}_t \ h_t &= o_t \odot \tanh(c_t) \end{aligned}$$

In densely connected stacked BiLSTM (“DC-Bi-LSTM”), each layer receives the concatenation of all previous layer outputs: $$H^{(l)}_t = \left[h^{(0)}_t; h^{(1)}_t; \ldots; h^{(l-1)}_t\right],\qquad h_t^{(0)} := e_t$$ so the recurrence becomes $$\overrightarrow{h}^{(l)}_t = \mathrm{LSTM}(\overrightarrow{h}^{(l)}_{t-1}, H^{(l)}_t)$$ and analogously for $\overleftarrow{h}^{(l)}_t$ (Ding et al., 2018).

This structure enables higher layers to directly access low-level (e.g., word embedding) representations and all intermediate features.

2. Depth, Connectivity, and Gradient Propagation

Stacked BiLSTMs extend modeling depth by chaining layers, typically propagating inputs layer-to-layer ($x^{(l)}_t = h^{(l-1)}_t$). While increased depth theoretically supports higher-level abstractions, conventional stacking can suffer from vanishing or exploding gradients due to successive nonlinearities. Dense variants like DC-Bi-LSTM inject shortcut concatenations from all preceding layers to every subsequent layer, providing more direct paths for both information and error signal flow. This mitigates degradation in both feature propagation and gradient magnitude, allowing for successful end-to-end training at greater depths ($\geq$20 layers demonstrated) (Ding et al., 2018).

Standard stacking saturates or degrades in accuracy as depth increases beyond $\approx$5 layers, while DC-Bi-LSTM continues to improve—a phenomenon empirically demonstrated in sentence classification benchmarks.

3. Parameter Efficiency and Practical Scalability

The parameter count of standard stacked BiLSTMs increases linearly in both the number of layers and per-layer width ($O(Ld^2)$ for $L$ layers of $d$ units). In densely connected architectures, narrower layers can be employed since the feature space's expressive capacity is built through concatenation. For example, a DC-Bi-LSTM with 10 layers at 100 units per direction (per layer) achieves comparable or superior performance to a standard 3-layer stack of the same width, at similar parameter budgets ($\sim$0.25M vs 0.20M) (Ding et al., 2018). DC-Bi-LSTM with 20 layers at 100 units per direction remains within 0.45M parameters.

Memory requirements grow with the dimensionality of the concatenated inputs at high layers, necessitating careful management via moderate hidden sizes or projection layers. Computational cost per step is higher in DC-Bi-LSTM due to larger concatenated vectors, but empirical results justify this for tasks demanding deep abstraction.

4. Training Setups and Hyperparameterization

Typical configuration and training regimen for stacked BiLSTMs include:

• Depth: Common configurations for practical tasks use 2–4 stacked BiLSTM layers (Vamvouras et al., 28 Aug 2025, Biswas et al., 2021, Akhter et al., 2024) but successful training up to 20 layers is reported with dense interconnections (Ding et al., 2018).
• Hidden units: 8–300 units per direction per layer; most often 100–256 in published ablations.
• Dropout: Input dropout ($\sim$0.5), inter-layer dropout ($\sim$0.3), and occasional recurrent dropout ($\sim$0.02) applied to mitigate overfitting.
• Optimization: AdaDelta (Ding et al., 2018) or Adam (Biswas et al., 2021, Akhter et al., 2024). Learning rates are selected per problem (e.g., 0.01 in cyclone prediction (Biswas et al., 2021), 0.003 in weather forecasting (Vamvouras et al., 28 Aug 2025)).
• Weight initialization: Xavier/Glorot uniform for LSTM weights, with forget-gate bias set to 1.
• Regularization: L2 weight decay ($10^{-5}$ to none), and early stopping on dev accuracy, as appropriate per deployment.

Model selection is task- and resource-specific, with dense connectivity advantageous under parameter constraints (Ding et al., 2018).

5. Applications in Sequence Modeling and Forecasting

Stacked BiLSTM architectures appear in various domains:

• Sentence classification (NLP): DC-Bi-LSTM and standard stacked BiLSTMs achieve competitive accuracies on MR, SST, SUBJ, and TREC datasets, with DC-Bi-LSTM achieving up to 84.6% on MR at 20 layers, outperforming standard stacking at the same or even reduced width (Ding et al., 2018).
• Time-series forecasting: Two-layer stacked BiLSTM is used for short-term electricity demand prediction (Dhaka) in a CNN–stacked BiLSTM hybrid, yielding MAPE 1.64%, MSE 0.015, RMSE 0.122, and MAE 0.092, outperforming shallower LSTM/Conv baselines (Akhter et al., 2024).
• Meteorological forecasting: A two-layer stacked BiLSTM with attention models 48-hour joint prediction of temperature, irradiance, and relative humidity, yielding MAEs of 1.3°C (temperature), 31 W/m2 (irradiance), and 6.7% (humidity), outperforming both numerical and ML baselines (Vamvouras et al., 28 Aug 2025).
• Meteorological extreme events: Four-layer stacked BiLSTM regresses multi-horizon cyclone wind speed, achieving MAE 1.52 at 3h and 11.92 at 72h lead, using a 7-dimensional input space with minimal dropout for regularization (Biswas et al., 2021).
• Signal fault diagnosis: While (Abdelli et al., 2022) primarily uses a single BiLSTM plus CNN for fiber fault multitask learning, the framework generalizes to stacking in more complex settings.

6. Variants, Enhancements, and When to Use

DC-Bi-LSTM, as a densely connected variant, empirically demonstrates better gradient flow, depth scalability, and parameter efficiency over standard stacking:

• Dense skip connections: Each layer consumes all preceding hidden activations (including embeddings), alleviating vanishing gradients and preserving low-level representations. At deep setting, ablation of these connections results in a 1.0–1.5% accuracy drop, confirming their significance (Ding et al., 2018).
• Attention modules: Integration with stacked BiLSTM further enhances modeling by permitting the network to selectively focus on important timesteps (e.g., attention-enhanced BiLSTM in weather forecasting) (Vamvouras et al., 28 Aug 2025).
• Efficient deep learning under budget constraints: Narrower layers with dense connectivity or stacking provide richer expressivity at similar or lower cost compared to wider, shallow architectures.

Dense stacking should be chosen when feature abstraction at multiple temporal scales is essential, especially in regimes demanding high depth and efficient gradient propagation. For shallow networks or latency-critical settings, simpler two- or three-layer stacks may suffice.

7. Empirical Results and Comparative Performance

Empirical results consistently demonstrate the superiority of stacked and densely connected BiLSTM in deep, expressive sequence modeling tasks. Key findings include:

Application	Architecture	Depth (L)	Main Result(s)	Source
Sentence classification (MR)	DC-Bi-LSTM	20	84.6% accuracy, surpasses parametric-matched standard BiLSTM	(Ding et al., 2018)
Electricity load forecasting	CNN + stacked BiLSTM	2	MAPE 1.64%, best among LSTM/CNN baselines	(Akhter et al., 2024)
48-hour weather forecasts	Stacked BiLSTM + Attn	2	MAEs: 1.3°C (temp), 31 W/m2 (irr), 6.7% (RH)	(Vamvouras et al., 28 Aug 2025)
Cyclone intensity (MSWS)	Stacked BiLSTM	4	MAE 1.52 @3h, scaling monotonic to 11.92 @72h horizon	(Biswas et al., 2021)

At shallow depths ($\leq$3), standard stacked BiLSTM and DC-Bi-LSTM yield equivalent accuracy, but deeper DC-Bi-LSTM models improve steadily where conventional stacking plateaus.

• "Densely Connected Bidirectional LSTM with Applications to Sentence Classification" (Ding et al., 2018)
• "Short-Term Electricity Demand Forecasting of Dhaka City Using CNN with Stacked BiLSTM" (Akhter et al., 2024)
• "An Explainable, Attention-Enhanced, Bidirectional Long Short-Term Memory Neural Network for Joint 48-Hour Forecasting of Temperature, Irradiance, and Relative Humidity" (Vamvouras et al., 28 Aug 2025)
• "Intensity Prediction of Tropical Cyclones using Long Short-Term Memory Network" (Biswas et al., 2021)
• "A BiLSTM-CNN based Multitask Learning Approach for Fiber Fault Diagnosis" (Abdelli et al., 2022)