TRANS-BLSTM: Hybrid Transformer-BLSTM Model

• TRANS-BLSTM is a neural architecture that combines Transformer self-attention with BLSTM recurrence to capture both global and sequential context.
• It interleaves BLSTM sublayers within Transformer encoder blocks, maintaining training stability through residual connections and LayerNorm.
• Empirical results on SQuAD and GLUE benchmarks show significant improvements over BERT, underscoring the benefits of architectural enrichment.

TRANS-BLSTM is a neural architecture for language understanding that interleaves Bidirectional Long Short-Term Memory (BLSTM) modules within the block structure of the Transformer encoder. Designed as a unification of self-attentive and recurrent modeling, TRANS-BLSTM simultaneously leverages the global dependency capture of multi-head self-attention with the sequential context modeling of BLSTM units. Empirical evaluations demonstrate that this architecture outperforms strong BERT baselines on both SQuAD 1.1 and GLUE benchmarks, achieving an F1 score of 94.01% on SQuAD 1.1 development, comparable to prevailing state-of-the-art models (Huang et al., 2020).

1. Architectural Overview

At its core, TRANS-BLSTM modifies each Transformer encoder block by integrating a BLSTM sublayer in parallel with the standard feed-forward network (FFN). This design yields a composite block in which each input sequence $$X^{(\ell)} = [x_1, ..., x_n] \in \mathbb{R}^{n \times H}$$ passes through the following sublayers in each layer $\ell$:

1. Multi-Head Self-Attention (MHSA):
   • Computes queries, keys, and values for each head:

   $$Q_i = X^{(\ell)} W_i^Q, \quad K_i = X^{(\ell)} W_i^K, \quad V_i = X^{(\ell)} W_i^V$$

• Scaled dot-product attention per head:

  $$\text{Attention}_i(X) = \text{softmax}\left(\frac{Q_i K_i^T}{\sqrt{d_k}}\right) V_i$$

• Outputs of all heads are concatenated and projected.
• Residual connection and layer normalization:

  $$A^{(\ell)} = \text{LayerNorm}\left(X^{(\ell)} + MHSA\left(X^{(\ell)}\right)\right)$$

1. Feed-Forward Network (FFN):
   • Position-wise MLP with ReLU:

   $FFN(z) = \max(0,\, z W_1 + b_1 ) W_2 + b_2$

• Where $W_1 \in \mathbb{R}^{H \times D}$, $W_2 \in \mathbb{R}^{D \times H}$, $D=4H$.

1. Bidirectional LSTM Sublayer:
   • Given $A^{(\ell)} = [a_1, ..., a_n]$, for $t=1,\dots,n$:

   $\vec{h}_t = LSTM_f(a_t, \vec{h}_{t-1}),\quad \cev{h}_t = LSTM_b(a_t, \cev{h}_{t+1})$

• Concatenate forward and backward states:

  $H_t = [\vec{h}_t; \cev{h}_t],\;\; H_t \in \mathbb{R}^{2H_{BL}}$

• If $2H_{BL} \ne H$, linear projection $P \in \mathbb{R}^{H \times 2H_{BL}}$ is applied: $L_t = P H_t$.

1. Final Combination:
   • The FFN and BLSTM outputs are summed:

   $$Y^{(\ell)} = \text{LayerNorm}\left( A^{(\ell)} + FFN(A^{(\ell)}) + L \right)$$

• $Y^{(\ell)}$ is fed as input $X^{(\ell+1)}$ to the next block.

This block structure is repeated for $N$ layers (12 or 24 in BASE and LARGE variants, respectively). The resultant model nearly doubles the parameter count compared to pure Transformers, but maintains training stability owing to the residual and LayerNorm scaffolding.

2. Training Corpus, Tokenization, and Objectives

TRANS-BLSTM adopts the same pre-training corpora and tokenization as BERT. The corpora include:

• BooksCorpus (800M words)
• English Wikipedia (2.5B words)

WordPiece tokenization with a vocabulary of 30,000 is used, and whole-word masking is applied to 15% of input tokens. Sequence length is capped at 256 tokens.

The model jointly optimizes two objectives:

• Masked LLM (MLM): Predict 15% randomly masked tokens using a softmax over the vocabulary.
• Next Sentence Prediction (NSP): Predict whether two input sentences are contiguous or randomly paired (50/50 split).

3. Implementation Hyperparameters

Key implementation and optimization details are summarized:

Variant	Layers $N$	Hidden Size $H$	Attention Heads $h$	BLSTM Units	Dropout	Batch Size	Optimizer	Learning Rate
BASE	12	768	12	384 or 768	0.1	256	Adam	1e-4
LARGE	24	1,024	16	512 or 1,024	0.1	256	Adam	1e-4

• FFN inner size is $D = 4H$.
• In TRANS-BLSTM-SMALL, BLSTM units per direction are $H/2$.
• Dropout follows post-attention and post-FFN, as in BERT.
• Optimization uses Adam ($\beta_1=0.9, \beta_2=0.999, \epsilon=10^{-6}$).
• Pre-training is conducted for roughly 1 million steps on 8×V100 GPUs.
• Fine-tuning for SQuAD uses $lr=3e-5$, batch size 12, 2 epochs; for GLUE, $lr \in \{2e-5, ..., 5e-5\}$, batch size 32, 3 epochs, with random restarts.

4. Experimental Protocol and Baselines

Experiments are performed on two families of tasks:

• GLUE Benchmark: MNLI, QQP, QNLI, SST-2, CoLA, STS-B, MRPC, RTE. (WNLI excluded)
• SQuAD v1.1: 100K question–answer pairs.

Performance is assessed via accuracy (MNLI, QNLI, SST-2, CoLA, RTE), F1 (QQP, MRPC), Spearman $\rho$ (STS-B), and exact match/F1 (SQuAD).

Baselines include:

• BERT-base/large (re-implemented with whole-word masking)
• BLSTM-only (12 LSTM layers, $H$ units per direction) to ablate the Transformer component.

Variants for ablation include two BLSTM–Transformer fusion designs; the “parallel” version (TRANS-BLSTM-2) shows near-identical performance to its alternative and is reported for simplicity. Inclusion of BLSTM layers in the SQuAD decoder is assessed for additional gain.

5. Empirical Results

TRANS-BLSTM consistently outperforms both BERT and BLSTM-only baselines:

Masked Language Modeling Loss

Both TRANS-BLSTM-SMALL and full TRANS-BLSTM converge to lower MLM+NSP pre-training losses than the vanilla Transformer, evidencing increased capacity.

SQuAD v1.1 (Dev Set) F1

Model (BASE)	F1 (%)	∆ over BERT-base
BERT-base (ours)	90.05	—
BLSTM-only	83.43	-
TRANS-BLSTM-SMALL (base)	90.76	+0.71
TRANS-BLSTM (base)	91.53	+1.48

Model (LARGE/Full)	F1 (%)	∆ over BERT-large
BERT-large (ours)	92.34	—
TRANS-BLSTM-SMALL (large)	92.86	+0.52
TRANS-BLSTM (large)	93.82	+1.48
+BLSTM decoder	94.01	—

Pure scaling—e.g. deepening BERT to 48 layers (638M parameters) or doubling hidden size—does not match these gains (F1=92.32 and 86.3, respectively), establishing that architectural enrichment, not mere scale, is responsible for improvement.

GLUE Benchmark (Dev Set Average)

Model	Avg. Score (%)	∆ over BERT
BERT-base (ours)	84.63	—
TRANS-BLSTM-SMALL (base)	84.77	+0.14
TRANS-BLSTM (base)	85.35	+0.72
BERT-large (ours)	85.59	—
TRANS-BLSTM-SMALL (large)	86.23	+0.64
TRANS-BLSTM (large)	86.50	+0.91

BLSTM-only models remain inferior to any Transformer variant, underscoring the complementarity of attention and recurrence.

6. Analysis and Intuition

The self-attention mechanism in the Transformer is well suited to global dependency modeling but lacks built-in inductive bias for sequential information. The BLSTM sublayer, by contrast, excels at sculpting local and sequential context via its gating functions. By combining these sublayers, TRANS-BLSTM constructs token embeddings that jointly encode global context and rich sequential dependencies at every layer.

Despite nearly doubling the number of trainable parameters, TRANS-BLSTM maintains training tractability, facilitated by careful use of residual connections and LayerNorm within each block. This scaffolding ensures that more intricate interleaving of attention and recurrence remains stable during optimization.

Architectural diversity, not simple scaling, is critical: simply increasing depth or width in a pure Transformer leads to plateaued or reduced performance, whereas incorporating BLSTM modules into each block yields robust, consistent gains.

7. Concluding Perspective and Significance

TRANS-BLSTM constitutes a practical enhancement for BERT-style pre-training regimes by embedding bidirectional LSTMs within Transformer blocks, thereby synthesizing the optimization of both parallel attention and sequential recurrence. The observed improvements—up to 1.5 F1 on SQuAD and ~1 point on GLUE—demonstrate that combining disparate architectural principles can extend the state-of-the-art in language understanding. This evidence suggests that architectural enrichment, rather than monotonic scaling, is a promising avenue for model advancement (Huang et al., 2020).