Suffix Bidirectional LSTM (SuBiLSTM)

• SuBiLSTM is a modification of BiLSTM that encodes both the prefix and suffix of a sequence, reducing the standard sequential bias.
• It employs four directional LSTMs with element-wise max pooling to fuse local cues and far-reaching context, enhancing sentence representations.
• Despite its O(n²) time complexity, SuBiLSTM yields state-of-the-art results in text classification, entailment, and paraphrase detection.

Searching arXiv for the specified paper to ground the article and citation. Suffix Bidirectional LSTM (SuBiLSTM) is a simple, general modification of the standard BiLSTM encoder that explicitly encodes both the prefix and the suffix of a sequence in both directions at every token position. By pooling with element-wise max between complementary encodings in each direction, it nudges the contextual representation toward long-range information while preserving local cues, thereby mitigating the “sequential bias” of standard LSTMs and BiLSTMs. It was introduced as a drop-in replacement for BiLSTM in sentence-modeling architectures and was evaluated on general sentence representation, text classification, textual entailment, and paraphrase detection, where it yielded consistent gains and new state-of-the-art results on fine-grained sentiment classification and question classification (Brahma, 2018).

1. Motivation: sequential bias and long-range context

The central motivation for SuBiLSTM is the sequential bias of LSTMs and BiLSTMs. Because an LSTM updates its hidden state in sequence order, the hidden state at step $i$ is most strongly shaped by the recent inputs it has just processed. This tendency, often attributed to gradient dynamics and gating behavior, means that the contextual vector for token $x_i$ is dominated by tokens near $i$ in the processing order, while tokens farther away have progressively less influence (Brahma, 2018).

In a standard BiLSTM, this asymmetry appears in both directions. The forward state $h_i^{\rightarrow}$ is most influenced by nearby left context, and the backward state $h_i^{\leftarrow}$ by nearby right context. SuBiLSTM addresses this by adding representations whose own sequential bias points toward distant context. If, at position $i$, one also encodes the suffix in the forward direction $(x_i,\ldots,x_n)$, the last few updates of that forward-suffix LSTM correspond to far-right tokens. Analogously, encoding the prefix in the reverse direction $(x_i,\ldots,x_1)$ makes the representation sensitive to far-left tokens. Pooling these far-favoring representations with the usual near-favoring ones yields a contextual vector intended to capture both local and distant dependencies.

This design does not remove recurrence or replace LSTM dynamics with a new interaction mechanism. Instead, it introduces an alternate bias that favors long-range dependencies while retaining the inductive structure of BiLSTM-based sentence encoders.

2. Formal construction and representation geometry

Let the input sentence be the token sequence $x_1, x_2, \ldots, x_n$, with embeddings $e_1,\ldots,e_n$, where $x_i$0 is written for brevity. In a standard BiLSTM, the contextual representation at position $x_i$1 concatenates a forward prefix encoding and a backward suffix encoding:

$x_i$2

$x_i$3

If each LSTM has hidden size $x_i$4, then $x_i$5.

For each position $x_i$6, SuBiLSTM computes four encodings: the prefix and suffix, each in both directions (Brahma, 2018):

• Forward prefix:

$x_i$7

• Backward suffix:

$x_i$8

• Forward suffix:

$x_i$9

• Backward prefix:

$i$0

The composition rule pools the two forward encodings via element-wise max and similarly pools the two backward encodings, then concatenates the results:

$i$1

$i$2

The pooling operator is element-wise max; no learned gating or normalization is introduced. If each directional LSTM has hidden size $i$3, then $i$4 and $i$5, so $i$6, the same size as a standard BiLSTM representation.

The intended interpretation follows directly from the directional construction. $i$7 is sequentially biased toward far-right tokens, while $i$8 is biased toward near-left tokens. Max-pooling the two forward encodings selects salient features from both local and distant rightward context; the same logic applies in reverse for the backward encodings. Equivalently, SuBiLSTM builds bidirectional encodings for both the prefix and the suffix and then compacts them via max-pooling.

3. Variants, parameter sharing, and computational profile

Two variants are reported. SuBiLSTM uses distinct LSTM parameters for prefix versus suffix encoders in each direction, yielding four LSTMs total: $i$9, $h_i^{\rightarrow}$0, $h_i^{\rightarrow}$1, and $h_i^{\rightarrow}$2. Its parameter count is twice that of a standard BiLSTM (Brahma, 2018).

SuBiLSTM-Tied shares parameters between prefix and suffix encoders within each direction:

$h_i^{\rightarrow}$3

This reduces the number of LSTMs back to two, matching the parameter count of a standard BiLSTM. Tying acts as a form of regularization and was observed to be beneficial on smaller datasets.

The principal computational distinction from BiLSTM is time complexity. Computing $h_i^{\rightarrow}$4 for all $h_i^{\rightarrow}$5 is one forward pass over $h_i^{\rightarrow}$6, and computing $h_i^{\rightarrow}$7 for all $h_i^{\rightarrow}$8 is one backward pass, each costing $h_i^{\rightarrow}$9. By contrast, computing $h_i^{\leftarrow}$0 for all $h_i^{\leftarrow}$1 naively requires running the forward LSTM over each suffix $h_i^{\leftarrow}$2 for $h_i^{\leftarrow}$3, which is $h_i^{\leftarrow}$4 total work; the same applies to $h_i^{\leftarrow}$5. Overall worst-case time complexity per sentence is therefore quadratic in sequence length, $h_i^{\leftarrow}$6, versus $h_i^{\leftarrow}$7 for BiLSTM.

The reported implementation realizes this by batching “successively smaller suffixes” across the mini-batch to amortize overhead; several shorter suffix batches can be combined, and all four streams can be computed in parallel on modern hardware, which substantially mitigates wall-clock cost. Representation size per token remains $h_i^{\leftarrow}$8, but intermediate four-stream computations must be stored. The paper emphasizes quadratic complexity and parallelization but does not report precise slowdown factors; training time is higher in principle than for BiLSTM but was considered acceptable given the observed gains.

4. Use as a drop-in replacement in sentence modeling

SuBiLSTM is integrated as a drop-in replacement for BiLSTM wherever token-level contextual vectors are consumed. In sentence encoders for transfer learning, such as InferSent-style models, a sentence vector is formed by max-pooling across token positions:

$h_i^{\leftarrow}$9

In attention-based models, including BCN for classification and ESIM for textual entailment, the first BiLSTM layer is replaced with SuBiLSTM while downstream attention, pooling, and classification layers remain unchanged. In Siamese paraphrase-detection architectures, sentence vectors built by SuBiLSTM via max over positions are combined as $i$0 and fed to MLP classifiers (Brahma, 2018).

The reported training configurations illustrate that the substitution is architectural rather than procedural. For general sentence representation, training is performed on AllNLI, which combines SNLI and MultiNLI; the LSTM hidden size is $i$1, producing a 4096-d sentence vector via max-pooling over tokens; two fully connected layers of size 512 are used; GloVe embeddings are fixed during training; and pairwise classification uses $i$2. For text classification with BCN, the hidden dimension is 300 for the LSTMs, only the first BiLSTM is replaced, embeddings are GloVe or GloVe+CoVe and are fixed, dropout is applied after the embedding layer and before the classification layer, maxout layers use reduction factors 4 and 2, weight decay is used, and the optimizer is Adam with learning rate $i$3. For ESIM on SNLI, only the first BiLSTM is replaced, embeddings are 300-d GloVe updated during training, and the LSTMs are 300-d. For paraphrase detection, embeddings are 300-d GloVe updated during training, LSTMs use hidden size 600, one 600-d fully connected layer with ReLU is used, dropout follows embeddings and ReLU, and Adam uses learning rate $i$4.

A plausible implication is that SuBiLSTM was designed to preserve the interface expected by existing BiLSTM-centric pipelines: token representations retain the same dimensionality as standard BiLSTM outputs, and sentence-level aggregation can remain unchanged.

5. Empirical results across transfer, classification, entailment, and paraphrase detection

The empirical evaluation covers general sentence representation via SentEval, text classification, textual entailment, and paraphrase detection (Brahma, 2018). The transfer-learning setup trains on AllNLI and evaluates on MR, CR, SUBJ, MPQA, SST (binary), TREC (6-way), MRPC (acc/F1), SICK-R (Pearson), SICK-E (acc), and STSB (Pearson). Task-specific evaluations use SST-2, SST-5, TREC-6, TREC-50, SNLI, and QUORA. Example dataset sizes reported are SST-2 56.4k, SST-5 94.2k, TREC-6/50 4.3k, SNLI 550k, MultiNLI 393k, and QUORA 384k.

For SentEval transfer, gains over the Base BiLSTM (InferSent) are described as consistent. Base BiLSTM scores are MR 81.1, CR 86.3, SUBJ 92.4, MPQA 90.2, SST 84.6, TREC 88.2, MRPC 76.2/83.1, SICK-R 0.884, SICK-E 86.3, and STSB 0.758. SuBiLSTM scores are MR 81.4, CR 86.4, SUBJ 93.2, MPQA 90.7, SST 85.0, TREC 89.8, MRPC 76.3/83.4, SICK-R 0.886, SICK-E 86.7, and STSB 0.770. SuBiLSTM-Tied scores are MR 81.6, CR 86.5, SUBJ 93.0, MPQA 90.5, SST 85.1, TREC 90.4, MRPC 76.3/83.3, SICK-R 0.885, SICK-E 86.3, and STSB 0.771. The reported gains include SUBJ $i$5, MPQA $i$6, TREC $i$7, and STSB $i$8 for SuBiLSTM, and SUBJ $i$9, SST $(x_i,\ldots,x_n)$0, TREC $(x_i,\ldots,x_n)$1, and STSB $(x_i,\ldots,x_n)$2 for the tied variant.

For task-specific classification with BCN, replacing the first BiLSTM yields the following best results: SST-2 accuracy of 91.2 for SuBiLSTM-Tied+CoVe versus 90.1 for BiLSTM+CoVe; SST-5 accuracy of 56.2 for SuBiLSTM-Tied+CoVe versus 53.6 for BiLSTM+CoVe; TREC-6 accuracy of 96.2 for SuBiLSTM-Tied with GloVe versus 95.2 for BiLSTM; and TREC-50 accuracy of 90.2 for SuBiLSTM+CoVe versus 90.0 for BiLSTM+CoVe. The paper identifies new state-of-the-art results on SST-5 at 56.2% and on TREC-6 at 96.2%, while also noting that gains on TREC-50 are modest.

For textual entailment on SNLI with ESIM, the single-model scores are 87.8 for ESIM+BiLSTM, 87.9 for ESIM+2-layer BiLSTM, 88.3 for ESIM+SuBiLSTM, and 88.2 for ESIM+SuBiLSTM-Tied. In 5-model ensembles, ESIM+SuBiLSTM and ESIM+SuBiLSTM-Tied each reach 89.1, compared with 88.6 for ESIM+BiLSTM and 88.7 for ESIM+2-layer BiLSTM.

For paraphrase detection on QUORA with a Siamese encoder without attention, the reported results are 87.8 for BiLSTM, 87.9 for 2-layer BiLSTM, 88.2 for SuBiLSTM, and 88.1 for SuBiLSTM-Tied. The gains persist even without attention, though they are smaller than in attention-based models; the best SuBiLSTM result of 88.2 is described as on par with more complex attention-based systems.

Evaluation	Baseline	SuBiLSTM / SuBiLSTM-Tied
SentEval TREC	88.2	89.8 / 90.4
SentEval STSB	0.758	0.770 / 0.771
SST-5	53.6	56.2
TREC-6	95.2	96.2
SNLI with ESIM	87.8	88.3 / 88.2
QUORA	87.8	88.2 / 88.1

These results support the paper’s claim that substituting SuBiLSTM for BiLSTM improves performance in learning general sentence representations, text classification, textual entailment, and paraphrase detection.

6. Interpretation, ablations, limitations, and relation to adjacent approaches

Several ablations clarify which design choices matter. The paper considers concatenation, averaging, and learned gating as alternatives to the max-pooling composition, but uses element-wise max because it is parameterless and performed best empirically (Brahma, 2018). Parameter sharing also matters: SuBiLSTM-Tied often performs better on smaller datasets such as SST and TREC, likely due to regularization from weight sharing, whereas on larger datasets such as SNLI and QUORA, the untied variant slightly edges out the tied version, benefiting from higher capacity. A 2-layer BiLSTM baseline did not match the gains from SuBiLSTM, suggesting that the long-range bias introduced by prefix/suffix encoding is not easily replicated by simply stacking BiLSTMs.

The paper identifies sequence-length-sensitive settings as especially favorable. Improvements on semantic textual similarity in STSB and on question classification in TREC are highlighted as evidence that richer long-range context is useful. Attention-heavy architectures such as BCN and ESIM benefit notably, likely because downstream attention can better leverage enhanced token vectors. Gains are still present in simple Siamese setups without attention, but they are smaller.

The main limitation is computational. SuBiLSTM remains slower than BiLSTM because computing both directional suffix and prefix streams at all positions yields $(x_i,\ldots,x_n)$3 time complexity. Parallelization and batching mitigate the overhead, but the paper does not report precise runtime slowdowns. It also notes that improvements are modest on some datasets, explicitly including TREC-50.

A common misunderstanding would be to treat SuBiLSTM as interchangeable with self-attention or Transformers. The paper distinguishes the mechanisms: self-attention directly aggregates global context in $(x_i,\ldots,x_n)$4 through learned content-based interactions, whereas SuBiLSTM is an RNN-based drop-in replacement for BiLSTM that strengthens long-range bias without changing downstream components. No head-to-head comparison is provided, so broader claims about relative superiority would go beyond the reported evidence.

Taken together, the reported results motivate a clear usage pattern. If a model already relies on BiLSTM token encodings, SuBiLSTM provides a direct substitution that preserves output dimensionality and often improves long-range contextualization. The tied variant matches BiLSTM’s parameter count and often excels on smaller datasets; the untied variant offers more capacity and tends to perform better on larger datasets. This suggests that SuBiLSTM is best understood not as a replacement for sentence-modeling pipelines in general, but as a targeted modification of BiLSTM-based encoders designed to counteract sequential bias while leaving the rest of the architecture intact.