HOb2sRNN: High-Order Recurrent Networks

• HOb2sRNN is a family of higher-order recurrent neural networks that use multiple delayed hidden states to improve gradient propagation over long sequences.
• The architecture features variants such as max pooling, FOFE-based pooling, and gating to regulate contributions from different time delays.
• Empirical studies show that HOb2sRNNs achieve competitive language modeling and acoustic modeling performance with lower parameter counts and computation compared to LSTMs.

Searching arXiv for papers on HOb2sRNN / Higher Order Recurrent Neural Networks to ground the article in the relevant literature. HOb2sRNN denotes a family of higher-order recurrent neural networks—more commonly abbreviated HORNNs—in which the hidden-state update depends on multiple delayed hidden states rather than only the immediately preceding state. In the literature, the model is introduced as a modification of the standard recurrent neural network intended to improve the modelling of long-term dependencies in sequential data by creating shorter back-propagation paths through time, while retaining compatibility with ordinary back-propagation through time (BPTT) and efficient matrix-based implementations (Soltani et al., 2016). A later acoustic-modelling formulation uses the same central idea in speech recognition, where high-order temporal skip connections are positioned as a lower-parameter alternative to LSTM-style memory cells for mitigating vanishing long-term gradients (Zhang et al., 2018).

1. Terminology and defining idea

The defining architectural difference between a standard RNN and a HOb2sRNN/HORNN is the order of recurrence. A classical first-order RNN updates its hidden state as

$${\bf h}_t = f(W_{in} {\bf x}_t + W_{h} {\bf h}_{t-1} )$$

and produces its output as

${\bf y}_t=g( W_{out} {\bf h}_{t}) .$

In this formulation, the new hidden state depends only on ${\bf x}_t$ and ${\bf h}_{t-1}$. HORNNs generalize this to an $N$-th order recurrence,

$${\bf h}_t= f \left(W_{in} {\bf x}_t + \sum_{n=1}^N W_{hn} {\bf h}_{t-n} \right),$$

where separate recurrent matrices $\{W_{hn}\mid n=1,\cdots,N\}$ act on distinct delayed hidden states. The standard RNN is therefore the 1st-order special case of the higher-order formulation (Soltani et al., 2016).

The motivation is the familiar optimization pathology of recurrent networks: although an RNN is theoretically capable of remembering arbitrarily long history, BPTT often suffers from vanishing gradients, and sometimes exploding gradients, across long temporal chains. HORNNs address this by feeding back multiple previous hidden states. This gives the network direct access to richer short-term history and simultaneously creates more direct backward paths for gradient propagation.

The original presentation explicitly likens these structures to digital filter structures in signal processing: multiple delayed signals are combined with different weights to form the current state. In that analogy, “higher order” refers to the presence of multiple delayed feedback taps rather than to greater depth in the feed-forward sense. In the acoustic-modelling formulation, the same idea is described as relaxing the standard Elman RNN’s 1st-order Markov conditional independence structure, so that temporal dependence is not compressed solely into $\mathbf{h}_{t-1}$ (Zhang et al., 2018).

The nomenclature is somewhat uneven across sources. The model family is consistently called HORNN or High Order Recurrent Neural Network. A plausible interpretation is that the “b2s” style in HOb2sRNN refers to the second-order skip-connection style emphasized in some formulations, but the explicit architectural definitions are given by the recurrence equations rather than by a separate formal acronym expansion.

2. Core recurrence forms and architectural variants

The generic HORNN combines several delayed hidden states through separate learned transforms. In the language-modelling formulation, this is extended with three mechanisms intended to regulate the relative scale of signals arriving from different delays (Soltani et al., 2016, Zhang et al., 2018).

The first is max-based pooling:

$${\bf h}_t= f \Big(W_{in} {\bf x}_t + {\max}_{n=1}^N \; ( W_{hn} {\bf h}_{t-n} ) \Big),$$

where the max is applied elementwise. This selects, at each hidden dimension, the strongest transformed contribution across the delayed states.

The second is FOFE-based pooling:

$${\bf h}_t= f \left(W_{in} {\bf x}_t + \sum_{n=1}^N \; \alpha^n \cdot W_{hn} {\bf h}_{t-n} \right),$$

with forgetting factor ${\bf y}_t=g( W_{out} {\bf h}_{t}) .$0. Here the weighting is borrowed from fixed-size ordinally-forgetting encoding (FOFE). Older states are downweighted by ${\bf y}_t=g( W_{out} {\bf h}_{t}) .$1, which introduces an explicit forgetting mechanism and reduces dynamic-range disparity across delays.

The third is the gated HORNN:

${\bf y}_t=g( W_{out} {\bf h}_{t}) .$2

with gates

${\bf y}_t=g( W_{out} {\bf h}_{t}) .$3

These gate vectors allow the network to learn how much each delayed hidden state should contribute. The paper characterizes this as conceptually similar to LSTM/GRU gating or soft attention.

In acoustic modelling, two specific high-order recurrences are emphasized. For ReLU HORNNs, the proposed update is

${\bf y}_t=g( W_{out} {\bf h}_{t}) .$4

while for sigmoid HORNNs the preferred form is

${\bf y}_t=g( W_{out} {\bf h}_{t}) .$5

The direct addition of ${\bf y}_t=g( W_{out} {\bf h}_{t}) .$6 in the sigmoid case is used to preserve access to earlier hidden information without another learned transform. The reported final setup for the sigmoid models uses ${\bf y}_t=g( W_{out} {\bf h}_{t}) .$7 and ${\bf y}_t=g( W_{out} {\bf h}_{t}) .$8.

To further reduce cost, the same work introduces projected HORNNs (HORNNP) with a recurrent projection ${\bf y}_t=g( W_{out} {\bf h}_{t}) .$9 and factorized recurrent transforms,

${\bf x}_t$0

The projected ReLU form becomes

${\bf x}_t$1

and the projected sigmoid form adds ${\bf x}_t$2 analogously.

3. Optimization rationale and computational characteristics

HORNNs are trained with the same BPTT framework as ordinary RNNs, but the hidden-layer error at time ${\bf x}_t$3 must be propagated through multiple recurrent branches rather than a single one. In a 3rd-order HORNN, for example, the error at ${\bf x}_t$4 is propagated to ${\bf x}_t$5, ${\bf x}_t$6, and ${\bf x}_t$7, in addition to the input side. The intended effect is not the elimination of vanishing gradients in a formal sense, but the creation of shorter and more direct temporal routes along which gradient information can flow (Soltani et al., 2016, Zhang et al., 2018).

This multi-branch structure also motivates the pooling variants. Signals from different delays may have very different magnitudes in the forward pass, and, during BPTT, very long shortcut paths can overwhelm shorter ones. FOFE-based decay and learned gates are introduced precisely to regulate these multi-timescale contributions.

The language-modelling implementation unfolds the network fully and notes that the computation remains GPU-friendly because most operations are still matrix multiplications; only the recursive update step retains the usual temporal dependence. In that study, training uses mini-batch SGD, BPTT length 30, mini-batches of 20 subsequences of length 30, Gaussian initialization with mean ${\bf x}_t$8 and standard deviation ${\bf x}_t$9, gradient clipping at ${\bf h}_{t-1}$0, initial learning rate ${\bf h}_{t-1}$1 with halving when validation cross-entropy stops improving, plus weight decay, momentum, and column normalization. The models do not use dropout. For FOFE pooling, ${\bf h}_{t-1}$2; hidden size is 400 for PTB and 500 for text8.

The acoustic-modelling study emphasizes cost relative to LSTM variants. With input dimension ${\bf h}_{t-1}$3, hidden dimension ${\bf h}_{t-1}$4, and projection dimension ${\bf h}_{t-1}$5, it gives the recurrent-layer parameter counts as

${\bf h}_{t-1}$6

for a standard RNN,

${\bf h}_{t-1}$7

for a HORNN, and

${\bf h}_{t-1}$8

for the particular peephole LSTM form used there. For the projected version, the HORNN count is

${\bf h}_{t-1}$9

with an approximate reduction ratio

$N$0

Ignoring elementwise operations, the same paper estimates testing complexity as

$N$1

for HORNNP and

$N$2

for LSTMP, concluding that HORNNPs use less than 3/5 of the calculations of LSTMPs and, in the reported HTK implementation, often ran about 50% faster.

4. Language-modelling results

The initial empirical study evaluates HORNNs on Penn Treebank (PTB) and English text8, using perplexity (PPL) as the metric (Soltani et al., 2016). PTB comprises about 930K training words, 74K validation words, and 82K test words, with vocabulary limited to 10k. English text8 is substantially larger, with 16.8M training words and 0.17M test words, and vocabulary around 44k after replacing rare words with $N$3.

On PTB, orders 2, 3, and 4 are compared, and performance improves from 2nd to 3rd order, with little gain beyond 3rd order. The reported test perplexities are as follows.

Model	PTB test PPL	text8 test PPL
Regular RNN	123	184
LSTM	117	156
HORNN (3rd order)	108	172
Max HORNN (3rd order)	109	163
FOFE HORNN (3rd order)	101	154
Gated HORNN (3rd order)	100	144

The PTB comparison in the same study also includes Stack RNN: 110, Deep RNN: 107, and FOFE-FNN: 108. Under the reported training conditions, the paper states that the FOFE-based and gated HORNNs achieve the best reported performance on PTB. On text8, only 3rd-order HORNNs are evaluated due to training cost; the comparison includes SCRNN: 161 and E2E Memory Net: 147, with the gated HORNN reaching 144, slightly better than the end-to-end memory network.

The reported interpretation is twofold. First, directly incorporating multiple previous hidden states yields a richer and more explicit short-term memory of the sequence. Second, the additional direct paths help gradients propagate backward more effectively during BPTT. The pooling mechanisms then calibrate contributions across time scales: FOFE imposes controlled decay, whereas gating learns when each delayed state should matter.

5. Acoustic modelling and speech-recognition usage

A subsequent formulation specializes high-order recurrent networks for acoustic modelling in speech recognition on MGB3 British English multi-genre broadcast data (Zhang et al., 2018). The task uses BBC TV programmes across many genres, with both a 55h subset sampled from the 275h training set and the 275h full training set. The vocabulary is 63k words, and the trigram word-level LLM is trained on the acoustic transcripts together with an additional 640 million-word MGB subtitle archive. Evaluation is performed on dev17b, consisting of 5.55 hours, 5,201 manually segmented utterances, and material from 14 episodes of 13 shows, with both Viterbi 1-best and confusion network (CN) decoding.

The acoustic front end uses 40-dim log-Mel filterbanks, expanded to 80-dim with $N$4 coefficients, with utterance-level mean normalization, show-segment-level variance normalization, and frames delayed by 5 steps. Training uses an extended HTK 3.5 toolkit, NewBob$N$5 learning rate schedule, initial learning rate $N$6 for ReLU models and $N$7 for the others, and gradient clipping threshold 0.32.

The abstract summary reports that HORNN architectures for rectified linear unit and sigmoid activation functions reduced WERs by 4.2% and 6.3% over the corresponding RNNs, and gave similar WERs to a projected LSTM while using only 20%--50% of the recurrent-layer parameters and computation. In the 55h single-layer experiments, ReLU HORNNs achieved WERs at least as low as LSTM, sigmoid HORNNs outperformed sigmoid ResRNNs and matched LSTM performance, and both HORNN types were faster than LSTM and ResRNN.

The parameter comparisons are explicit. The standard RNN baseline has 0.29M recurrent parameters; the LSTM baseline has 1.16M. For one-layer projected systems with $N$8 and $N$9, LSTMP has 0.79M parameters and HORNNP has 0.42M, with similar WERs. Smaller projected HORNNs with 0.23M parameters are reported to achieve similar WER to LSTM/LSTMP while using only 20% to 29% of the recurrent parameters of LSTM/LSTMP.

On the 275h full training set, the reported CN WERs are:

• 1L LSTMP: 26.0
• 1L sigmoid HORNNP: 25.8
• 1L ReLU HORNNP: 25.9

For two-layer systems:

• 2L LSTMP: 25.2
• 2L sigmoid HORNNP: 25.2
• 2L ReLU HORNNP: 25.0

Within this experimental setup, the conclusion is that HORNNs achieve similar or slightly better WERs than LSTMs/LSTMPs while using markedly fewer parameters and less computation.

6. Relation to LSTMs, residual RNNs, orthogonal RNNs, and hypercomplex variants

HORNNs occupy a distinct position among recurrent architectures designed to address long-range temporal dependency. Their primary mechanism is high-order temporal feedback rather than explicit memory cells, orthogonality constraints, or hypercomplex algebra.

Relative to LSTM, the main contrast is architectural economy. The acoustic-modelling paper frames LSTM as an effective remedy for vanishing gradients, but one that incurs substantially higher parameter cost because it computes input, forget, output, and cell-candidate transformations. HORNNs instead introduce temporal shortcut connections. This makes them cheaper than LSTM-style gating at a fixed hidden size, although the gated HORNN variant reintroduces learned modulation over the delayed paths (Zhang et al., 2018).

Relative to residual RNNs, the same paper compares HORNNs against a residual formulation

$${\bf h}_t= f \left(W_{in} {\bf x}_t + \sum_{n=1}^N W_{hn} {\bf h}_{t-n} \right),$$0

with the standard residual setting using $${\bf h}_t= f \left(W_{in} {\bf x}_t + \sum_{n=1}^N W_{hn} {\bf h}_{t-n} \right),$$1, and reports that HORNNs are both faster and better in WER. This indicates that not every temporal shortcut is equivalent; the specific high-order recurrent structure matters.

Relative to orthogonal/unitary RNNs, the distinction is methodological. "Orthogonal Recurrent Neural Networks with Scaled Cayley Transform" (Helfrich et al., 2017) addresses vanishing and exploding gradients by constraining the recurrent matrix to remain orthogonal,

$${\bf h}_t= f \left(W_{in} {\bf x}_t + \sum_{n=1}^N W_{hn} {\bf h}_{t-n} \right),$$2

where $${\bf h}_t= f \left(W_{in} {\bf x}_t + \sum_{n=1}^N W_{hn} {\bf h}_{t-n} \right),$$3 is skew-symmetric and $${\bf h}_t= f \left(W_{in} {\bf x}_t + \sum_{n=1}^N W_{hn} {\bf h}_{t-n} \right),$$4 is a fixed diagonal sign matrix. That line of work stabilizes repeated recurrent multiplication by norm preservation. HORNNs, by contrast, keep ordinary recurrent transforms but alter the temporal graph by adding multiple delayed feedback paths. Both approaches target related optimization pathologies, but through different structural priors.

Relative to hypercomplex recurrent networks, the connection is looser and more analogical. "Real to H-space Encoder for Speech Recognition" (Parcollet et al., 2019) states that quaternion recurrent systems are part of the broader idea behind hypercomplex recurrent networks, including HOb2sRNN-style models, even though it does not mention HOb2sRNN by name. In that framing, hypercomplex RNNs seek structured interactions among components of inputs and hidden states using algebras such as the quaternions, whereas HORNNs seek better temporal credit assignment through multiple feedback delays. A plausible implication is that both families are responses to limitations of plain real-valued first-order recurrence, but they operate on different axes: temporal order in HORNNs, representational algebra in quaternion models.

A recurring misconception is to treat HOb2sRNN as merely a deeper or wider vanilla RNN. The literature instead presents it as a change in recurrent order. Another is to identify it with gating per se. In fact, gating is only one HORNN variant; the simpler ReLU, sigmoid, max-pooling, and FOFE-pooling forms already instantiate the core higher-order idea.