---
title: 'HOb2sRNN: High-Order Recurrent Networks'
url: https://www.emergentmind.com/topics/hob2srnn
type: topic
---

# HOb2sRNN: High-Order Recurrent Networks

Searching arXiv for papers on HOb2sRNN / Higher Order Recurrent Neural Networks to ground the article in the relevant literature.
HOb2sRNN denotes a family of **higher-order recurrent neural networks**—more commonly abbreviated **HORNNs**—in which the hidden-state update depends on multiple delayed hidden states rather than only the immediately preceding state. In the literature, the model is introduced as a modification of the standard recurrent neural network intended to improve the modelling of long-term dependencies in sequential data by creating shorter back-propagation paths through time, while retaining compatibility with ordinary back-propagation through time (BPTT) and efficient matrix-based implementations [1605.00064]. A later acoustic-modelling formulation uses the same central idea in speech recognition, where high-order temporal skip connections are positioned as a lower-parameter alternative to LSTM-style memory cells for mitigating vanishing long-term gradients [1802.08314].

## 1. Terminology and defining idea

The defining architectural difference between a standard RNN and a HOb2sRNN/HORNN is the order of recurrence. A classical first-order RNN updates its hidden state as
$$
{\bf h}_t = f(W_{in} {\bf x}_t  +   W_{h} {\bf h}_{t-1} )
$$
and produces its output as
$$
{\bf y}_t=g(  W_{out} {\bf h}_{t}) .
$$
In this formulation, the new hidden state depends only on ${\bf x}_t$ and ${\bf h}_{t-1}$. HORNNs generalize this to an $N$-th order recurrence,
$$
{\bf h}_t= f \left(W_{in} {\bf x}_t + \sum_{n=1}^N W_{hn} {\bf h}_{t-n} \right),
$$
where separate recurrent matrices $\{W_{hn}\mid n=1,\cdots,N\}$ act on distinct delayed hidden states. The standard RNN is therefore the **1st-order** special case of the higher-order formulation [1605.00064].

The motivation is the familiar optimization pathology of recurrent networks: although an RNN is theoretically capable of remembering arbitrarily long history, BPTT often suffers from vanishing gradients, and sometimes exploding gradients, across long temporal chains. HORNNs address this by feeding back **multiple previous hidden states**. This gives the network direct access to richer short-term history and simultaneously creates more direct backward paths for gradient propagation.

The original presentation explicitly likens these structures to **digital filter structures in signal processing**: multiple delayed signals are combined with different weights to form the current state. In that analogy, “higher order” refers to the presence of multiple delayed feedback taps rather than to greater depth in the feed-forward sense. In the acoustic-modelling formulation, the same idea is described as relaxing the standard Elman RNN’s **1st-order Markov conditional independence** structure, so that temporal dependence is not compressed solely into $\mathbf{h}_{t-1}$ [1802.08314].

The nomenclature is somewhat uneven across sources. The model family is consistently called **HORNN** or **High Order Recurrent Neural Network**. A plausible interpretation is that the “b2s” style in **HOb2sRNN** refers to the second-order skip-connection style emphasized in some formulations, but the explicit architectural definitions are given by the recurrence equations rather than by a separate formal acronym expansion.

## 2. Core recurrence forms and architectural variants

The generic HORNN combines several delayed hidden states through separate learned transforms. In the language-modelling formulation, this is extended with three mechanisms intended to regulate the relative scale of signals arriving from different delays [1605.00064; 1802.08314].

The first is **max-based pooling**:
$$
{\bf h}_t= f \Big(W_{in} {\bf x}_t + {\max}_{n=1}^N \; ( W_{hn} {\bf h}_{t-n} ) \Big),
$$
where the max is applied elementwise. This selects, at each hidden dimension, the strongest transformed contribution across the delayed states.

The second is **FOFE-based pooling**:
$$
{\bf h}_t= f \left(W_{in} {\bf x}_t + \sum_{n=1}^N \;  \alpha^n \cdot W_{hn} {\bf h}_{t-n} \right),
$$
with forgetting factor $0<\alpha<1$. Here the weighting is borrowed from **fixed-size ordinally-forgetting encoding (FOFE)**. Older states are downweighted by $\alpha^n$, which introduces an explicit forgetting mechanism and reduces dynamic-range disparity across delays.

The third is the **gated HORNN**:
$$
{\bf h}_t= f \left(W_{in} {\bf x}_t + \sum_{n=1}^N \;  {\bf r}_n \odot \Big(W_{hn} {\bf h}_{t-n} \Big) \right),
$$
with gates
$$
{\bf r}_n=\sigma \left(W_{1n}^{g}  {\bf x}_t +   W_{2n}^{g}   {\bf h}_{t-n} \right).
$$
These gate vectors allow the network to learn how much each delayed hidden state should contribute. The paper characterizes this as conceptually similar to LSTM/GRU gating or soft attention.

In acoustic modelling, two specific high-order recurrences are emphasized. For **ReLU HORNNs**, the proposed update is
$$
\mathbf{h}_t = f(\mathbf{W}\mathbf{x}_t + \mathbf{U}_1\mathbf{h}_{t-1} + \mathbf{U}_n\mathbf{h}_{t-n} + \mathbf{b}),
$$
while for **sigmoid HORNNs** the preferred form is
$$
\mathbf{h}_t = f(\mathbf{W}\mathbf{x}_t + \mathbf{U}_1\mathbf{h}_{t-1} + \mathbf{U}_n\mathbf{h}_{t-n} + \mathbf{h}_{t-m} + \mathbf{b}).
$$
The direct addition of $\mathbf{h}_{t-m}$ in the sigmoid case is used to preserve access to earlier hidden information without another learned transform. The reported final setup for the sigmoid models uses $m=1$ and $n=2$.

To further reduce cost, the same work introduces **projected HORNNs (HORNNP)** with a recurrent projection $\mathbf{P}\in\mathbb{R}^{D_p\times D_h}$ and factorized recurrent transforms,
$$
\mathbf{U}_1 \approx \mathbf{U}_{p1}\mathbf{P}, \qquad \mathbf{U}_n \approx \mathbf{U}_{pn}\mathbf{P}.
$$
The projected ReLU form becomes
$$
\mathbf{h}_t = f(\mathbf{W}\mathbf{x}_t + \mathbf{U}_{p1}\mathbf{P}\mathbf{h}_{t-1} + \mathbf{U}_{pn}\mathbf{P}\mathbf{h}_{t-n} + \mathbf{b}),
$$
and the projected sigmoid form adds $\mathbf{h}_{t-m}$ analogously.

## 3. Optimization rationale and computational characteristics

HORNNs are trained with the **same BPTT framework** as ordinary RNNs, but the hidden-layer error at time $t$ must be propagated through multiple recurrent branches rather than a single one. In a 3rd-order HORNN, for example, the error at ${\bf h}_t$ is propagated to ${\bf h}_{t-1}$, ${\bf h}_{t-2}$, and ${\bf h}_{t-3}$, in addition to the input side. The intended effect is not the elimination of vanishing gradients in a formal sense, but the creation of shorter and more direct temporal routes along which gradient information can flow [1605.00064; 1802.08314].

This multi-branch structure also motivates the pooling variants. Signals from different delays may have very different magnitudes in the forward pass, and, during BPTT, very long shortcut paths can overwhelm shorter ones. FOFE-based decay and learned gates are introduced precisely to regulate these multi-timescale contributions.

The language-modelling implementation unfolds the network fully and notes that the computation remains GPU-friendly because most operations are still matrix multiplications; only the recursive update step retains the usual temporal dependence. In that study, training uses mini-batch SGD, **BPTT length 30**, mini-batches of 20 subsequences of length 30, Gaussian initialization with mean $0$ and standard deviation $0.1$, gradient clipping at $5.0$, initial learning rate $0.5$ with halving when validation cross-entropy stops improving, plus weight decay, momentum, and column normalization. The models do **not** use dropout. For FOFE pooling, $\alpha=0.6$; hidden size is 400 for PTB and 500 for text8.

The acoustic-modelling study emphasizes cost relative to LSTM variants. With input dimension $D_x$, hidden dimension $D_h$, and projection dimension $D_p$, it gives the recurrent-layer parameter counts as
$$
(D_x + D_h)D_h + D_h
$$
for a standard RNN,
$$
(D_x + 2D_h)D_h + D_h
$$
for a HORNN,
and
$$
4(D_x + D_h)D_h + 7D_h
$$
for the particular peephole LSTM form used there. For the projected version, the HORNN count is
$$
D_hD_p + (D_x + 2D_p)D_h + D_h,
$$
with an approximate reduction ratio
$$
\approx \frac{2D_h}{3D_p} \quad \text{when } D_h > D_p \gg D_x.
$$
Ignoring elementwise operations, the same paper estimates testing complexity as
$$
\mathcal{O}\bigl(T(D_x + 3D_p)D_h\bigr)
$$
for HORNNP and
$$
\mathcal{O}\bigl(TD_hD_p + 4T(D_x + D_p)D_h\bigr)
$$
for LSTMP, concluding that HORNNPs use **less than 3/5 of the calculations** of LSTMPs and, in the reported HTK implementation, often ran about **50% faster**.

## 4. Language-modelling results

The initial empirical study evaluates HORNNs on **Penn Treebank (PTB)** and **English text8**, using perplexity (PPL) as the metric [1605.00064]. PTB comprises about **930K training words**, **74K validation words**, and **82K test words**, with vocabulary limited to **10k**. English text8 is substantially larger, with **16.8M training words** and **0.17M test words**, and vocabulary around **44k** after replacing rare words with $<\mathrm{UNK}>$.

On PTB, orders 2, 3, and 4 are compared, and performance improves from 2nd to 3rd order, with little gain beyond 3rd order. The reported **test perplexities** are as follows.

| Model | PTB test PPL | text8 test PPL |
|---|---:|---:|
| Regular RNN | 123 | 184 |
| LSTM | 117 | 156 |
| HORNN (3rd order) | 108 | 172 |
| Max HORNN (3rd order) | 109 | 163 |
| FOFE HORNN (3rd order) | 101 | 154 |
| Gated HORNN (3rd order) | 100 | 144 |

The PTB comparison in the same study also includes **Stack RNN: 110**, **Deep RNN: 107**, and **FOFE-FNN: 108**. Under the reported training conditions, the paper states that the **FOFE-based** and **gated HORNNs** achieve the **best reported performance** on PTB. On text8, only 3rd-order HORNNs are evaluated due to training cost; the comparison includes **SCRNN: 161** and **E2E Memory Net: 147**, with the **gated HORNN** reaching **144**, slightly better than the end-to-end memory network.

The reported interpretation is twofold. First, directly incorporating multiple previous hidden states yields a richer and more explicit short-term memory of the sequence. Second, the additional direct paths help gradients propagate backward more effectively during BPTT. The pooling mechanisms then calibrate contributions across time scales: FOFE imposes controlled decay, whereas gating learns when each delayed state should matter.

## 5. Acoustic modelling and speech-recognition usage

A subsequent formulation specializes high-order recurrent networks for **acoustic modelling** in speech recognition on **MGB3 British English multi-genre broadcast** data [1802.08314]. The task uses BBC TV programmes across many genres, with both a **55h subset** sampled from the **275h** training set and the **275h full training set**. The vocabulary is **63k words**, and the trigram word-level language model is trained on the acoustic transcripts together with an additional **640 million-word MGB subtitle archive**. Evaluation is performed on **dev17b**, consisting of **5.55 hours**, **5,201 manually segmented utterances**, and material from **14 episodes of 13 shows**, with both **Viterbi 1-best** and **confusion network (CN)** decoding.

The acoustic front end uses **40-dim log-Mel filterbanks**, expanded to **80-dim with $\Delta$ coefficients**, with utterance-level mean normalization, show-segment-level variance normalization, and frames delayed by **5 steps**. Training uses an extended **HTK 3.5** toolkit, **NewBob$^+$** learning rate schedule, initial learning rate $5\times 10^{-4}$ for ReLU models and $2\times 10^{-3}$ for the others, and gradient clipping threshold **0.32**.

The abstract summary reports that HORNN architectures for rectified linear unit and sigmoid activation functions reduced WERs by **4.2%** and **6.3%** over the corresponding RNNs, and gave similar WERs to a projected LSTM while using only **20%--50%** of the recurrent-layer parameters and computation. In the 55h single-layer experiments, **ReLU HORNNs** achieved WERs at least as low as LSTM, **sigmoid HORNNs** outperformed sigmoid ResRNNs and matched LSTM performance, and both HORNN types were faster than LSTM and ResRNN.

The parameter comparisons are explicit. The standard RNN baseline has **0.29M** recurrent parameters; the LSTM baseline has **1.16M**. For one-layer projected systems with $D_h=500$ and $D_p=250$, **LSTMP** has **0.79M** parameters and **HORNNP** has **0.42M**, with similar WERs. Smaller projected HORNNs with **0.23M** parameters are reported to achieve similar WER to LSTM/LSTMP while using only **20%** to **29%** of the recurrent parameters of LSTM/LSTMP.

On the **275h full training set**, the reported CN WERs are:
- **1L LSTMP**: **26.0**
- **1L sigmoid HORNNP**: **25.8**
- **1L ReLU HORNNP**: **25.9**

For two-layer systems:
- **2L LSTMP**: **25.2**
- **2L sigmoid HORNNP**: **25.2**
- **2L ReLU HORNNP**: **25.0**

Within this experimental setup, the conclusion is that HORNNs achieve similar or slightly better WERs than LSTMs/LSTMPs while using markedly fewer parameters and less computation.

## 6. Relation to LSTMs, residual RNNs, orthogonal RNNs, and hypercomplex variants

HORNNs occupy a distinct position among recurrent architectures designed to address long-range temporal dependency. Their primary mechanism is **high-order temporal feedback** rather than explicit memory cells, orthogonality constraints, or hypercomplex algebra.

Relative to **LSTM**, the main contrast is architectural economy. The acoustic-modelling paper frames LSTM as an effective remedy for vanishing gradients, but one that incurs substantially higher parameter cost because it computes input, forget, output, and cell-candidate transformations. HORNNs instead introduce temporal shortcut connections. This makes them cheaper than LSTM-style gating at a fixed hidden size, although the **gated HORNN** variant reintroduces learned modulation over the delayed paths [1802.08314].

Relative to **residual RNNs**, the same paper compares HORNNs against a residual formulation
$$
\mathbf{h}_t = f\bigl(\mathbf{U}_{d2} f(\mathbf{W}\mathbf{x}_t + \mathbf{U}_{d1}\mathbf{h}_{t-1} + \mathbf{b}) + \mathbf{h}_{t-m}\bigr),
$$
with the standard residual setting using $m=1$, and reports that HORNNs are both faster and better in WER. This indicates that not every temporal shortcut is equivalent; the specific high-order recurrent structure matters.

Relative to **orthogonal/unitary RNNs**, the distinction is methodological. "Orthogonal Recurrent Neural Networks with Scaled Cayley Transform" [1707.09520] addresses vanishing and exploding gradients by constraining the recurrent matrix to remain orthogonal,
$$
W = (I + A)^{-1}(I - A)D,
$$
where $A$ is skew-symmetric and $D$ is a fixed diagonal sign matrix. That line of work stabilizes repeated recurrent multiplication by norm preservation. HORNNs, by contrast, keep ordinary recurrent transforms but alter the temporal graph by adding multiple delayed feedback paths. Both approaches target related optimization pathologies, but through different structural priors.

Relative to **hypercomplex recurrent networks**, the connection is looser and more analogical. "Real to H-space Encoder for Speech Recognition" [1906.08043] states that quaternion recurrent systems are part of the broader idea behind **hypercomplex recurrent networks, including HOb2sRNN-style models**, even though it does **not** mention HOb2sRNN by name. In that framing, hypercomplex RNNs seek structured interactions among components of inputs and hidden states using algebras such as the quaternions, whereas HORNNs seek better temporal credit assignment through multiple feedback delays. A plausible implication is that both families are responses to limitations of plain real-valued first-order recurrence, but they operate on different axes: temporal order in HORNNs, representational algebra in quaternion models.

A recurring misconception is to treat HOb2sRNN as merely a deeper or wider vanilla RNN. The literature instead presents it as a change in **recurrent order**. Another is to identify it with gating per se. In fact, gating is only one HORNN variant; the simpler ReLU, sigmoid, max-pooling, and FOFE-pooling forms already instantiate the core higher-order idea.

Source: https://www.emergentmind.com/topics/hob2srnn