Gamma-LSTM: Hierarchical Memory in RNNs

• Gamma-LSTM is a recurrent neural architecture that replaces the traditional LSTM cell state with hierarchical sub-cells regulated by learnable gamma gates.
• It employs adaptive gating and softmax attention to blend information across multiple temporal scales, optimizing both short- and long-term feature retention.
• Empirical evaluations, notably on MNIST and SNLI, demonstrate Gamma-LSTM’s improved accuracy and parameter efficiency compared to traditional stacked LSTMs.

Gamma-LSTM (Γ-LSTM) is an augmented recurrent neural network architecture designed to model complex temporal dependencies via a hierarchical memory structure. It extends the classic Long Short-Term Memory (LSTM) framework by replacing the single internal cell state with a cascade of internal sub-cells operating at multiple temporal scales, regulated by learned gamma (hierarchy-selecting) gates. The architecture aims to provide adaptive memory allocation and processing depth within a single recurrent unit, reducing the need for stacking multiple LSTM layers while improving the ability to capture both short- and long-term sequence dependencies (Aenugu, 2019).

1. Hierarchical Memory Architecture

The core innovation of Γ-LSTM is the substitution of the standard LSTM cell state $c_t$ with a gamma-memory of order $K$, which consists of $K+1$ internal sub-cells $[c_0(t), c_1(t), ..., c_K(t)]$. Each sub-cell $c_k(t)$ integrates information from a different temporal abstraction level. This arrangement enables the model to function as a learned leaky integrator at each stage, systematically blending new information with longer-term representations.

The architecture allows information to propagate upward through the sub-cell hierarchy, with a softmax "attention" mechanism selecting the appropriate mixture across these hierarchical sub-cells. The resulting aggregated memory $c_t$ acts as the effective cell state and is used in the computation of the hidden state as in standard LSTMs.

Visually, this structure can be conceptualized as a single LSTM cell embedding multiple memory layers of increasing temporal coarseness—effectively arranging memory in a quasi-hierarchical stack within a single recurrent cell rather than across multilayer networks (Aenugu, 2019).

2. Gating Mechanisms and State Update Equations

The Γ-LSTM retains the core LSTM gating mechanisms but generalizes the forget-gate paradigm via adaptive, hierarchy-selecting gamma gates for each memory level. For input $x_t$ at time $t$ and previous hidden state $h_{t-1}$, the mathematical formalism proceeds as follows:

1. Standard LSTM Gates:

$$\begin{align*} i_t &= \sigma(W_{x,i} x_t + W_{h,i} h_{t-1} + b_i) \ g_t &= \tanh(W_{x,g} x_t + W_{h,g} h_{t-1} + b_g) \ o_t &= \sigma(W_{x,o} x_t + W_{h,o} h_{t-1} + b_o) \end{align*}$$

2. Gamma Gates: For each hierarchy level $k=1,\ldots,K$:

$$f_{k,t} = \sigma(W_{x,f_k} x_t + W_{h,f_k} h_{t-1} + b_{f_k})$$

These serve as learnable, input-dependent mixing coefficients, generalizing the role of fixed time constants in traditional gamma memory models.

3. Sub-cell Updates:

$$\begin{align*} c_0(t) &= i_t \odot g_t \ c_k(t) &= (1 - f_{k,t}) \odot c_k(t-1) + f_{k,t} \odot c_{k-1}(t-1), \quad k=1,...,K \end{align*}$$

4. Softmax Attention over Sub-cells:

$$\begin{align*} u_k &= v^\top c_k(t) + d_k \ a_k &= \frac{\exp(u_k)}{\sum_{j=0}^K \exp(u_j)} \end{align*}$$

The coefficients $[a_0, ..., a_K]^T$ sum to one and indicate the relative weight for each hierarchy level in the aggregated output.

5. Aggregated State and Output:

$$\begin{align*} c_t &= \sum_{k=0}^K a_k c_k(t) \ h_t &= o_t \odot \tanh(c_t) \end{align*}$$

3. Functional Role of Gamma Gates

Gamma gates $f_{k,t}$ in Γ-LSTM generalize the concept of a fixed time constant $\mu$ from classical gamma-memory models (De Vries & Principe ’92) to a learnable, input-selective function. Rather than statically blending prior and lower-level sub-cell states, the gamma gates determine at each time step how much each sub-cell should update with new information versus retain its historical content.

This dynamic allocation allows the network to adaptively distribute information flow: transient features (such as image edges in MNIST) preferentially update the finest-scale memory $c_0$, while more slowly varying or abstracted features percolate into higher-k sub-cells. This results in enhanced long-term retention for features with extended temporal dependencies and enables the cell to multiplex memory at multiple scales (Aenugu, 2019).

4. Training Regimen and Optimization

Γ-LSTM is trained on supervised classification tasks using the categorical cross-entropy loss. For sequence learning, backpropagation through time (BPTT) is employed, and the Adam optimizer with a typical learning rate of approximately $0.001$ is effective. RMSProp is also suitable.

To maintain numerical stability, gradients are clipped at a global norm (e.g., 5.0) when processing long sequences. Regularization strategies include input- and inter-layer dropout, particularly in conjunction with multilayer perceptrons as classifiers (as used in the SNLI experiments), and L$_2$ weight decay on recurrent weights, although weight decay is not strictly necessary for moderate memory orders (e.g., $K=3$). The SNLI classifier employs a four-layer ReLU-activated multilayer perceptron with dropout, while overfitting is monitored and addressed accordingly (Aenugu, 2019).

5. Empirical Evaluation: MNIST and SNLI Benchmarks

Γ-LSTM is evaluated on benchmark datasets for image sequence and natural language inference tasks:

Pixel-by-pixel MNIST

Each $28\times28$ image is linearized into a sequence of $784$ steps (with an alternate reshaping into $7$ subsequences of $112$ steps). Comparative models include:

• Vanilla LSTM (1 layer)
• Stacked LSTM (2 and 3 layers)
• Γ-LSTM (memory order $K=3$)

After 10 epochs, test set accuracies and parameter counts are as follows:

Model	Test Accuracy (%)	Parameter Count (input size=7, h-size≈128)
LSTM (1 layer)	93.52	71,434
LSTM (2 layers)	96.73	203,530
LSTM (3 layers)	96.95	335,626
Γ-LSTM (K=3)	97.94	123,018

Γ-LSTM with a single cell employing $K=3$ sub-cells surpasses both 2- and 3-layer stacked LSTMs in accuracy and parameter efficiency.

SNLI Natural Language Inference

Premise–hypothesis pairs are encoded via a bidirectional Γ-LSTM (K=3) or standard/stacked LSTM, then classified by a four-layer MLP.

Model	Test Accuracy (%)
LSTM (1 layer)	72.27
LSTM (2 layers)	71.96
Γ-LSTM (K=3)	73.29

Γ-LSTM demonstrates both improved performance and convergence, particularly in settings requiring modeling of long and complex sequences (Aenugu, 2019).

6. Hierarchical Memory Analysis and Comparative Insights

Qualitative analysis indicates that even with only $K=3$ sub-cells, Γ-LSTM achieves or exceeds the representational capacity of multi-layer stacked LSTMs. The internal gamma-memory provides a compact, efficient alternative to conventional deep stacking by enabling a single cell to manage information at multiple timescales.

The learned dynamic allocation of memory bandwidth, via the gamma gates $f_{k,t}$, allows for selective retention and updating. This mechanism is particularly effective in sequence domains such as pixelwise image processing or sentence encoding, where spatial or syntactic structure is distributed over a wide range of temporal dependencies. A plausible implication is that the incorporation of hierarchical memory within the recurrent unit itself renders deep stacking of LSTM layers less essential for tasks demanding long-term sequence modeling (Aenugu, 2019).

7. Summary of Theoretical and Practical Contributions

Γ-LSTM introduces a memory architecture wherein a cascade of learned time-scales, governed by hierarchy-selecting gamma gates and aggregated by soft-attention, equips the recurrent unit with adaptive depth and multi-scale memory. In empirical evaluation, this design yields more rapid convergence, improved generalization on long sequences, and reduced parameter count compared to deep stacks of conventional LSTMs. The core contribution lies in replacing flat cell-state dynamics with a hierarchical, attention-weighted memory, making Γ-LSTM a robust candidate for temporal abstraction tasks in sequence learning (Aenugu, 2019).