Reduced-Gate ConvLSTM (rgcLSTM)

• Reduced-gate ConvLSTM (rgcLSTM) is a deep learning architecture that merges LSTM’s input, forget, and output gates into a single convolutional gate for efficient spatiotemporal sequence modeling.
• The design achieves approximately 37.5% parameter reduction and significant reductions in training time and memory usage while maintaining or improving prediction accuracy.
• Empirical evaluations on Moving MNIST and KITTI datasets demonstrate the rgcLSTM’s practical advantages for real-time video prediction and resource-constrained deployments.

The reduced-gate convolutional Long Short-Term Memory (rgcLSTM) architecture is a specialized recurrent neural unit for spatiotemporal sequence modeling, notably next-frame prediction in video, which achieves parameter and computational efficiency by merging the canonical LSTM’s three gates (input, forget, output) into a single multi-function convolutional gate. This design preserves the essential recurrent-memory dynamics of the LSTM, including the constant-error carousel (CEC) and nonlinear output activation, while sharing convolutional weights across the traditional gate functions. Empirical evaluation within a predictive coding framework demonstrates that rgcLSTM matches or surpasses standard convolutional LSTM (convLSTM) performance in accuracy on video prediction benchmarks while enabling substantial reductions in model size, training time, and memory requirements (Elsayed et al., 2018).

1. Architectural Formulation

The rgcLSTM cell operates on the following input and state tensors at each time step $t$: input feature map $$x^{(t)} \in \mathbb{R}^{\eta\times\upsilon\times\gamma}$$, previous hidden state $$h^{(t-1)} \in \mathbb{R}^{\eta\times\upsilon\times n}$$, and previous cell state $$c^{(t-1)} \in \mathbb{R}^{\eta\times\upsilon\times n}$$, where $n$ is the number of output channels. The cell comprises two main submodules:

• Single Multi-Function Gate:

The module-gate preactivation is computed via

$$a^{(t)} = W_{fx}*x^{(t)} + U_{fh}*h^{(t-1)} + W_{fc}*c^{(t-1)} + b_f$$

Stacking the terms:

$a^{(t)} = W_f * I_f + b_f$

with $W_f = [W_{fx}, U_{fh}, W_{fc}]$ and $I_f = [x^{(t)}, h^{(t-1)}, c^{(t-1)}]$, where $*$ denotes “same” convolution. This yields the gate activation:

$$x^{(t)} \in \mathbb{R}^{\eta\times\upsilon\times\gamma}$$0

where $$x^{(t)} \in \mathbb{R}^{\eta\times\upsilon\times\gamma}$$1 is typically a hard sigmoid.

• Input Update (Memory Activation):

With $$x^{(t)} \in \mathbb{R}^{\eta\times\upsilon\times\gamma}$$2 and $$x^{(t)} \in \mathbb{R}^{\eta\times\upsilon\times\gamma}$$3, compute

$$x^{(t)} \in \mathbb{R}^{\eta\times\upsilon\times\gamma}$$4

Cell updates are then defined by:

$$x^{(t)} \in \mathbb{R}^{\eta\times\upsilon\times\gamma}$$5

$$x^{(t)} \in \mathbb{R}^{\eta\times\upsilon\times\gamma}$$6

using the single gate $$x^{(t)} \in \mathbb{R}^{\eta\times\upsilon\times\gamma}$$7 for all classical gate roles. The parameter count for an $$x^{(t)} \in \mathbb{R}^{\eta\times\upsilon\times\gamma}$$8 kernel, input depth $$x^{(t)} \in \mathbb{R}^{\eta\times\upsilon\times\gamma}$$9, and output depth $$h^{(t-1)} \in \mathbb{R}^{\eta\times\upsilon\times n}$$0 is:

$$h^{(t-1)} \in \mathbb{R}^{\eta\times\upsilon\times n}$$1

By comparison, a standard convLSTM cell with three gates and input-update uses:

$$h^{(t-1)} \in \mathbb{R}^{\eta\times\upsilon\times n}$$2

When $$h^{(t-1)} \in \mathbb{R}^{\eta\times\upsilon\times n}$$3, the rgcLSTM reduces parameter count per cell by approximately 37.5%.

2. Integration with Predictive Coding Architectures

The rgcLSTM replaces the convLSTM as the recurrent (“R”) module in the PredNet predictive coding framework. At each layer $$h^{(t-1)} \in \mathbb{R}^{\eta\times\upsilon\times n}$$4:

• $$h^{(t-1)} \in \mathbb{R}^{\eta\times\upsilon\times n}$$5 denotes the current ground-truth signal (in layer 0, the video frame).
• $$h^{(t-1)} \in \mathbb{R}^{\eta\times\upsilon\times n}$$6 is the rgcLSTM, consuming concatenated error $$h^{(t-1)} \in \mathbb{R}^{\eta\times\upsilon\times n}$$7 (bottom-up), previous hidden state $$h^{(t-1)} \in \mathbb{R}^{\eta\times\upsilon\times n}$$8, and feedback $$h^{(t-1)} \in \mathbb{R}^{\eta\times\upsilon\times n}$$9 (top-down).
• The rgcLSTM output $$c^{(t-1)} \in \mathbb{R}^{\eta\times\upsilon\times n}$$0 is processed by a $$c^{(t-1)} \in \mathbb{R}^{\eta\times\upsilon\times n}$$1 convolution to yield the prediction $$c^{(t-1)} \in \mathbb{R}^{\eta\times\upsilon\times n}$$2.
• Error modules compute:

$$c^{(t-1)} \in \mathbb{R}^{\eta\times\upsilon\times n}$$3

$$c^{(t-1)} \in \mathbb{R}^{\eta\times\upsilon\times n}$$4

The global loss for next-frame prediction is a weighted sum of squared layer-wise errors:

$$c^{(t-1)} \in \mathbb{R}^{\eta\times\upsilon\times n}$$5

Best performance is achieved with weighting $$c^{(t-1)} \in \mathbb{R}^{\eta\times\upsilon\times n}$$6, corresponding to lowest-layer loss only.

3. Empirical Evaluation

The rgcLSTM was benchmarked within PredNet on the Moving MNIST and KITTI datasets.

A. Moving MNIST

• Dataset: 10,000 sequences (20 frames each, $$c^{(t-1)} \in \mathbb{R}^{\eta\times\upsilon\times n}$$7).
• Model: 4 layers, feature depths $$c^{(t-1)} \in \mathbb{R}^{\eta\times\upsilon\times n}$$8.
• Hyperparameters: Adam ($$c^{(t-1)} \in \mathbb{R}^{\eta\times\upsilon\times n}$$9, $n$0, $n$1).
• Results:

Model	Total Params	Training Time (min)	MSE	MAE	SSIM
PredNet (convLSTM)	6,909,834	100.3	0.011	0.049	0.915
Pred-rgcLSTM	4,316,235	73.6	0.009	0.017	0.924

• Parameter Reduction: 37.5% fewer parameters; training time reduced by 26.7%.
• Qualitative Results: Sharper digit contours with rgcLSTM predictions.

B. KITTI Traffic

• Dataset: Real-world driving frames, resized to $n$2.
• Model: 4 layers, feature depths $n$3, 150 epochs.
• Results:

Model	Total Params	Training Time (min)	MSE	MAE	SSIM
PredNet (convLSTM)	6,915,948	230.98	0.0044	0.0347	0.911
Pred-rgcLSTM	4,320,273	140.53	0.0035	0.0300	0.932

• Parameter Reduction: 37.5%; training time reduced by 39%.
• Both architectures exhibit nearly identical training-validation loss curves, confirming dynamical equivalence.

4. Comparative Analysis and Gate Design Ablation

A systematic ablation of 20 gate variants (14 single-function, 6 multi-function, with/without peepholes) was conducted for insight into gating tradeoffs:

• Top SSIM models on Moving MNIST:
  • M1 (cLSTM no peepholes): 0.915
  • M8 (convLSTM with peepholes): 0.913
  • M15 (np-rgcLSTM, 1 gate/no peephole): 0.919
  • M18 (rgcLSTM, 1 gate + conv peephole): 0.924

A two-way ANOVA (factors: peephole yes/no, gate function number) identified a significant interaction ($n$4):

• Three-function gate models consistently outperformed one- or two-function gate variants.
• Convolutional peepholes degrade 3-gate model performance but benefit single multi-function gate models.
• On KITTI, convolutional peephole no longer impairs convLSTM.

Post-hoc analyses demonstrate that 3-function gate models statistically outperform all 3-gate architectures on these tasks.

5. Hardware Efficiency and Implementation Considerations

The merged-gate structure confers distinct hardware and deployment advantages:

• ~40% parameter reduction per layer decreases on-chip SRAM/DRAM demand.
• Corresponding ~40% reduction in memory bandwidth for weight fetches.
• Peak multiply–accumulate operations reduced (only 3 convolutions, down from 8 per cell in convLSTM), resulting in up to 25% faster training and inference.
• These properties address the constraints of embedded and mobile systems, making rgcLSTM suitable for deployment in automotive vision and other low-power contexts where memory and compute are at a premium.

6. Significance and Broader Impact

The rgcLSTM architecture demonstrates that the core computational benefit of LSTM-style recurrence can be realized with substantially fewer parameters and convolutions, provided gate roles are shared and a single convolutional peephole is employed. Empirical results suggest no trade-off in predictive accuracy for standard video modeling benchmarks. The ablation study’s delineation of gating mechanisms offers a principled basis for future recurrent cell design, particularly for resource-limited hardware platforms. These findings position reduced-gate architectures as a compelling alternative for real-time spatiotemporal processing (Elsayed et al., 2018).