Video Ladder Network

• Video Ladder Network is a neural video prediction model that uses an encoder–decoder framework with both feedforward and recurrent lateral connections to capture spatial and temporal details.
• Its architecture employs dilated convolutions, convLSTM modules, and ResNet-style residual blocks to enhance prediction accuracy and computational efficiency.
• Empirical results on Moving MNIST demonstrate that VLN reduces test BCE by up to 9% compared to baselines, with extended versions improving location-dependent performance.

The Video Ladder Network (VLN) is a neural video prediction model that employs an encoder–decoder structure augmented by both recurrent and feedforward lateral connections at all layers. Its architecture is designed to efficiently capture both spatial and temporal information at multiple resolutions, with explicit mechanisms to propagate detail and context across layers and time. VLN has been applied primarily to self-supervised next-frame prediction tasks and achieves competitive performance on standard video benchmarks, with extensions for increased depth and spatial sensitivity further enhancing its capabilities (Cricri et al., 2016, Azizi et al., 2018).

1. Architectural Design and Principles

VLN utilizes a fully convolutional encoder–decoder framework in which, at every layer $\ell$, two lateral pathways connect encoder and decoder: a feedforward "skip connection" and a recurrent "residual" realized via a convolutional LSTM (convLSTM). The encoder consists of multiple dilated convolutional layers, each followed by batch normalization and leaky-ReLU activation, forming feature maps at increasing levels of abstraction. The decoder mirrors this structure with upsampling and convolutional layers to reconstruct or predict future frames.

At each layer, the decoder's input is produced by merging the corresponding encoder feature, the temporal summary from the convLSTM, and the top-down signal from the higher decoder layer. This merging operation typically consists of a 1×1 convolution after channel concatenation, followed by a nonlinearity (Cricri et al., 2016).

VLN-ResNet extends this construction by employing ResNet-style residual blocks within both encoder and decoder. These blocks replace simple convolutional layers with stacks of two 3×3 convolutions followed by a skip-sum, with encoder blocks utilizing varied dilations and decoder blocks beginning with upsampling (Cricri et al., 2016).

2. Mathematical Formulation

Let $x_t \in [0,1]^{H \times W}$ denote the input frame at time $t$. Encoder activations at layer $\ell$ are computed as

$z_t^\ell = f_\ell(h_t^{\ell-1})$

where $h_t^0 = x_t$, and $f_\ell$ is a dilated convolution followed by batch normalization and leaky-ReLU.

Temporal context is introduced through convLSTM units at each layer: \begin{align*} (h_t^\ell, c_t^\ell) &\leftarrow \mathrm{convLSTM}\ell(z_t^\ell, h{t-1}^\ell, c_{t-1}^\ell) \end{align*} with gate updates as: \begin{align*} i_t^\ell &= \sigma(z_t^\ell * W_{zi}^\ell + h_{t-1}^\ell * W_{hi}^\ell + b_i^\ell) \ f_t^\ell &= \sigma(z_t^\ell * W_{zf}^\ell + h_{t-1}^\ell * W_{hf}^\ell + b_f^\ell) \ o_t^\ell &= \sigma(z_t^\ell * W_{zo}^\ell + h_{t-1}^\ell * W_{ho}^\ell + b_o^\ell) \ \tilde{c}t^\ell &= \tanh(z_t^\ell * W{z\tilde{c}}^\ell + h_{t-1}^\ell * W_{h\tilde{c}}^\ell + b_{\tilde{c}}^\ell) \ c_t^\ell &= f_t^\ell \odot c_{t-1}^\ell + i_t^\ell \odot \tilde{c}_t^\ell \ h_t^\ell &= o_t^\ell \odot \tanh(c_t^\ell) \end{align*} where $*$ denotes 2D convolution, $\odot$ is the element-wise product, and $\sigma$ is the sigmoid.

In the decoder, the merged activation at layer $\ell$ at time $t+1$ is given by

$$\hat{z}_{t+1}^\ell = \mathrm{LReLU}(W_h^\ell * h_t^\ell + W_z^\ell * z_t^\ell + \tilde{z}^{\ell+1}_{t+1})$$

with $W_h^\ell$ and $W_z^\ell$ as 1×1 convolution kernels, and $\tilde{z}_{t+1}^{\ell+1}$ the top–down input from the above decoder layer. The output prediction $\hat{x}_{t+1}$ is generated by applying a final sigmoid to $\hat{z}^1_{t+1}$ (Cricri et al., 2016, Azizi et al., 2018).

3. Training Procedures and Objectives

VLN is trained in a self-supervised regime on video frame sequences. The loss is the pixelwise binary cross-entropy:

$$\mathcal{L} = -\frac{1}{T H W} \sum_{t=1}^T \sum_{i,j} \left[ x_{ij}^t \log \hat{x}_{ij}^t + (1 - x_{ij}^t)\log(1 - \hat{x}_{ij}^t) \right]$$

Each training batch involves predicting a fixed number of future frames (e.g., next 10 frames given the preceding 10) and averages the loss over all predictions.

Optimization uses RMSprop with layer- and model-specific learning rates (e.g., $1\times10^{-4}$ for basic VLN, $5\times10^{-4}$ for VLN-ResNet) and extended training over thousands of epochs. Mini-batch generation is performed on-the-fly with Moving MNIST, and hidden states for the convLSTM units are preserved across frames in a training sequence but reset between test sequences.

No additional regularization or auxiliary objectives are used; improvements stem directly from the network’s architecture and training objective (Cricri et al., 2016).

4. Empirical Results and Comparative Analysis

On the Moving MNIST dataset, VLN achieves strong performance in next-frame prediction as measured by test-set binary cross-entropy (BCE), using considerably fewer parameters and inference steps than autoregressive generative models. Key results from (Cricri et al., 2016) are:

Model	Test BCE	Depth	Params
conv-LSTM [Shi et al.]	367.1	–	7.6 M
fc-LSTM [Srivastava et al.]	341.2	2	143 M
DFN [Brabandere et al.]	285.2	–	0.6 M
VPN-BL [Kalchbrenner et al.]	110.1	81	30 M
VLN-BL (basic)	222.3	7	1.2 M
VLN-BL-FF	220.1	8	1.2 M
VLN	207.0	9	1.2 M
VLN-ResNet	187.7	15	1.3 M

Relative to the baseline, the full VLN and its ResNet extension deliver $\sim$7% and $\sim$9% further reduction in BCE, respectively. Qualitative results indicate that VLN-ResNet maintains digit identity and sharpness over long-range (10-frame) predictions. VLN requires only one forward pass per time step, whereas pixelCNN-style models require orders of magnitude more iterations per frame (Cricri et al., 2016).

Enhanced spatial sensitivity through location-dependent biasing further reduces test BCE by $\sim$9-10% on occluded Moving MNIST, demonstrating that standard VLN architectures are limited by spatial invariance in modeling location-dependent events such as occlusions and boundary effects (Azizi et al., 2018).

5. Design Motivations, Lateral Connections, and Temporal Modeling

In VLN, convLSTM modules at every encoder (and corresponding decoder) layer enable explicit modeling of temporal dependencies at all spatial resolutions. Low-level convLSTM layers capture local pixel trajectories, while higher layers process more abstract, long-range dynamics. The lateral skip connection provides the decoder with the most current spatial features, while the convLSTM residual supplies a temporally aggregated summary. This dual-lateral pathway realizes a "recurrent residual block" at each layer.

A primary motivation is to offload detailed representation of low-level motion and appearance from higher layers, allowing these upper layers to concentrate on coarse, global predictions. Empirically, this separation of spatial and temporal responsibility improves both prediction accuracy and computational efficiency. The design also accelerates inference, eliminating the need for computationally intensive autoregressive pixel-wise generation (Cricri et al., 2016, Azizi et al., 2018).

6. Extensions for Location Dependency and Broader Impact

VLN’s original architecture, being fully convolutional, is spatially invariant and unable to model location-dependent dynamics. To address this, Azizi and Farazi (Azizi et al., 2018) introduced three schemes:

• VLN-AI (Added Input Channels): Augments the input with two ramp (positional) channels and an occluder mask; learnable weights then infer location dependence.
• VLN-LDC (Learnable Location-Dependent Bias): Employs learnable spatial bias maps in the first convolution of each encoder block.
• VLN-LDCAI (Location-Dependent Bias + Added Inputs): Combines input ramps and learnable bias for enhanced spatial priors.

Empirically, all three yield $\sim$9–10% lower test BCE than the spatially invariant baseline, with VLN-AI as the strongest individual enhancement (Azizi et al., 2018). This indicates that location biasing is crucial in video environments with fixed spatial cues (e.g., occlusion patterns, boundaries).

A plausible implication is that future video prediction architectures should integrate explicit spatial encoding mechanisms when scene geometry induces non-stationarities not capturable by standard convolutions.

7. Relation to Other Ladder-style and Recurrent Architectures

VLN shares core architectural philosophy with the Recurrent Ladder Network (RLadder) (Prémont-Schwarz et al., 2017), which generalizes the static feedforward Ladder structure into a temporally recurrent form. Both RLadder and VLN introduce per-layer recurrent states—implemented as ConvLSTM in VLN—and augment encoder–decoder paths with lateral connections. In RLadder, these lateral and recurrent signals enable multi-level, multi-iteration bottom-up/top-down inference, particularly useful for temporal modeling of video data and perceptual grouping tasks.

Empirical comparisons suggest that these ladder-style recurrent models outperform single-path, purely feedforward, or single-level recurrent baselines across video prediction, segmentation, and music modeling. Both achieve efficient spatiotemporal abstraction by facilitating bi-directional message passing and granular temporal-state retention (Prémont-Schwarz et al., 2017, Cricri et al., 2016).

---

References:

• "Video Ladder Networks" (Cricri et al., 2016)
• "Location Dependency in Video Prediction" (Azizi et al., 2018)
• "Recurrent Ladder Networks" (Prémont-Schwarz et al., 2017)