MIPAE Model: Disentangled Video Prediction

• MIPAE Model is a framework for unsupervised disentanglement that separates video frames into static content and dynamic pose codes using a mutual information penalty.
• The model utilizes dedicated encoders and an LSTM-based pose predictor to generate sharp, temporally coherent future video frames.
• Empirical evaluations demonstrate improved metrics such as MIG, SSIM, PSNR, and LPIPS compared to approaches like DRNET.

The Mutual Information Predictive Auto-Encoder (MIPAE) is a framework for unsupervised disentanglement of video representations, targeting the efficient prediction of future video frames by explicitly factorizing latent variables into time-invariant content and low-dimensional, time-dependent pose codes. MIPAE leverages a mutual information penalty, rather than adversarial methods, to enforce disentanglement between static and dynamic components, facilitating improved long-range video prediction, sharper qualitative reconstructions, and superior quantitative metrics compared to previous approaches such as DRNET (Sreekar et al., 2020).

1. Factorized Latent Representation and Model Architecture

MIPAE decomposes each video frame $x^t$ into two complementary latent variables:

• Content vector $z_{\mathrm{c}}$: A time-independent vector in $\mathbb{R}^{c}$, representing static object attributes over the sequence.
• Pose vector $z_{\mathrm{p}}^t$: A low-dimensional, time-dependent vector in $\mathbb{R}^{p}$, capturing temporal and motion-related aspects.

The architecture consists of three principal modules:

• Content encoder ($E_{\mathrm{c}}$): Maps each observed frame $x^t$ into a single sequence-level content code $z_{\mathrm{c}} = E_{\mathrm{c}}(x^t)$.
• Pose encoder ($E_{\mathrm{p}}$): Encodes each frame $x^t$ into its specific pose code $z_{\mathrm{c}}$0.
• Decoder ($z_{\mathrm{c}}$1): Decodes the content and pose code pairs to reconstruct or generate pixel frames, $z_{\mathrm{c}}$2.

At training time, similarity between content codes across time is enforced ($z_{\mathrm{c}}$3 is sequence-constant), pose codes between timesteps are regularized using a mutual information loss, and all frames are reconstructed with pixel-wise losses. During prediction, the LSTM pose predictor $z_{\mathrm{c}}$4 (two layers, 256 units each) generates future pose sequences conditioned on the last observed pose and content code, with predicted frames reconstructed by the decoder.

2. Mathematical Specification of Module Operations

2.1 Encoder and Decoder Operations

For each sequence:

• Content:
  • $z_{\mathrm{c}}$5 (kept constant per sequence)
• Pose:
  • $z_{\mathrm{c}}$6
• Decoding:
  • $z_{\mathrm{c}}$7

2.2 LSTM Pose Prediction

For time indices $z_{\mathrm{c}}$8 (where $z_{\mathrm{c}}$9 is the number of context frames), the pose code is recursively generated:

• For $\mathbb{R}^{c}$0: $\mathbb{R}^{c}$1
• For $\mathbb{R}^{c}$2: $\mathbb{R}^{c}$3
• Recursive prediction: $\mathbb{R}^{c}$4

2.3 Reconstruction Loss

Frame-wise reconstruction is optimized using

$\mathbb{R}^{c}$5

3. Mutual Information-Based Disentanglement Mechanism

The central advance of MIPAE is the replacement of DRNET's adversarial content-pose confusion by a direct penalty on the mutual information between pose vectors at different times:

• Mutual Information Definition:

$\mathbb{R}^{c}$6

• Critic Network ($\mathbb{R}^{c}$7): A 2-layer MLP estimates the joint and marginal likelihoods by distinguishing between paired $\mathbb{R}^{c}$8 samples from the same sequence ("joint") and randomly paired pose samples ("marginal"). The critic is trained with a GAN-style objective,

$\mathbb{R}^{c}$9

where $z_{\mathrm{p}}^t$0 denotes the sigmoid function.

• Jensen–Shannon MI Lower Bound:

$z_{\mathrm{p}}^t$1

Minimizing $z_{\mathrm{p}}^t$2 reduces shared information between temporally distant pose vectors, enforcing that pose captures only time-varying factors.

4. Mutual Information Gap (MIG) Metric for Disentanglement

The effectiveness of the model in disentangling latent factors is assessed by an adaptation of the Mutual Information Gap (MIG) metric:

$z_{\mathrm{p}}^t$3

Here $z_{\mathrm{p}}^t$4 and $z_{\mathrm{p}}^t$5 denote the ground-truth content and pose factors, respectively, and $z_{\mathrm{p}}^t$6 is entropy. High MIG values indicate effective disentanglement: content codes are informative for static content and independent of pose, and vice versa.

5. Optimization Procedure and Network Hyperparameters

The training procedure alternates between two steps:

• Step A: Minimize a total loss over $z_{\mathrm{p}}^t$7, $z_{\mathrm{p}}^t$8, and $z_{\mathrm{p}}^t$9:

$\mathbb{R}^{p}$0

with content-invariance loss $\mathbb{R}^{p}$1.

• Step B: Update the critic $\mathbb{R}^{p}$2 via maximization of its adversarial objective.

Key hyperparameters:

• $\mathbb{R}^{p}$3, $\mathbb{R}^{p}$4
• Learning rate: $\mathbb{R}^{p}$5 with Adam ($\mathbb{R}^{p}$6)
• Batch size: typically 64
• Pose dimension $\mathbb{R}^{p}$7: 5 (MNIST, DSprites), 10 (MPI3D)
• Content dimension $\mathbb{R}^{p}$8: 128

Architectural choices:

• MNIST & DSprites: 4-layer DCGAN-style convolutional encoders; decoder as mirrored deconvolutional net.
• MPI3D-Real: Pose encoder as ResNet-18; content encoder/decoder as VGG-16 with U-Net skip-connections.
• Critic: 2-layer MLP, 512 units per layer.
• LSTM pose predictor: 2 layers, 256 units each.

6. Empirical Results and Comparative Analysis

MIPAE is evaluated across three standard datasets: Moving DSprites, MPI3D-Real, and Moving MNIST.

Summary of Quantitative Results

Dataset	MIG (MIPAE)	MIG (DRNET)	SSIM ↑	PSNR ↑	LPIPS ↓
Moving DSprites	0.8975	0.8574	0.85 (vs 0.75)	1–2 dB higher	0.12 (vs 0.18)
MPI3D-Real	0.6126	0.5658	Similar trends	Higher	Lower
Moving MNIST	–	–	Qualitatively sharper	Outperforms beyond 5 frames	Outperforms beyond 5 frames

Note: Higher SSIM and PSNR, and lower LPIPS, indicate better-quality predictions.

Qualitative Observations

Latent swaps, such as combining the content vector from one sequence and the pose from another, yield coherent frame generations where static object identity is preserved, and motion is transferred without distortion. MIPAE demonstrates increased stability for longer-range frame prediction, producing visually sharper and more temporally coherent generations than DRNET under analogous settings.

A plausible implication is that by enforcing minimal mutual information between temporally distant pose vectors through a neural MI-estimator, MIPAE achieves a structural disentanglement that is empirically advantageous for unsupervised video prediction tasks.

7. Distinctive Advances and Relationship to Prior Work

MIPAE’s main contribution lies in shifting disentanglement enforcement from adversarial confusion (as in DRNET) to an information-theoretic constraint, explicitly minimizing $\mathbb{R}^{p}$9 via a neural estimator. This mechanism consistently yields improved MIG scores and superior video prediction quality on both synthetic and real-world datasets (Sreekar et al., 2020). The factorization approach with explicit mutual information minimization addresses a central challenge in video modeling: separating static content from dynamic variation without supervision.

MIPAE’s evaluation against established metrics (MIG, SSIM, PSNR, LPIPS), across diverse datasets, substantiates its effectiveness in unsupervised disentanglement and long-range video forecasting, establishing a new benchmark for models based on factorized latent representation and mutual information regularization.