Stability Enhanced Gaussian Process Variational Autoencoders

Abstract: A novel stability-enhanced Gaussian process variational autoencoder (SEGP-VAE) is proposed for indirectly training a low-dimensional linear time invariant (LTI) system, using high-dimensional video data. The mean and covariance function of the novel SEGP prior are derived from the definition of an LTI system, enabling the SEGP to capture the indirectly observed latent process using a combined probabilistic and interpretable physical model. The search space of LTI parameters is restricted to the set of semi-contracting systems via a complete and unconstrained parametrisation. As a result, the SEGP-VAE can be trained using unconstrained optimisation algorithms. Furthermore, this parametrisation prevents numerical issues caused by the presence of a non-Hurwitz state matrix. A case study applies SEGP-VAE to a dataset containing videos of spiralling particles. This highlights the benefits of the approach and the application-specific design choices that enabled accurate latent state predictions.

• The paper introduces a SEGP-VAE framework that leverages a semi-contracting GP prior to guarantee bounded and physically realistic latent dynamics.
• It employs an unconstrained parametrization of the state transition matrix to enable efficient gradient-based optimization while avoiding numerical instabilities.
• Empirical results demonstrate superior uncertainty propagation and latent trajectory recovery compared to standard GP kernels on complex sequential datasets.

Stability Enhanced Gaussian Process Variational Autoencoders: A Technical Summary

Introduction

The paper "Stability Enhanced Gaussian Process Variational Autoencoders" (2604.09331) introduces the SEGP-VAE, a framework for incorporating stability-driven inductive biases into GP-regularized VAEs, targeting the unsupervised recovery of interpretable and stable low-dimensional latent dynamics from high-dimensional observations. Specifically, the SEGP prior is designed by encoding the state transition structure of semi-contracting LTI systems, thereby constraining the feasible set of learned dynamics to those that guarantee boundedness and physical realism. A complete, unconstrained parametrization is constructed to facilitate efficient and robust learning via gradient-based optimizers and avoid instability-related numerical pathologies.

Theoretical Construction of SEGP Priors

A central methodological advancement is the derivation of a GP prior whose mean and covariance are exactly those of the output of a semi-contracting LTI system driven by a Gaussian process input. The general LTI operator is parameterized as:

$$\dot{\mathbf{x}}(t) = A\mathbf{x}(t) + B\mathbf{u}(t), \quad \mathbf{y}(t) = C\mathbf{x}(t) + D\mathbf{u}(t),$$

where $\mathbf{u}(t)$ is a Gaussian process, the initial condition $\mathbf{x}(0)$ is Gaussian, and the state matrix $A$ is explicitly enforced to satisfy the semi-contraction LMI $PA + A^\top P \preceq 0$ for some $P \succ 0$. This ensures eigenvalues of $A$ lie in the closed left half-plane, preventing unbounded latent excursions.

The mean and covariance of the output $\mathbf{y}(t)$ are calculated using the LFM framework, incorporating both transient (initial state) and input-driven components. The crucial technical element is an explicit parametrization of the set of all semi-contracting $A$ matrices in terms of unconstrained latent variables, accomplished via block-triangular, skew-symmetric, and diagonal factors. This enables the SEGP-VAE to be trained in an entirely unconstrained manner, side-stepping issues associated with direct semi-definite programming or projection methods.

Figure 1: The SEGP-VAE architecture integrates a CNN encoder, SEGP prior with semi-contracting parametrization, and a simple MLP-based decoder.

SEGP-VAE Framework

The SEGP prior is embedded within a VAE framework, operating on high-dimensional sequential data (e.g., videos). The encoder consists of a CNN with a spatial-softmax-to-polar latent representation appropriate for systems with intrinsic rotational degrees of freedom. Subsequent steps compute polar coordinates $(r, \theta)$, followed by unwrapping $\mathbf{u}(t)$0 across time to ensure smoothness in the latent evolution, allowing for effective learning of circular or phase dynamics.

The decoder is kept intentionally shallow (2-layer MLP with batch normalization), mapping the latent sequence $\mathbf{u}(t)$1 back into the observed image space using a Bernoulli likelihood. Training is performed by optimizing an augmented ELBO with $\mathbf{u}(t)$2-regularization on the KL term and an $\mathbf{u}(t)$3 penalty on $\mathbf{u}(t)$4, promoting model sparsity and interpretability.

The variational posterior is implemented via amortized inference, with per-frame Gaussian likelihoods for the latent path, enabling tractable GP inference over sequences.

Empirical Evaluation

The approach is validated on a synthetic particle-in-plane dataset, with ground-truth latent dynamics given by a 2D LTI (radius, angle), a known input-output structure, and Gaussian process input excitation. The key empirical findings include:

• Learning Dynamics with Stability Guarantees: The SEGP prior consistently produces bounded mean and covariance trajectories, and the system matrix $\mathbf{u}(t)$5 remains within the semi-contracting set throughout training.
• Superior Covariance Modeling: Comparing with standard multi-output squared exponential GPs, the SEGP can model long-term uncertainty propagation (e.g., input-integrated angular variance) accurately, while traditional SE kernels fail to capture such non-stationarity.

  Figure 2: The learned SEGP prior covariance (left) encodes correct cross-variable and time-dependent variance, outperforming a standard SE kernel (right) in physical fidelity.

• Training Dynamics: The reconstruction loss and ELBO converge smoothly with no signs of mode collapse; the KL and sparsity regularizers maintain active constraints throughout training.

  Figure 3: Per-pixel reconstruction error rapidly falls during training, indicating effective amortized inference.

  

  Figure 4: KL divergence and L1 norm of $\mathbf{u}(t)$6 tracked during training; KL remains non-trivial, demonstrating a non-degenerate posterior.

• Posterior Identification and Latent Trajectory Recovery: Posterior means from the trained SEGP-VAE closely track ground-truth latent paths with dramatically reduced uncertainty, showing the model is able to invert high-dimensional video observations to accurate, interpretable low-dimensional state estimates.

  Figure 5: Test videos, associated latent trajectories, and reconstructed sequences, demonstrating accurate latent inversion and generation.

  

  Figure 6: Posterior means (bottom) fit the latent trajectories (ground-truth) with high confidence (narrow variance), illustrating the benefit of the Bayesian update over the prior (top).

• Quantitative Analysis: The absolute error between posterior mean and ground-truth for angular position is minimized across the test set, and the learned $\mathbf{u}(t)$7 matrix approaches the ground-truth with negligible spectral norm error.

  Figure 7: Left: posterior mean error for angular position; right: posterior predictive variance, both significantly outperform the prior.

Practical and Theoretical Implications

The SEGP-VAE framework enables robust recovery of physically plausible and interpretable latent dynamics from high-dimensional sequential data while providing intrinsic guarantees of stability and numerical reliability absent in unconstrained or generic kernel-based models. From a practical perspective, this is significant for scientific domains (e.g., material science via in situ video analysis) where physics-informed priors are necessary for both generalization and control-relevance. On a theoretical level, the unconstrained parametrization of the semi-contracting dynamic family establishes a new tool for reliable amortized inference in structured SSMs.

The comparison with standard GPs also highlights clear failure modes in existing black-box approaches for physical time series—specifically, their inability to propagate process noise and initial condition uncertainty according to underlying system structure without explicit modeling.

Outlook and Future Developments

The main limitation encountered is the necessity of prior knowledge of input-output structure for identifiability and learning stability in practice. Further work is anticipated in relaxing this requirement and generalizing the SEGP to broader parametric and nonparametric dynamical families (potentially via the Koopman operator framework). The current methodology is poised for adaptation to more complex nonlinear systems by leveraging finite-dimensional linearizable embeddings.

Conclusion

The Stability Enhanced Gaussian Process Variational Autoencoder advances structured unsupervised learning of time series by formulating a tractable, stable, and interpretable latent prior based on LTI semi-contraction theory. The new parametrization ensures numerical stability and encourages physical plausibility, empirically outperforming standard GP baselines in latent recovery, uncertainty quantification, and generative fidelity. The implications are broad for high-dimensional scientific inference, with principled routes for controlling and understanding complex systems through learned latent dynamics.