VAEs for Dynamical Systems

• Variational autoencoders for dynamical systems are generative models that extract low-dimensional representations from high-dimensional, nonlinear, and noisy time-series data.
• They employ a double projection framework to disentangle deterministic state dynamics from stochastic process noise, enhancing system reconstruction and forecasting.
• Teacher forcing strategies are integrated to stabilize multi-step predictions, ensuring robust simulation of both chaotic and noise-driven systems.

Variational autoencoders (VAEs) for dynamical systems are a family of generative latent variable models that learn to represent high-dimensional time-series data through a lower-dimensional latent space, while incorporating models of temporal evolution—often stochastic—either directly in the latent dynamics or via explicit dynamical priors. These approaches are designed for system identification, reconstruction, forecasting, denoising, and uncertainty quantification in physical, biological, and engineered systems where the true system state is only partially observed and the generative mechanisms may be highly nonlinear and stochastic. Recent research emphasizes the integration of variational inference strategies, double latent projections, and teacher forcing into sequential VAE frameworks to recover both latent trajectories and the underlying noise, enabling robust multi-step system simulation from limited or partial observations (Sip et al., 1 Oct 2025).

1. Double Projection VAE Framework for Stochastic Dynamical System Reconstruction

The Double Projection Dynamical System Reconstruction (DPDSR) methodology uses two parallel encoders within the VAE framework. One encoder estimates the sequence of latent system states $\hat{z}_{1:T}$ and the other estimates the latent noise sequence $\varepsilon_{1:T}$, conditioned on the full observed data.

The generative model implements stochastic, possibly nonlinear, state evolution: $$z_t = \tanh\left( f(z_{t-1}) + B \varepsilon_t \right)$$ where $f$ is a residual multilayer perceptron: $$f(z_{t-1}) = z_{t-1} + W_2 \sigma(W_1 z_{t-1} + b_1) + b_2$$ with ReLU nonlinearity $\sigma$, and a $\tanh$ nonlinearity to enforce boundedness of $z_t$.

The observation model is given by: $x_t = g(z_t) + \Sigma_{\eta} \eta_t$ where $g(\cdot)$ is a neural network emission model, and $\eta_t$ is additive measurement noise.

The VAE loss function has three components: $$L = L_\text{rec}^x + L_\text{rec}^{\hat{z}} + D_\mathrm{KL}\left( q(\varepsilon_{1:T} | x, \hat{z}_{1:T}) \parallel p(\varepsilon_{1:T}) \right)$$ where $L_\text{rec}^x$ is the observation reconstruction loss, $L_\text{rec}^{\hat{z}}$ is for the latent state, and the KL divergence regularizes the inferred noise sequence against a white noise prior.

This dual projection allows the model to allocate observed variability between deterministic structure (latent dynamics) and process noise, critical for capturing both chaotic and noise-driven systems.

2. Teacher Forcing Strategy in Sequential Stochastic VAEs

Teacher forcing is used to mitigate the compounding error endemic to multi-step evolution in sequential models trained primarily on one-step predictions. In DPDSR, at every $\tau$ steps, the evolving state trajectory $\tilde{z}_t$ is restarted (or partially re-initialized) from encoder-estimated state $\hat{z}_t$: $$\tilde{z}_{t+1} = \begin{cases} \tanh(f(\tilde{z}_t) + B\varepsilon_t), & t \bmod \tau \neq 0 \ \tanh(f([\hat{z}_t ; T_{d_{\hat{z}}:d_z}(\tilde{z}_t)]) + B\varepsilon_t), & t \bmod \tau = 0 \end{cases}$$ where $T_{d_{\hat{z}}:d_z}$ selects the unobserved components. This design ensures stable propagation and retains temporal fidelity in latent trajectories, particularly for long sequences or systems with strong stochasticity.

3. Empirical Evaluation Across Deterministic and Stochastic Dynamical Systems

DPDSR was systematically evaluated on six challenging benchmark settings, including:

• The Lorenz system and a 6D cell cycle model (deterministic, chaotic)
• A double-well system exhibiting noise-induced switching
• High-dimensional recurrent neural network (RNN) generated data with partial observability
• Experimental recordings from rat cortical neurons
• Electrocardiogram (ECG) datasets

Empirical findings:

• For deterministic chaotic systems, both stochastic (DPDSR) and deterministic (SPDSR, zero-noise variant) models perform comparably in short-term trajectory reconstruction, but DPDSR provides similar or superior distributional and spectral matching.
• In noise-driven phenomena (double-well, RNN, neuron data), the stochastic DPDSR substantially outperforms deterministic models, as the latter must mimic random transitions using chaotic dynamics, which is inefficient and less accurate.
• In ECG data, DPDSR uniquely recovers the variability of interspike intervals (ISIs), versus over-regularized predictions from deterministic baselines.
• Varying the teacher forcing interval $\tau$ modulates the “internal regime”: frequent forcing ($\tau$ small) promotes deterministic chaos (positive Lyapunov exponents), while infrequent forcing shifts the model into a regime where noise dominates, attractors are stabilized, and KL divergence between latent noise and its prior rises.

4. Comparison with Deterministic Latent Dynamical Models

The deterministic baseline (SPDSR) shares the architecture and training scheme, but omits noise (sets $B=0$) and the corresponding noise encoder/KL regularization. The effect is as follows:

• SPDSR can match performance in deterministic regimes.
• DPDSR’s explicit noise modeling is preferable for systems where variability is exogenous/stochastic.
• When only partial observations are available, noise-driven DPDSR can more accurately reproduce stochastic, multi-modal, or intermittent transitions, while deterministic models typically generate overly smooth or regular patterns.

The ability to allocate variance between structured latent evolution and explicit noise leads to superior reproduction of realistic system behaviors, particularly in physical and biological systems characterized by both deterministic and stochastic factors.

5. Practical Implications of Double Latent Projections and Teacher Forcing

The double projection mechanism (state and noise) enables the model to disentangle intrinsic system dynamics from extrinsic or unmodeled stochasticity. The teacher forcing interval acts as a knob to select between model regimes (deterministic-chaotic versus noise-dominated). This flexibility is crucial:

• For uncertain, under-observed settings, the noise encoder can “explain away” fluctuations without overfitting the latent state trajectory.
• For forecasting and system identification, DPDSR admits stable, multi-step free evolution of the internal state, as opposed to single-step fitting.
• The latent state–noise separation is especially beneficial for systems with partial observation, such as high-dimensional RNNs observed through a single variable, or biological signals where only aggregate or indirect measurements are available.

The method accommodates both short-term prediction quality and longer-horizon statistical consistency in reconstructed trajectories.

6. Summary Table: Core Distinctions in the DPDSR Approach

Aspect	DPDSR	Deterministic Baseline (SPDSR)
Latent variables	State + noise	State only
KL regularization	On noise sequence	None
Teacher forcing	Periodic, tunable	Same (without latent noise)
Flexibility	Chaotic and stochastic	Chaotic only
Performance (noise-driven systems)	High	Lower, over-regularized

7. Implications, Limitations, and Outlook

Variational autoencoders augmented with double projection and periodic teacher-forcing reconstruct both state trajectories and the underlying driving noise, affording robust system identification from partial, noisy, or high-dimensional observations. This paradigm is validated across canonical deterministic/stochastic dynamic models and experimental data.

Open challenges include systematic selection of teacher forcing interval $\tau$, efficient handling of very high-dimensional latent and observation spaces, and integration with more complex priors or explicit domain knowledge. A plausible implication is that the latent state–noise decomposition could be of particular value in biological and physical applications with layered sources of variability, where the measurement process is indirect and heterogeneous.

The method advances the field by allowing a VAE to flexibly balance the tradeoff between recovering deterministic structure and modeling intrinsic or unobserved randomness, establishing a unified framework for dynamical system reconstruction from partial or noisy observations (Sip et al., 1 Oct 2025).