Deep Variational Bayes Filters (DVBF)

• DVBF are probabilistic state-space models that extract latent Markovian dynamics from sequential data using deterministic transitions and structured variational inference.
• They leverage reparameterization and annealing strategies to optimize the ELBO, ensuring robust long-term predictions across complex, nonlinear systems.
• DVBF frameworks adapt to diverse applications, including image-based systems and physics-integrated simulations like FEMIN for uncertainty-aware surrogate modeling.

Deep Variational Bayes Filters (DVBF) are probabilistic state-space models designed for unsupervised learning and identification of latent Markovian dynamics from raw sequential data. DVBF combines structured variational inference, reparameterized latent transitions enabling full gradient flow through time, and scalable stochastic optimization, thereby learning interpretable latent representations and enabling long-term predictions in high-dimensional, nonlinear environments. DVBF has been adapted to various domains, including raw image-based systems and, more recently, integration with physics-based simulation frameworks such as Finite Element Method Integrated Networks (FEMIN), for enhanced confidence quantification and uncertainty-aware surrogate modeling (Karl et al., 2016, Thel et al., 2024).

1. Probabilistic Model Architecture and Generative Process

DVBF postulates a generative model for sequences of observations $x_{1:T}$ with controls $u_{1:T-1}$, introducing a low-dimensional latent state sequence $z_{0:T}$ governed by Markovian transition dynamics and an explicit emission process. The generative factorization is:

• Initial latent state prior: $p(z_0)$.
• Transition-parameter prior over $\beta_{1:T}=(w_{1:T},v_{1:T})$ (sample-specific noise and global parameters).
• Markovian latent transitions: $p(z_{t+1}|z_t,u_t,\beta_t)$, with deterministic update $z_{t+1} = f_\theta(z_t,u_t,w_t,v_t)$ ensuring $$p(z_{t+1}|z_t,u_t,\beta_t)=\delta(z_{t+1}-f_\theta(z_t,u_t,\beta_t))$$.
• Conditioned emission: $p(x_{t+1}|z_{t+1})$.

The locally‐linear variant (“DVBF‐LL”) employs

$z_{t+1} = A_t z_t + B_t u_t + C_t w_t$

where $A_t$, $B_t$, $C_t$ are dynamic mixtures of global basis matrices determined by $\alpha_t = \mathrm{MLP}_\psi(z_t,u_t)$, and $w_t \sim \mathcal{N}(0,I)$. The emission model is typically Gaussian or Bernoulli as appropriate for the data type (Karl et al., 2016).

2. Variational Inference and Evidence Lower Bound (ELBO)

Exact inference over the latent trajectory and transitions is intractable. DVBF utilizes a structured variational posterior:

$$q_\phi(w_{1:T},v_{1:T}\mid x_{1:T},u_{1:T-1}) = \prod_{t=0}^{T-1} q_\phi(w_t\mid z_t,x_{t+1},u_t) \cdot q_\phi(v_t)$$

and reconstructs $z_{1:T}$ deterministically via unrolled $f_\theta$ transitions.

The training objective is the ELBO,

$$\ln p_\theta(x_{1:T}\mid u_{1:T-1}) \geq \mathbb{E}_{q_\phi} \left[\sum_{t=1}^T \ln p_\theta(x_t | z_t)\right] - \mathrm{KL}\left(q_\phi(w_{1:T},v_{1:T}|x_{1:T},u_{1:T-1})\,\|\;p(w_{1:T},v_{1:T})\right)$$

allowing per-time-step KL regularization. The deterministic transitions allow gradients to be propagated back through all $z_t$, enforcing that each latent state encodes full information for future prediction and emission (Karl et al., 2016).

3. Learning Algorithm and Practical Implementation

DVBF is trained via stochastic gradient ascent on the annealed ELBO using mini-batch sampling, the reparameterization trick for variance reduction, and SGD optimizers such as Adadelta or Adam. Annealing the reconstruction loss coefficient $c_i$ from 0.01 to 1 over $T_A$ iterations mitigates poor local minima. Pseudocode for a single mini-batch step:

for each minibatch {x_{1:T}, u_{1:T-1}} do sample w_t ~ q_phi(w_t | z_t, x_{t+1}, u_t) for t = 0..T-1 sample v_t ~ q_phi(v_t) for t = 0..T-1 deterministically compute z_{t+1} = f_theta(z_t, u_t, w_t, v_t) compute L_rec = sum_{t=1}^T log p_theta(x_t | z_t) compute L_KL = sum_{t=0}^{T-1} KL(q_phi(w_t)||p(w_t)) + KL(q_phi(v_t)||p(v_t)) L = c_i * L_rec - L_KL grad_{theta, phi} L via backprop through transitions update theta, phi end for

Critical hyperparameters for typical experiments include sequence length $T$, latent dimension, network architectures, and batch size (e.g., $T=15$, latent dim $=3$, batch $=500$ for pendulum experiments).

4. Extension to FEMIN and Confidence Estimation

Recent advancements adapt DVBF to surrogate modeling for crash simulations within the FEMIN framework, enabling probabilistic prediction and uncertainty quantification (Thel et al., 2024):

• The generative model incorporates FEM states as $x_t = [d_{\rm B,t}, v_{\rm B,t}, f_{\rm B,t}]$.
• Encoder $q_{\rm enc}(z_t|x_t)$ and decoder $p_{\rm lik}(x_t|z_t)$ are MLPs mapping physical variables to means and variances.
• A “predictive decoding” procedure samples from the transition prior $p_{\rm trans}$, decodes preliminary force predictions, and redefines online encoder inputs as $$x_t = [d_t, v_t, \tilde\mu_{\rm force,t}, \tilde\sigma_{\rm force,t}]$$, avoiding ground-truth force leakage.
• The decoder outputs $(\mu_{f,t}, \sigma^2_{f,t})$ serve as surrogate force and uncertainty metrics, respectively. Empirically, $\sigma_{f,t}$ correlates ($r \sim 60\%$) with absolute force error, thus providing a qualitative confidence measure.

DVBF-based surrogates outperform deterministic neural architectures (e.g., LSTM) across accuracy metrics and confidence calibration, exemplified in crash cases such as Box Impact and Tension‐Compression‐Tension:

Quantity	DVBF MSE / $R^2$	LSTM (full) MSE / $R^2$	LSTM (window) MSE / $R^2$
Displacement	$9.42\times10^{-5} / 0.9977$	$1.39\times10^{-4} / 0.9966$	$3.11\times10^{-4} / 0.9923$
Velocity	$4.27\times10^{-4} / 0.9917$	$4.17\times10^{-4} / 0.9919$	$8.29\times10^{-4} / 0.9838$
Force	$1.46\times10^{-3} / 0.9604$	$1.71\times10^{-3} / 0.9537$	$2.58\times10^{-3} / 0.9302$
Combined	$6.61\times10^{-4} / 0.9832$	$7.55\times10^{-4} / 0.9807$	$1.24\times10^{-3} / 0.9688$

Confidence metrics for force predictions show high PICP (coverage probability) and tightly calibrated uncertainty bands.

5. Distinctive Features and Comparative Advantages

DVBF enforces the state-space structure by ensuring latent variables must encode all predictive and generative information, in contrast to earlier approaches such as VRNN and Deep Kalman Filters, which do not propagate reconstruction gradients through their transitions. As a result:

• Latent embeddings capture all physically relevant aspects (e.g., both position and velocity in pendulum; checkerboard positioning and velocity in bouncing balls).
• Free-running generative rollouts remain physiologically and physically realistic well beyond training horizons.
• The model is robust to high-dimensional, nonlinear raw data (e.g., pixel sequences).
• Scalable learning via stochastic gradient variational Bayes is maintained (Karl et al., 2016).
• The probabilistic structure admits principled uncertainty quantification, vital in scientific computing surrogates (Thel et al., 2024).

6. Empirical Evaluation and Benchmarks

DVBF demonstrates superior performance in diverse physical systems:

• For the dynamic pendulum, latent representations exhibit full circular topology with an axis for angular velocity. DVBF‐LL achieves $R^2$ regression scores near unity for $\sin\phi$, $\cos\phi$, and $\dot\phi$ (e.g., $0.961$, $0.982$, $0.916$), whereas DKF fails to capture velocity ($0.035$).
• In the bouncing ball system, latent space reflects both positional grid and velocity axes.
• Long-term, free-run generations closely match ground-truth trajectories.
• For crash simulations in FEMIN, DVBF yields mean squared errors and $R^2$ values exceeding deterministic surrogates, along with well-calibrated uncertainty intervals (PICP $88.2\%$ for force; NLL $-156.1$; average confidence band $5.85$ kN).
• Training and inference are computationally competitive, with DVBF training times ($37.7$ min) significantly below full-sequence LSTM ($310.8$ min), and inference cost remaining negligible for real applications.

7. Methodological Implications and Uncertainty Quantification

DVBF furnishes a principled, probabilistic surrogate for dynamic and physical systems, combining unsupervised learning of latent state-space models with rigorous Bayesian uncertainty quantification. The geometric structure and probability measure over latent spaces enable:

• Qualitative and quantitative confidence estimation through decoder variance.
• Recognition and calibration of model uncertainty, especially in ill-posed or extrapolative regimes.
• Robust performance and reliability alongside simulation outputs, as validated by empirical correlations and calibrated confidence intervals.

A plausible implication is that DVBF frameworks may become increasingly foundational for uncertainty-aware surrogate modeling in data-intensive scientific and engineering domains. The architecture’s capacity for interpretable latent representations and calibration of predictive confidence presents methodological advantages over deterministic and non-Bayesian approaches, particularly in high-risk environments requiring reliable decision support (Karl et al., 2016, Thel et al., 2024).