Deep State-Space Encoders Overview

• Deep state-space encoders are neural recognition models that estimate latent states from sequences while preserving the state-space separation between dynamics and observation.
• They employ variational autoencoder frameworks to approximate intractable posteriors, using neural parameterizations to enhance forecasting, system identification, and interpretability.
• Applications range from discrete to continuous time and structured data (e.g., graphs, videos), addressing challenges like latent redundancy, oversmoothing, and encoder expressivity.

Across recent literature, deep state-space encoders are neural encoder functions or recognition models that estimate latent states from observations, histories, or spatial tokens inside a state-space model. In discrete-time deep state-space models, they typically appear as amortized posteriors $q_\phi(z_{1:T}\mid x_{1:T})$; in nonlinear system identification, they estimate unknown initial states for short simulated subsequences; and in continuous-time settings they initialize latent trajectories for neural ODE or neural state-space rollouts (Lin et al., 2024, Beintema et al., 2020, Beintema et al., 2022). The common objective is to retain the state-space separation between latent dynamics and observation generation while replacing fixed linear/Gaussian components with neural parameterizations, often to improve forecasting, system identification, irregular-sampling support, or latent interpretability (Wu et al., 2022, Lin et al., 2024).

1. Formal state-space setting

A general discrete-time deep state-space model posits a latent Markov chain $\{z_t\}$ and an observation series $\{x_t\}$, with transition and emission distributions

$$z_t \mid z_{t-1}\sim p_\theta(z_t\mid z_{t-1}), \qquad x_t \mid z_t\sim p_\theta(x_t\mid z_t).$$

In a deep variant, both components are parameterized by neural networks, for example

$$z_t = f_{\theta_f}(z_{t-1},\varepsilon_t), \quad \varepsilon_t\sim\mathcal N(0,I_d), \qquad x_t\sim \mathcal N(\mu_{\theta_g}(z_t),\Sigma_{\theta_g}(z_t)),$$

or equivalently through Gaussian transition and emission families whose means and covariances are neural-network outputs (Lin et al., 2024).

Exact inference $p_\theta(z_{1:T}\mid x_{1:T})$ is generally intractable, so deep state-space encoders are introduced as amortized variational approximations. A standard variational autoencoder formulation maximizes

$$\log p_\theta(x_{1:T}) \ge \mathbb E_{q_\phi(z_{1:T}\mid x_{1:T})} \bigl[\log p_\theta(x_{1:T},z_{1:T})-\log q_\phi(z_{1:T}\mid x_{1:T})\bigr],$$

with a common factorization

$$q_\phi(z_{1:T}\mid x_{1:T}) = q_\phi(z_1\mid x_{1:T}) \prod_{t=2}^T q_\phi(z_t\mid z_{t-1},x_{1:T}).$$

The reconstruction term is $\mathbb{E}_{q}[\sum_t\log p_\theta(x_t\mid z_t)]$, the KL penalty is $\sum_t\KL[q_\phi(z_t\mid\cdot)\|p_\theta(z_t\mid z_{t-1})]$, and gradients are commonly obtained by the reparameterization trick (Lin et al., 2024).

A later critique is that maximizing the ELBO alone does not guarantee that the transition model learns the true dynamics. In that account, an over-regularized posterior can collapse or hide dynamics in the recognition RNN, and a fixed prior $\{z_t\}$0 can break generation from the prior. This motivates a constrained optimization framework that minimizes rate subject to a distortion constraint, rather than treating ELBO maximization as sufficient for system identification (Klushyn et al., 26 Feb 2026).

2. Discrete-time encoder parameterizations

Representative discrete-time encoder families differ mainly in how much temporal context they use and how explicitly they separate stochastic latent state from deterministic recurrence. Deep Markov Models use an encoder $\{z_t\}$1 based on a backward RNN over future observations plus $\{z_t\}$2, whereas the Variational RNN maintains a deterministic hidden state $\{z_t\}$3 and conditions both generation and inference on that state (Lin et al., 2024).

A more specialized architecture is the interpretable deep state-space model of Ansari, Stella, and Dumancas. Its encoder approximates

$\{z_t\}$4

by a structured, time-factorized inference network. At each time $\{z_t\}$5, a deterministic GRU state is updated as

$\{z_t\}$6

local shrinkage variables are inferred through a log-normal decomposition of $\{z_t\}$7, and the latent innovation is inferred through a non-centered scale mixture: $\{z_t\}$8 The corresponding generative transition uses the same non-centered parameterization,

$\{z_t\}$9

with $\{x_t\}$0 and $\{x_t\}$1 implemented as two separate MLPs (Wu et al., 2022).

The same model modifies the decoder to improve latent interpretability. Instead of a black-box neural decoder, it uses

$\{x_t\}$2

where $\{x_t\}$3 is a fixed non-time-varying $\{x_t\}$4 matrix and $\{x_t\}$5. Combined with a regularized horseshoe prior,

$\{x_t\}$6

this yields sparse, non-redundant latent dimensions that can be interpreted as random effects in a linear mixed model. The paper attributes three consequences to the combination of a linear decoder and global-local shrinkage: closed-form KL terms in the variational ELBO, sparse latent dimensions with simple random-effect interpretations, and no loss in forecasting accuracy with slightly improved ND and RMSE on standard electricity and traffic benchmarks (Wu et al., 2022).

A recurrent misconception is that any expressive encoder automatically produces interpretable states. The evidence from deep state-space forecasting points in the opposite direction: previous DSSMs with black-box decoders typically produced latent variables that were very difficult to interpret, and prior DSSMs without shrinkage allowed many redundant or spurious latent dimensions (Wu et al., 2022).

3. Initial-state encoders for nonlinear system identification

A distinct line of work uses deep state-space encoders not as variational posteriors over full latent trajectories, but as reconstructability maps that estimate the initial state of a short simulation window. In the discrete-time nonlinear identification formulation,

$\{x_t\}$7

training by full simulation loss

$\{x_t\}$8

is computationally expensive on large datasets and strongly non-convex. The proposed remedy is to split the data into multiple independent sections, analogous to multiple shooting, and to replace per-section initial-state optimization with an encoder

$\{x_t\}$9

This encoder is typically a fully connected network with one or two hidden layers, tanh or ReLU activations, and an optional linear bypass connection. Joint training then optimizes the model parameters and encoder parameters together by minibatch stochastic optimization (Beintema et al., 2020).

The central claim of this formulation is not merely computational convenience. The splitting operation is said to allow stochastic gradient optimization methods that scale well with data size and to have a smoothing effect on the non-convex cost function. The per-segment independence and encoder initialization are described as smoothing out the loss landscape, while the encoder avoids extra per-segment parameters and provides good jump-start states (Beintema et al., 2020).

On the Wiener–Hammerstein benchmark, the reported setup used $$z_t \mid z_{t-1}\sim p_\theta(z_t\mid z_{t-1}), \qquad x_t \mid z_t\sim p_\theta(x_t\mid z_t).$$0, a single hidden layer with 15 tanh units, $$z_t \mid z_{t-1}\sim p_\theta(z_t\mid z_{t-1}), \qquad x_t \mid z_t\sim p_\theta(x_t\mid z_t).$$1, $$z_t \mid z_{t-1}\sim p_\theta(z_t\mid z_{t-1}), \qquad x_t \mid z_t\sim p_\theta(x_t\mid z_t).$$2, $$z_t \mid z_{t-1}\sim p_\theta(z_t\mid z_{t-1}), \qquad x_t \mid z_t\sim p_\theta(x_t\mid z_t).$$3, batch size $$z_t \mid z_{t-1}\sim p_\theta(z_t\mid z_{t-1}), \qquad x_t \mid z_t\sim p_\theta(x_t\mid z_t).$$4, and Adam with learning rate $$z_t \mid z_{t-1}\sim p_\theta(z_t\mid z_{t-1}), \qquad x_t \mid z_t\sim p_\theta(x_t\mid z_t).$$5. The test result was RMS $$z_t \mid z_{t-1}\sim p_\theta(z_t\mid z_{t-1}), \qquad x_t \mid z_t\sim p_\theta(x_t\mid z_t).$$6 mV, corresponding to NRMS $$z_t \mid z_{t-1}\sim p_\theta(z_t\mid z_{t-1}), \qquad x_t \mid z_t\sim p_\theta(x_t\mid z_t).$$7, described as the lowest reported in literature. On the Silverbox benchmark, a model with $$z_t \mid z_{t-1}\sim p_\theta(z_t\mid z_{t-1}), \qquad x_t \mid z_t\sim p_\theta(x_t\mid z_t).$$8, two hidden layers of 64 units, tanh activations, $$z_t \mid z_{t-1}\sim p_\theta(z_t\mid z_{t-1}), \qquad x_t \mid z_t\sim p_\theta(x_t\mid z_t).$$9, $$z_t = f_{\theta_f}(z_{t-1},\varepsilon_t), \quad \varepsilon_t\sim\mathcal N(0,I_d), \qquad x_t\sim \mathcal N(\mu_{\theta_g}(z_t),\Sigma_{\theta_g}(z_t)),$$0, batch size $$z_t = f_{\theta_f}(z_{t-1},\varepsilon_t), \quad \varepsilon_t\sim\mathcal N(0,I_d), \qquad x_t\sim \mathcal N(\mu_{\theta_g}(z_t),\Sigma_{\theta_g}(z_t)),$$1, and learning rate $$z_t = f_{\theta_f}(z_{t-1},\varepsilon_t), \quad \varepsilon_t\sim\mathcal N(0,I_d), \qquad x_t\sim \mathcal N(\mu_{\theta_g}(z_t),\Sigma_{\theta_g}(z_t)),$$2 achieved validation RMS $$z_t = f_{\theta_f}(z_{t-1},\varepsilon_t), \quad \varepsilon_t\sim\mathcal N(0,I_d), \qquad x_t\sim \mathcal N(\mu_{\theta_g}(z_t),\Sigma_{\theta_g}(z_t)),$$3 mV and test RMS $$z_t = f_{\theta_f}(z_{t-1},\varepsilon_t), \quad \varepsilon_t\sim\mathcal N(0,I_d), \qquad x_t\sim \mathcal N(\mu_{\theta_g}(z_t),\Sigma_{\theta_g}(z_t)),$$4 mV in the extrapolation region, with $$z_t = f_{\theta_f}(z_{t-1},\varepsilon_t), \quad \varepsilon_t\sim\mathcal N(0,I_d), \qquad x_t\sim \mathcal N(\mu_{\theta_g}(z_t),\Sigma_{\theta_g}(z_t)),$$5 mV if extrapolation was excluded (Beintema et al., 2020).

The same encoder principle was extended to high-dimensional video observations. In that setting, the model approximates

$$z_t = f_{\theta_f}(z_{t-1},\varepsilon_t), \quad \varepsilon_t\sim\mathcal N(0,I_d), \qquad x_t\sim \mathcal N(\mu_{\theta_g}(z_t),\Sigma_{\theta_g}(z_t)),$$6

and introduces an encoder

$$z_t = f_{\theta_f}(z_{t-1},\varepsilon_t), \quad \varepsilon_t\sim\mathcal N(0,I_d), \qquad x_t\sim \mathcal N(\mu_{\theta_g}(z_t),\Sigma_{\theta_g}(z_t)),$$7

Training uses overlapping shots of length $$z_t = f_{\theta_f}(z_{t-1},\varepsilon_t), \quad \varepsilon_t\sim\mathcal N(0,I_d), \qquad x_t\sim \mathcal N(\mu_{\theta_g}(z_t),\Sigma_{\theta_g}(z_t)),$$8, with the batch objective

$$z_t = f_{\theta_f}(z_{t-1},\varepsilon_t), \quad \varepsilon_t\sim\mathcal N(0,I_d), \qquad x_t\sim \mathcal N(\mu_{\theta_g}(z_t),\Sigma_{\theta_g}(z_t)),$$9

For a 25$p_\theta(z_{1:T}\mid x_{1:T})$025 pixel ball-in-box video system with $p_\theta(z_{1:T}\mid x_{1:T})$1, $p_\theta(z_{1:T}\mid x_{1:T})$2, and $p_\theta(z_{1:T}\mid x_{1:T})$3, the state-space encoder outperformed the IO-autoencoder across all reported noise levels. On the noise-free test set, the state-space encoder obtained $p_\theta(z_{1:T}\mid x_{1:T})$4, $p_\theta(z_{1:T}\mid x_{1:T})$5, $p_\theta(z_{1:T}\mid x_{1:T})$6, $p_\theta(z_{1:T}\mid x_{1:T})$7, and $p_\theta(z_{1:T}\mid x_{1:T})$8 NRMS for noise levels $p_\theta(z_{1:T}\mid x_{1:T})$9, $$\log p_\theta(x_{1:T}) \ge \mathbb E_{q_\phi(z_{1:T}\mid x_{1:T})} \bigl[\log p_\theta(x_{1:T},z_{1:T})-\log q_\phi(z_{1:T}\mid x_{1:T})\bigr],$$0, $$\log p_\theta(x_{1:T}) \ge \mathbb E_{q_\phi(z_{1:T}\mid x_{1:T})} \bigl[\log p_\theta(x_{1:T},z_{1:T})-\log q_\phi(z_{1:T}\mid x_{1:T})\bigr],$$1, $$\log p_\theta(x_{1:T}) \ge \mathbb E_{q_\phi(z_{1:T}\mid x_{1:T})} \bigl[\log p_\theta(x_{1:T},z_{1:T})-\log q_\phi(z_{1:T}\mid x_{1:T})\bigr],$$2, and $$\log p_\theta(x_{1:T}) \ge \mathbb E_{q_\phi(z_{1:T}\mid x_{1:T})} \bigl[\log p_\theta(x_{1:T},z_{1:T})-\log q_\phi(z_{1:T}\mid x_{1:T})\bigr],$$3, respectively, versus $$\log p_\theta(x_{1:T}) \ge \mathbb E_{q_\phi(z_{1:T}\mid x_{1:T})} \bigl[\log p_\theta(x_{1:T},z_{1:T})-\log q_\phi(z_{1:T}\mid x_{1:T})\bigr],$$4, $$\log p_\theta(x_{1:T}) \ge \mathbb E_{q_\phi(z_{1:T}\mid x_{1:T})} \bigl[\log p_\theta(x_{1:T},z_{1:T})-\log q_\phi(z_{1:T}\mid x_{1:T})\bigr],$$5, $$\log p_\theta(x_{1:T}) \ge \mathbb E_{q_\phi(z_{1:T}\mid x_{1:T})} \bigl[\log p_\theta(x_{1:T},z_{1:T})-\log q_\phi(z_{1:T}\mid x_{1:T})\bigr],$$6, $$\log p_\theta(x_{1:T}) \ge \mathbb E_{q_\phi(z_{1:T}\mid x_{1:T})} \bigl[\log p_\theta(x_{1:T},z_{1:T})-\log q_\phi(z_{1:T}\mid x_{1:T})\bigr],$$7, and $$\log p_\theta(x_{1:T}) \ge \mathbb E_{q_\phi(z_{1:T}\mid x_{1:T})} \bigl[\log p_\theta(x_{1:T},z_{1:T})-\log q_\phi(z_{1:T}\mid x_{1:T})\bigr],$$8 for the IO-autoencoder (Beintema et al., 2020).

4. Continuous-time encoders and irregular sampling

Continuous-time deep state-space encoders extend the latent-state idea to trajectories defined by differential equations. In the latent Neural ODE formulation, the state evolves by

$$\log p_\theta(x_{1:T}) \ge \mathbb E_{q_\phi(z_{1:T}\mid x_{1:T})} \bigl[\log p_\theta(x_{1:T},z_{1:T})-\log q_\phi(z_{1:T}\mid x_{1:T})\bigr],$$9

with decoder $$q_\phi(z_{1:T}\mid x_{1:T}) = q_\phi(z_1\mid x_{1:T}) \prod_{t=2}^T q_\phi(z_t\mid z_{t-1},x_{1:T}).$$0. Irregular sampling is handled by an ODE solver through

$$q_\phi(z_{1:T}\mid x_{1:T}) = q_\phi(z_1\mid x_{1:T}) \prod_{t=2}^T q_\phi(z_t\mid z_{t-1},x_{1:T}).$$1

and the encoder is commonly a reverse-time ODE-RNN or an RNN that reads all $$q_\phi(z_{1:T}\mid x_{1:T}) = q_\phi(z_1\mid x_{1:T}) \prod_{t=2}^T q_\phi(z_t\mid z_{t-1},x_{1:T}).$$2. Latent Neural SDEs replace deterministic drift with stochastic evolution $$q_\phi(z_{1:T}\mid x_{1:T}) = q_\phi(z_1\mid x_{1:T}) \prod_{t=2}^T q_\phi(z_t\mid z_{t-1},x_{1:T}).$$3, and the review describes encoders that approximate $$q_\phi(z_{1:T}\mid x_{1:T}) = q_\phi(z_1\mid x_{1:T}) \prod_{t=2}^T q_\phi(z_t\mid z_{t-1},x_{1:T}).$$4 in parallel over each sample time (Lin et al., 2024).

The SUBNET method addresses continuous-time nonlinear state-space identification with external inputs, measurement noise, latent states, and robustness. It assumes

$$q_\phi(z_{1:T}\mid x_{1:T}) = q_\phi(z_1\mid x_{1:T}) \prod_{t=2}^T q_\phi(z_t\mid z_{t-1},x_{1:T}).$$5

and learns neural approximations $$q_\phi(z_{1:T}\mid x_{1:T}) = q_\phi(z_1\mid x_{1:T}) \prod_{t=2}^T q_\phi(z_t\mid z_{t-1},x_{1:T}).$$6 and $$q_\phi(z_{1:T}\mid x_{1:T}) = q_\phi(z_1\mid x_{1:T}) \prod_{t=2}^T q_\phi(z_t\mid z_{t-1},x_{1:T}).$$7. To avoid full-trajectory integration during training, the data are split into overlapping subsequences of length $$q_\phi(z_{1:T}\mid x_{1:T}) = q_\phi(z_1\mid x_{1:T}) \prod_{t=2}^T q_\phi(z_t\mid z_{t-1},x_{1:T}).$$8, and an encoder estimates the initial state of each subsection from the most recent $$q_\phi(z_{1:T}\mid x_{1:T}) = q_\phi(z_1\mid x_{1:T}) \prod_{t=2}^T q_\phi(z_t\mid z_{t-1},x_{1:T}).$$9 past inputs and $\mathbb{E}_{q}[\sum_t\log p_\theta(x_t\mid z_t)]$0 past outputs: $\mathbb{E}_{q}[\sum_t\log p_\theta(x_t\mid z_t)]$1 Training then rolls out the dynamics on short windows using an ODE solver and minimizes

$\mathbb{E}_{q}[\sum_t\log p_\theta(x_t\mid z_t)]$2

averaged over all valid subsection starts (Beintema et al., 2022).

A distinctive element of SUBNET is state-derivative normalization. Rather than learning $\mathbb{E}_{q}[\sum_t\log p_\theta(x_t\mid z_t)]$3 directly, it inserts a scale factor

$\mathbb{E}_{q}[\sum_t\log p_\theta(x_t\mid z_t)]$4

motivated by a theorem stating that a continuous-time system can be rescaled so that both the RMS of $\mathbb{E}_{q}[\sum_t\log p_\theta(x_t\mid z_t)]$5 and the RMS of $\mathbb{E}_{q}[\sum_t\log p_\theta(x_t\mid z_t)]$6 are unity over any finite horizon. The paper argues that this normalization is essential for reliable estimation of continuous-time nonlinear state-space models. It also proves a smoothness result: if $\mathbb{E}_{q}[\sum_t\log p_\theta(x_t\mid z_t)]$7 is $\mathbb{E}_{q}[\sum_t\log p_\theta(x_t\mid z_t)]$8-Lipschitz in $\mathbb{E}_{q}[\sum_t\log p_\theta(x_t\mid z_t)]$9, then the Lipschitz constant $\sum_t\KL[q_\phi(z_t\mid\cdot)\|p_\theta(z_t\mid z_{t-1})]$0 of the training objective grows only as

$\sum_t\KL[q_\phi(z_t\mid\cdot)\|p_\theta(z_t\mid z_{t-1})]$1

so shorter subsequences yield a smoother objective and more stable gradient descent. For encoder existence, the appendix gives a local invertibility condition requiring $\sum_t\KL[q_\phi(z_t\mid\cdot)\|p_\theta(z_t\mid z_{t-1})]$2 under mild technical assumptions (Beintema et al., 2022).

Empirically, SUBNET reports strong results on Cascade-Tank, Coupled Electric Drive, and EMPS. On CED, the reported CT SUBNET ($\sum_t\KL[q_\phi(z_t\mid\cdot)\|p_\theta(z_t\mid z_{t-1})]$3) achieved test RMSE $\sum_t\KL[q_\phi(z_t\mid\cdot)\|p_\theta(z_t\mid z_{t-1})]$4 on Set 1 and $\sum_t\KL[q_\phi(z_t\mid\cdot)\|p_\theta(z_t\mid z_{t-1})]$5 on Set 2. On EMPS, CT SUBNET achieved RMSE $\sum_t\KL[q_\phi(z_t\mid\cdot)\|p_\theta(z_t\mid z_{t-1})]$6 mm and is described as competitive with dynoNET $\sum_t\KL[q_\phi(z_t\mid\cdot)\|p_\theta(z_t\mid z_{t-1})]$7 mm, grey-box) while far outperforming purely black-box methods (Beintema et al., 2022).

5. Structural priors, relational encoders, and disentangled coordinates

Deep state-space encoders often incorporate structural priors to make latent states more identifiable, sparse, or semantically separated. One route is graph structure. Graph state-space models define a latent state-graph $\sum_t\KL[q_\phi(z_t\mid\cdot)\|p_\theta(z_t\mid z_{t-1})]$8, adjacency $\sum_t\KL[q_\phi(z_t\mid\cdot)\|p_\theta(z_t\mid z_{t-1})]$9, and observed graph $\{z_t\}$00, with joint model

$\{z_t\}$01

The encoder follows a Select–Reduce–Connect plus MPNN recipe: it learns a soft affiliation matrix $\{z_t\}$02, aggregates observation-node features into the latent node set, concatenates them with previous latent-state features, samples a new edge set from Bernoulli probabilities $\{z_t\}$03, and runs message passing to produce the next latent-state graph. In the fully probabilistic formulation, gradients through discrete adjacency sampling are obtained by a score-function estimator, while the decoder uses a reparameterization trick for distributional forecasting (Zambon et al., 2023).

Another route is manifold-coordinate discovery. The IRMAE-WD framework uses a nonlinear encoder $\{z_t\}$04, a stack of internal linear layers $\{z_t\}$05, and a nonlinear decoder $\{z_t\}$06, trained with reconstruction loss plus $\{z_t\}$07 regularization on every weight matrix. After training, the empirical covariance of latent codes

$\{z_t\}$08

develops a sharp spectral gap after the manifold dimension $\{z_t\}$09. The method then defines an orthogonal coordinate system by truncating $\{z_t\}$10 and uses the resulting coordinates $\{z_t\}$11 to fit either a Neural ODE $\{z_t\}$12 or a discrete-time map $\{z_t\}$13. Reported examples include exact recovery of $\{z_t\}$14 on a spiral-wrapped Lorenz system in $\{z_t\}$15, $\{z_t\}$16 and $\{z_t\}$17 on Kuramoto–Sivashinsky data with $\{z_t\}$18 and $\{z_t\}$19, and $\{z_t\}$20 on reaction–diffusion spiral waves (Zeng et al., 2023).

A third route is explicit static–dynamic disentanglement. The extended Kalman VAE introduces auxiliary variables $\{z_t\}$21 with linear observation model

$\{z_t\}$22

together with latent dynamics

$\{z_t\}$23

where $\{z_t\}$24 are mixtures of learned base matrices. The recognition model combines amortized inference $\{z_t\}$25 with EKF/EKS smoothing for $\{z_t\}$26. Static and dynamic features are separated by choosing

$\{z_t\}$27

so that the first $\{z_t\}$28 latent dimensions reconstruct the observation while the remaining dimensions absorb dynamics. Reported results include $\{z_t\}$29, $\{z_t\}$30, and prediction MSE $\{z_t\}$31 on a pendulum image task for VHP-EKVAE trained with constrained optimization (Klushyn et al., 26 Feb 2026).

A more radical interpretation appears in sparse-autoencoder analysis of 3D latent codes. There, a BatchTopK SAE maps each 64-dimensional latent $\{z_t\}$32 to a sparse 512-dimensional code $\{z_t\}$33, inducing a binary state vector $\{z_t\}$34. The paper argues that the model approximates a discrete state space driven by phase-like transitions from feature activations, with sigmoidal loss curves under feature ablation and a bimodal distribution of transition points $\{z_t\}$35. This suggests a discrete-state interpretation of latent feature dynamics rather than a purely continuous-coordinate one, although that interpretation is specific to the analyzed 3D reconstruction VAE (Miao et al., 12 Dec 2025).

6. Expressivity, applications, and recurrent controversies

Deep state-space encoders are now used well beyond classical latent-variable forecasting. In long-horizon time-series prediction, the SpaceTime architecture uses a companion-matrix state-space layer

$\{z_t\}$36

with $\{z_t\}$37 constrained to companion form. The paper states that if $\{z_t\}$38 and $\{z_t\}$39, then

$\{z_t\}$40

so the layer exactly implements a noiseless AR$\{z_t\}$41 process, and proves that a companion-matrix SSM with learned coefficients can represent exactly any AR$\{z_t\}$42 process. It further introduces a closed-loop variant for forecasting and an FFT/Woodbury algorithm reducing per-layer cost from $\{z_t\}$43 to near $\{z_t\}$44. Empirically, SpaceTime reports best or second-best AUROC on $\{z_t\}$45 ECG and speech tasks, best MSE on $\{z_t\}$46 Informer forecasting tasks, and wall-clock speedups of $\{z_t\}$47 versus Transformers and $\{z_t\}$48 versus LSTMs on ETTh1 (Zhang et al., 2023).

In event-based optical flow, Perturbed State Space Feature Encoders treat feature maps as sequences processed by a discretized SSM

$\{z_t\}$49

but regularize the state dynamics through a perturb-then-diagonalize step $\{z_t\}$50, with $\{z_t\}$51. The resulting P-SSE uses 2D-to-1D scanning, ViT-style residual blocks, and a bidirectional recurrent multi-frame optical-flow pipeline. On DSEC-Flow and MVSEC, the reported gains are $\{z_t\}$52 and $\{z_t\}$53 improvements in EPE, respectively, and the ablation reports EPE $\{z_t\}$54 for P-SSE$\{z_t\}$55 with PTD versus $\{z_t\}$56 with HiPPO initialization (Raju et al., 14 Apr 2025).

In vision-language modeling, state-space encoders appear as frozen visual backbones rather than latent-variable posteriors. VMamba uses a four-stage hierarchical SS4D architecture with four directional scans over the image grid, replacing self-attention with structured state propagation. Under matched ImageNet-1K initialization, the reported frozen-backbone VLM comparison gives VMamba-T $\{z_t\}$57M with VQA $\{z_t\}$58, localization $\{z_t\}$59, overall $\{z_t\}$60; VMamba-S $\{z_t\}$61M with VQA $\{z_t\}$62, localization $\{z_t\}$63, overall $\{z_t\}$64; and markedly lower localization for ViT-S and MaxViT-T. The study also reports that higher ImageNet accuracy or larger backbones do not reliably translate into better VLM performance, and that some detection-adapted checkpoints collapse in localization unless stabilization strategies such as a stronger connector or square evaluation geometry are applied (Kuo et al., 19 Mar 2026).

Several controversies recur across these literatures. One is representational adequacy: earlier SSM-based deep layers using continuous-time or diagonal $\{z_t\}$65 matrices are argued to be unable to capture even simple AR$\{z_t\}$66 processes, while the companion parameterization is designed precisely to overcome that limitation (Zhang et al., 2023). A second is objective mismatch: ELBO optimization may produce good reconstructions without ensuring that the latent transition actually learns the underlying dynamics (Klushyn et al., 26 Feb 2026). A third is cost versus fidelity: the selective review notes that more powerful encoders such as attentive RNNs or ODE-RNNs improve accuracy but raise GPU and autodiff overhead, that Neural ODE/SDE encoders are advantageous for irregular or multi-rate data but require solvers, and that diagonal-Gaussian encoders are fast but may under-fit multi-modal posteriors relative to normalizing-flow or mixture-Gaussian alternatives (Lin et al., 2024).

Taken together, these developments show that the phrase “deep state-space encoder” no longer refers to a single architecture. It denotes a family of neural mechanisms for estimating or propagating latent state in discrete-time, continuous-time, graph-structured, image-based, and event-based settings. The unifying theme is not a specific network block but the preservation of a state-space viewpoint—latent state, transition, observation, and inference—even as the encoder becomes recurrent, variational, graph-based, solver-driven, sparse, or selectively scanned (Lin et al., 2024).