Gaussian-Based Spatiotemporal Representation

• Gaussian-based spatiotemporal representation is a paradigm that decomposes space–time data into Gaussian functions to capture joint spatial and temporal structures.
• It leverages methods such as SPDE formulations, Gaussian process priors with Matérn covariances, and sparse GMRF approximations for scalable and efficient inference.
• Applications span geostatistics, dynamic scene rendering, and physics-informed modeling, enabling uncertainty quantification and adaptive nonstationary analysis.

A Gaussian-based spatiotemporal representation models joint spatial and temporal structure by decomposing observed data, fields, or dynamics into sums, mixtures, or fields of Gaussian functions parameterized over both space and time. This paradigm encompasses a wide array of methodologies across statistical modeling, physics-based inference, video and dynamic scene synthesis, and learning compact representations for high-dimensional spatiotemporal data. Core constructions include stochastic partial differential equation (SPDE) formulations yielding Matérn-type covariances, state-space models with Gaussian process (GP) priors, learnable sums of space–time Gaussian primitives for graphics and imaging, and explicit anisotropic Gaussians parameterized over space or spacetime. By leveraging analytic properties of Gaussians—such as closure under convolution, tractable marginalization, and closed-form operations under affine transformations—these models offer both theoretical tractability and computational efficiency, as well as practical advantages including uncertainty quantification, differentiability, and flexible representation of non-stationary phenomena.

1. SPDE and Gaussian Matérn Fields for Spatiotemporal Modeling

A foundational class of spatiotemporal Gaussian representations is constructed via SPDEs, where the solution field is defined as the response to space–time white noise filtered by a differential operator. Lindgren et al. devised the diffusion-based extension of the Matérn field (DEMF), defined by

$$\Bigl(-\gamma_t^2\,\partial_{tt} + L_s^{\alpha_s}\Bigr)^{\alpha_t/2}\,u(s,t) = \dot{\mathcal E}_{Q_s}(s,t)$$

for $s \in D \subseteq \mathbb R^d$, where $L_s = \gamma_s^2 - \Delta$ and $\dot{\mathcal E}_{Q_s}$ is white in time and correlated in space through precision $Q_s$ (Lindgren et al., 2020). The stationary marginal at fixed $t$ yields a spatial Matérn correlation, controlled by variance $\sigma^2$, smoothness $\nu$, and correlation length; temporal marginals and nonstationarity are governed by additional parameters $\gamma_t, \alpha_t$. The spatiotemporal spectral density is

$$S_u(\omega_s, \omega_t) = \frac{1}{(2\pi)^{d+1} \gamma_e^2 [\gamma_t^2 \omega_t^2 + (\gamma_s^2 + \|\omega_s\|^2)^{\alpha_s}]^{\alpha_t} (\gamma_s^2 + \|\omega_s\|^2)^{\alpha_e}}$$

enabling explicit control over separability and degrees of smoothness in both domains.

The SPDE framework extends to curved manifolds by replacing the Laplacian with the Laplace–Beltrami operator and to non-stationary or spatially heterogeneous domains via coefficient functions or operators (Lindgren et al., 2020). These models admit sparse finite-element approximations, making them tractable for large-scale statistical inference via the R-INLA software (Lindgren et al., 2020).

2. Covariance Structure: Separability, Non-Separability, and Parameterization

Gaussian-based spatiotemporal models offer precise control over the separability and non-separability of space–time covariances. In DEMF, the non-separability parameter

$$\beta_s = 1 - \frac{\alpha_e}{\alpha}, \quad \alpha = \alpha_e + \alpha_s(\alpha_t - 1/2)$$

varies smoothly from fully separable ($\beta_s = 0$) to maximally non-separable ($\beta_s = 1$) (Lindgren et al., 2020). Spatial and temporal smoothness are respectively

$$\nu_s = \alpha - d/2, \quad \nu_t = \min(\alpha_t - 1/2, \nu_s/\alpha_s)$$

with practical correlation ranges interpretable analytically.

Generalized STGP models can further accommodate temporal evolution of spatial dependence (TESD) by making the eigenvalues of the spatial kernel time-dependent in Mercer's decomposition,

$$K_{s;t}(x, x') = \sum_{\ell=1}^L \lambda_\ell^2(t) \phi_\ell(x) \phi_\ell(x')$$

which, combined in a “quasi Kronecker-sum” with block-diagonal structure, captures arbitrary time dynamics in spatial correlation (Lan, 2019). This flexibility is substantiated both statistically and computationally (Lan, 2019).

3. Computational Methods: Efficient Inference and Sparse Representations

The combination of SPDEs and Gaussian Markov random field (GMRF) approximations enables sparsity of the latent precision matrix: $$\mathbf{Q}_{\mathbf{u}} = \gamma_e^2 \sum_{k=0}^{2\alpha_t} \gamma_t^k \mathbf{J}_{\alpha_t, k/2} \otimes \mathbf{K}_{\alpha_s(\alpha_t - k/2)+\alpha_e}$$ where temporal matrices $\mathbf{J}$ are banded and spatial $\mathbf{K}$ sparse, enabling scalable computation for massive space–time meshes (Lindgren et al., 2020). Kronecker and sum-of-Kronecker structures also appear in state-space and hierarchical GP frameworks (Axen et al., 2022, Lan, 2019).

For time–serial data, state-space GP representations using Kalman filtering drastically reduce computational cost. If the space–time kernel is separable and the temporal kernel has rational spectrum, the process over grid sites is exactly represented by a low-dimensional linear state-space model, with the Kalman filter state being sufficient for minimum-variance prediction of the entire field (Todescato et al., 2017). This approach delivers orders-of-magnitude speedup over batch GP methods for spatiotemporal interpolation and is robust to dynamic adaptation of spatial measurement grids (Todescato et al., 2017).

4. Applications in Physical, Environmental, and Machine Learning Domains

Gaussian-based spatiotemporal representations have broad applicability:

• Geostatistics and environmental science: For paleoclimate field reconstruction from multiple data sources, doubly-sparse GPs use spatial inducing points and a one-dimensional Markov chain for time. The computational cost becomes linear in data and the number of temporal steps, and cubic only in (small) state dimension (Axen et al., 2022).
• Structured regression and temporal modeling: In the context of sparse signal recovery and dynamic source localization (e.g., neural data, compressive video), hierarchical GP priors over spatial and temporal components capture evolving support patterns, outperforming one-level GPs and other baselines in support recovery and accuracy (Kuzin et al., 2018).
• Data assimilation and physics-informed learning: Physics-augmented multi-task GPs incorporate mesh-based spectral kernels and PDE-driven constraints to enforce physical consistency in predictive models (e.g., cardiac electrodynamics), yielding physically faithful and data-efficient surrogates (Zhang et al., 15 Oct 2025).

5. Explicit Gaussian Splatting for Spatiotemporal Scene Representation

For high-dimensional data including dynamic scene rendering, real-time CBCT motion reconstruction, and video compression, explicit sums of spatiotemporal Gaussians (“Gaussian splatting”) have emerged as the workhorse (Yang et al., 2023, Fu et al., 7 Jan 2025, Lee et al., 6 Mar 2025, Xie et al., 28 Mar 2025):

• Dynamic rendering: 4D Gaussian Splatting models the dynamic scene as an explicit sum of N 4D Gaussian primitives in $(x, y, z, t)$, combining end-to-end differentiable spatial/temporal parameterization (with full anisotropic covariance and SO(4) rotation) with 4D spherindrical harmonics for appearance (Yang et al., 2023).
• Medical imaging: Spatiotemporal Gaussian primitives with accompanying motion bases enable “one-shot” time-resolved CBCT reconstruction, rapidly capturing intra-breathing motion patterns from sparse projections without preexisting motion models (Xie et al., 28 Mar 2025).
• Video compression: Deformable Gaussian splatting in 2D, with parameters predicted via a compact time-conditional decoder, discards redundant storage by modeling only per-frame deformations of base Gaussians, leading to dramatic reductions in memory and computation (Lee et al., 6 Mar 2025).

A distinguishing property is the use of differentiable rasterization for images/projections, supporting efficient and analytic gradients for direct optimization of Gaussian parameters (Yang et al., 2023, Fu et al., 7 Jan 2025).

6. Advanced Topics: Anisotropic, Nonstationary, and Multitask Extensions

Gaussian-based spatiotemporal models admit rich extensions:

• Anisotropic covariance modeling: Explicit modeling of anisotropy in space and time (by decomposing covariances $\Sigma=R S S^\top R^\top$) encodes physical dependencies such as motion directionality, spatiotemporal scaling, and semantic variation (Yang et al., 13 Nov 2025, Guha et al., 2014, Wang et al., 17 Feb 2025).
• Nonstationarity: Allowing local adaptation in variances, smoothness, or range parameters via nonstationary SPDEs or dynamic eigenvalue processes extends representational capacity to settings with spatially nonuniform or transient dynamics (Lindgren et al., 2020, Lan, 2019).
• Multi-output/multitask structure: Linear models of coregionalization or multitask GP kernels encode statistical dependencies among multiple spatial–temporal fields, as in coupled physics or multivariate sensing (Zhang et al., 15 Oct 2025, Li et al., 29 Jun 2024).
• Spatiotemporal symmetry and invariance: Analytical models based on spatiotemporal Gaussian derivatives achieve covariant and, when normalized, invariant responses under combined spatial/geometric and temporal transformations (Lindeberg, 2023).

7. Significance, Limitations, and Computational Implementation

Gaussian-based spatiotemporal representations combine theoretical rigor, computational efficiency, and empirical effectiveness across scientific, medical, environmental, and computer vision domains. Their salient strengths include closed-form inference (for GPs and GMRFs), scalability via sparse or low-rank representations, and support for uncertainty quantification (Axen et al., 2022, Todescato et al., 2017, Lindgren et al., 2020, Lee et al., 6 Mar 2025). Limitations arise in handling ultra-complex geometry, non-Gaussian process noise, or truly high-dimensional nonseparable dependence not captured by parameterized kernels. Emerging directions focus on learning nonparametric or latent-process Gaussian operators, integrating physics knowledge, and further scaling to peta-scale space–time deployments.

References:

• “A diffusion-based spatio-temporal extension of Gaussian Matérn fields” (Lindgren et al., 2020)
• “Efficient Spatio-Temporal Gaussian Regression via Kalman Filtering” (Todescato et al., 2017)
• “Real-time Photorealistic Dynamic Scene Representation and Rendering with 4D Gaussian Splatting” (Yang et al., 2023)
• “Spatiotemporal Gaussian Optimization for 4D Cone Beam CT Reconstruction from Sparse Projections” (Fu et al., 7 Jan 2025)
• “GaussianVideo: Efficient Video Representation and Compression by Gaussian Splatting” (Lee et al., 6 Mar 2025)
• “Spatiotemporal modeling of European paleoclimate using doubly sparse Gaussian processes” (Axen et al., 2022)
• “Learning Temporal Evolution of Spatial Dependence with Generalized Spatiotemporal Gaussian Process Models” (Lan, 2019)
• “Physics-augmented Multi-task Gaussian Process for Modeling Spatiotemporal Dynamics” (Zhang et al., 15 Oct 2025)
• “Spatio-Temporal Structured Sparse Regression with Hierarchical Gaussian Process Priors” (Kuzin et al., 2018)
• “Gaussian Random Functional Dynamic Spatio-Temporal Modeling of Discrete Time Spatial Time Series Data” (Guha et al., 2014)
• “Unified theory for joint covariance properties under geometric image transformations for spatio-temporal receptive fields according to the generalized Gaussian derivative model for visual receptive fields” (Lindeberg, 2023)