---
title: Gaussian-Based Spatiotemporal Representation
url: https://www.emergentmind.com/topics/gaussian-based-spatiotemporal-representation
type: topic
---

# Gaussian-Based Spatiotemporal Representation

A Gaussian-based spatiotemporal representation models joint spatial and temporal structure by decomposing observed data, fields, or dynamics into sums, mixtures, or fields of Gaussian functions parameterized over both space and time. This paradigm encompasses a wide array of methodologies across statistical modeling, physics-based inference, video and dynamic scene synthesis, and learning compact representations for high-dimensional spatiotemporal data. Core constructions include stochastic partial differential equation (SPDE) formulations yielding Matérn-type covariances, state-space models with Gaussian process (GP) priors, learnable sums of space–time Gaussian primitives for graphics and imaging, and explicit anisotropic Gaussians parameterized over space or spacetime. By leveraging analytic properties of Gaussians—such as closure under convolution, tractable marginalization, and closed-form operations under affine transformations—these models offer both theoretical tractability and computational efficiency, as well as practical advantages including uncertainty quantification, differentiability, and flexible representation of non-stationary phenomena.

## 1. SPDE and Gaussian Matérn Fields for Spatiotemporal Modeling

A foundational class of spatiotemporal Gaussian representations is constructed via SPDEs, where the solution field is defined as the response to space–time white noise filtered by a differential operator. Lindgren et al. devised the diffusion-based extension of the Matérn field (DEMF), defined by
\[
\Bigl(-\gamma_t^2\,\partial_{tt} + L_s^{\alpha_s}\Bigr)^{\alpha_t/2}\,u(s,t) = \dot{\mathcal E}_{Q_s}(s,t)
\]
for \(s \in D \subseteq \mathbb R^d\), where \(L_s = \gamma_s^2 - \Delta\) and \(\dot{\mathcal E}_{Q_s}\) is white in time and correlated in space through precision \(Q_s\) [2006.04917]. The stationary marginal at fixed \(t\) yields a spatial Matérn correlation, controlled by variance \(\sigma^2\), smoothness \(\nu\), and correlation length; temporal marginals and nonstationarity are governed by additional parameters \(\gamma_t, \alpha_t\). The spatiotemporal spectral density is
\[
S_u(\omega_s, \omega_t) = \frac{1}{(2\pi)^{d+1} \gamma_e^2 [\gamma_t^2 \omega_t^2 + (\gamma_s^2 + \|\omega_s\|^2)^{\alpha_s}]^{\alpha_t} (\gamma_s^2 + \|\omega_s\|^2)^{\alpha_e}}
\]
enabling explicit control over separability and degrees of smoothness in both domains. 

The SPDE framework extends to curved manifolds by replacing the Laplacian with the Laplace–Beltrami operator and to non-stationary or spatially heterogeneous domains via coefficient functions or operators [2006.04917]. These models admit sparse finite-element approximations, making them tractable for large-scale statistical inference via the R-INLA software [2006.04917].

## 2. Covariance Structure: Separability, Non-Separability, and Parameterization

Gaussian-based spatiotemporal models offer precise control over the separability and non-separability of space–time covariances. In DEMF, the non-separability parameter
\[
\beta_s = 1 - \frac{\alpha_e}{\alpha}, \quad \alpha = \alpha_e + \alpha_s(\alpha_t - 1/2)
\]
varies smoothly from fully separable (\(\beta_s = 0\)) to maximally non-separable (\(\beta_s = 1\)) [2006.04917]. Spatial and temporal smoothness are respectively
\[
\nu_s = \alpha - d/2, \quad \nu_t = \min(\alpha_t - 1/2, \nu_s/\alpha_s)
\]
with practical correlation ranges interpretable analytically.

Generalized STGP models can further accommodate temporal evolution of spatial dependence (TESD) by making the eigenvalues of the spatial kernel time-dependent in Mercer's decomposition,
\[
K_{s;t}(x, x') = \sum_{\ell=1}^L \lambda_\ell^2(t) \phi_\ell(x) \phi_\ell(x')
\]
which, combined in a “quasi Kronecker-sum” with block-diagonal structure, captures arbitrary time dynamics in spatial correlation [1901.04030]. This flexibility is substantiated both statistically and computationally [1901.04030].

## 3. Computational Methods: Efficient Inference and Sparse Representations

The combination of SPDEs and Gaussian Markov random field (GMRF) approximations enables sparsity of the latent precision matrix:
\[
\mathbf{Q}_{\mathbf{u}} = \gamma_e^2 \sum_{k=0}^{2\alpha_t} \gamma_t^k \mathbf{J}_{\alpha_t, k/2} \otimes \mathbf{K}_{\alpha_s(\alpha_t - k/2)+\alpha_e}
\]
where temporal matrices \(\mathbf{J}\) are banded and spatial \(\mathbf{K}\) sparse, enabling scalable computation for massive space–time meshes [2006.04917]. Kronecker and sum-of-Kronecker structures also appear in state-space and hierarchical GP frameworks [2211.08160, 1901.04030].

For time–serial data, state-space GP representations using Kalman filtering drastically reduce computational cost. If the space–time kernel is separable and the temporal kernel has rational spectrum, the process over grid sites is exactly represented by a low-dimensional linear state-space model, with the Kalman filter state being sufficient for minimum-variance prediction of the entire field [1705.01485]. This approach delivers orders-of-magnitude speedup over batch GP methods for spatiotemporal interpolation and is robust to dynamic adaptation of spatial measurement grids [1705.01485].

## 4. Applications in Physical, Environmental, and Machine Learning Domains

Gaussian-based spatiotemporal representations have broad applicability:

- **Geostatistics and environmental science:** For paleoclimate field reconstruction from multiple data sources, doubly-sparse GPs use spatial inducing points and a one-dimensional Markov chain for time. The computational cost becomes linear in data and the number of temporal steps, and cubic only in (small) state dimension [2211.08160].
- **Structured regression and temporal modeling:** In the context of sparse signal recovery and dynamic source localization (e.g., neural data, compressive video), hierarchical GP priors over spatial and temporal components capture evolving support patterns, outperforming one-level GPs and other baselines in support recovery and accuracy [1807.05561].
- **Data assimilation and physics-informed learning:** Physics-augmented multi-task GPs incorporate mesh-based spectral kernels and PDE-driven constraints to enforce physical consistency in predictive models (e.g., cardiac electrodynamics), yielding physically faithful and data-efficient surrogates [2510.13601].

## 5. Explicit Gaussian Splatting for Spatiotemporal Scene Representation

For high-dimensional data including dynamic scene rendering, real-time CBCT motion reconstruction, and video compression, explicit sums of spatiotemporal Gaussians (“Gaussian splatting”) have emerged as the workhorse [2310.10642, 2501.04140, 2503.04333, 2503.22139]:

- **Dynamic rendering:** 4D Gaussian Splatting models the dynamic scene as an explicit sum of N 4D Gaussian primitives in \((x, y, z, t)\), combining end-to-end differentiable spatial/temporal parameterization (with full anisotropic covariance and SO(4) rotation) with 4D spherindrical harmonics for appearance [2310.10642].
- **Medical imaging:** Spatiotemporal Gaussian primitives with accompanying motion bases enable “one-shot” time-resolved CBCT reconstruction, rapidly capturing intra-breathing motion patterns from sparse projections without preexisting motion models [2503.22139].
- **Video compression:** Deformable Gaussian splatting in 2D, with parameters predicted via a compact time-conditional decoder, discards redundant storage by modeling only per-frame deformations of base Gaussians, leading to dramatic reductions in memory and computation [2503.04333].

A distinguishing property is the use of differentiable rasterization for images/projections, supporting efficient and analytic gradients for direct optimization of Gaussian parameters [2310.10642, 2501.04140].

## 6. Advanced Topics: Anisotropic, Nonstationary, and Multitask Extensions

Gaussian-based spatiotemporal models admit rich extensions:

- **Anisotropic covariance modeling:** Explicit modeling of anisotropy in space and time (by decomposing covariances \(\Sigma=R S S^\top R^\top\)) encodes physical dependencies such as motion directionality, spatiotemporal scaling, and semantic variation [2511.10060, 1405.6531, 2502.14895].
- **Nonstationarity:** Allowing local adaptation in variances, smoothness, or range parameters via nonstationary SPDEs or dynamic eigenvalue processes extends representational capacity to settings with spatially nonuniform or transient dynamics [2006.04917, 1901.04030].
- **Multi-output/multitask structure:** Linear models of coregionalization or multitask GP kernels encode statistical dependencies among multiple spatial–temporal fields, as in coupled physics or multivariate sensing [2510.13601, 2407.00397].
- **Spatiotemporal symmetry and invariance:** Analytical models based on spatiotemporal Gaussian derivatives achieve covariant and, when normalized, invariant responses under combined spatial/geometric and temporal transformations [2311.10543].

## 7. Significance, Limitations, and Computational Implementation

Gaussian-based spatiotemporal representations combine theoretical rigor, computational efficiency, and empirical effectiveness across scientific, medical, environmental, and computer vision domains. Their salient strengths include closed-form inference (for GPs and GMRFs), scalability via sparse or low-rank representations, and support for uncertainty quantification [2211.08160, 1705.01485, 2006.04917, 2503.04333]. Limitations arise in handling ultra-complex geometry, non-Gaussian process noise, or truly high-dimensional nonseparable dependence not captured by parameterized kernels. Emerging directions focus on learning nonparametric or latent-process Gaussian operators, integrating physics knowledge, and further scaling to peta-scale space–time deployments.

**References:**

- “A diffusion-based spatio-temporal extension of Gaussian Matérn fields” [2006.04917]
- “Efficient Spatio-Temporal Gaussian Regression via Kalman Filtering” [1705.01485]
- “Real-time Photorealistic Dynamic Scene Representation and Rendering with 4D Gaussian Splatting” [2310.10642]
- “Spatiotemporal Gaussian Optimization for 4D Cone Beam CT Reconstruction from Sparse Projections” [2501.04140]
- “GaussianVideo: Efficient Video Representation and Compression by Gaussian Splatting” [2503.04333]
- “Spatiotemporal modeling of European paleoclimate using doubly sparse Gaussian processes” [2211.08160]
- “Learning Temporal Evolution of Spatial Dependence with Generalized Spatiotemporal Gaussian Process Models” [1901.04030]
- “Physics-augmented Multi-task Gaussian Process for Modeling Spatiotemporal Dynamics” [2510.13601]
- “Spatio-Temporal Structured Sparse Regression with Hierarchical Gaussian Process Priors” [1807.05561]
- “Gaussian Random Functional Dynamic Spatio-Temporal Modeling of Discrete Time Spatial Time Series Data” [1405.6531]
- “Unified theory for joint covariance properties under geometric image transformations for spatio-temporal receptive fields according to the generalized Gaussian derivative model for visual receptive fields” [2311.10543]

Source: https://www.emergentmind.com/topics/gaussian-based-spatiotemporal-representation