Spatio-Temporal Gaussian Embedding (STGE)

• STGE is a design pattern where Gaussian objects encode both spatial structure and temporal dynamics, providing a unified representation across domains.
• It leverages techniques such as hierarchical conditioning, diffusion kernels, and feed-forward tokenization to effectively model dynamic processes.
• Empirical evaluations show that STGE improves tasks like rendering, event prediction, and uncertainty estimation in avatar modeling, point processes, video tokenization, and temporal networks.

Searching arXiv for the specified papers and closely related STGE terminology to ground the article in current preprints. Spatio-Temporal Gaussian Embedding (STGE) denotes a class of representations in which Gaussian objects encode spatial structure together with temporal evolution. Across recent arXiv literature, the term is not fully standardized: some papers do not use the acronym explicitly, yet instantiate the underlying idea by parameterizing motion, diffusion, or uncertainty with Gaussian primitives whose parameters depend on both spatial context and time. In dynamic avatar modeling, STGE refers to conditioning 3D Gaussian radiance fields on articulated pose and motion sequences; in spatio-temporal point processes, it appears as mixtures of Gaussian diffusion kernels with temporal decay; in video tokenization, it denotes feed-forward generation of 2D Gaussian tokens from video latents; and in temporal network analysis, related Gaussian embeddings represent time-varying latent positions and uncertainty, with a proposed extension that incorporates explicit spatial signals (Xu et al., 2024, Zhu et al., 2019, Chen et al., 15 Aug 2025, Romero et al., 2024).

1. Terminology, scope, and conceptual unification

The literature uses closely related but not identical terminology. The dynamic-avatar framework GAST does not explicitly use the acronym “STGE,” but it is described as embodying the concept by hierarchically integrating spatial and temporal information into a 3D Gaussian radiance field and its non-rigid warping (Xu et al., 2024). The point-process model NEST likewise does not explicitly use the term “Spatio-Temporal Gaussian Embedding,” but operationalizes an equivalent concept through learned mixtures of spatio-temporal Gaussian diffusion kernels parameterized by neural embeddings of source-event location (Zhu et al., 2019). In the video-tokenization setting, STGE is the explicit name of the feed-forward mechanism used by the Gaussian Video Transformer to convert latent video features into generative 2D Gaussian primitives (Chen et al., 15 Aug 2025). In temporal networks, TGNE is not itself named STGE, but the work presents a Gaussian temporal embedding framework and then formulates STGE as an extension that integrates explicit spatial features into both latent trajectories and the edge-formation model (Romero et al., 2024).

A useful synthesis is that STGE is not a single model family with one canonical parameterization; rather, it is a representational principle. The common structure is that a Gaussian object—primitive, kernel, or latent distribution—acts as the carrier of spatial geometry, while temporal dependence enters through conditioning, sequential context, diffusion time, interpolation, or temporal attention. This suggests that STGE is best understood as a cross-domain design pattern rather than a narrowly delimited architecture.

The following table summarizes the principal instantiations described in the cited works.

Domain	STGE instantiation	Gaussian object
Dynamic human avatars	Hierarchical spatio-temporal conditioning of non-rigid deformation before skinning	3D Gaussians in a canonical radiance field
Spatio-temporal point processes	Learned event influence via neural mixtures of diffusion kernels	Spatial Gaussian kernels with temporal decay
Video tokenization	Feed-forward generation of adaptive video tokens	2D Gaussian primitives
Temporal networks	Time-varying latent uncertainty, with spatial extension proposed	Gaussian node embeddings

This cross-domain breadth also clarifies a frequent misconception: STGE is not synonymous with Gaussian splatting. In GAST and GVT, Gaussian splatting is central to rendering or rasterization, but in NEST and TGNE the Gaussian component instead appears in diffusion kernels or latent posterior distributions. The shared element is the Gaussian parameterization of space-time structure, not any single rendering pipeline.

2. Dynamic-avatar STGE in canonical-space Gaussian radiance fields

In the canonical-space avatar paradigm, a static canonical radiance field is warped per frame to observation space by articulated transformation together with residual non-rigid deformation. In GAST, the canonical radiance field is a set of 3D Gaussians initialized at SMPL(-X) template vertices in T-pose, with canonical parameters consisting of mean $x \in \mathbb{R}^3$ (also denoted $\mu$), covariance $\Sigma \in \mathbb{R}^{3\times 3}$, opacity $\alpha$, and color feature $sh$ (Xu et al., 2024). The covariance is decomposed as

$\Sigma = R S S^\top R^\top,$

with $R$ derived from a quaternion rotation parameter and $S$ from a diagonal scaling vector. Under 3D Gaussian splatting, the projected covariance is

$\Sigma' = J W \Sigma W^\top J^\top,$

and ray-wise color is alpha-composited as

$$C = \sum_{i \in N} c_i \alpha'_i \prod_{j=1}^{i-1}(1-\alpha'_j).$$

What makes this an STGE is the conditioning mechanism that drives non-rigid deformation. GAST defines a hierarchical spatio-temporal context composed of three elements: a coarse skeletal motion sequence $\mu$0 capturing global pose variation across a temporal window at stride $\mu$1; a fine point-wise velocity sequence $\mu$2 derived from displacements of SMPL template vertices; and multi-scale sampling over strides $\mu$3 in a coarse-to-fine manner. Sequential conditioning means that the deformation network is conditioned not only on the current pose $\mu$4, but also on motion-history embeddings $\mu$5 and $\mu$6 (Xu et al., 2024).

The per-frame warp is the composition of non-rigid deformation and skinning. For Gaussian $\mu$7, GAST predicts residual updates

$\mu$8

followed by

$\mu$9

The observation-space parameters are then obtained through skinning:

$\Sigma \in \mathbb{R}^{3\times 3}$0

where the LBS weights are predicted as template weights plus learned offsets:

$\Sigma \in \mathbb{R}^{3\times 3}$1

The spatial and temporal components are explicitly separable in the conditioning. The skeletal residuals are computed through relative rotations using Rodrigues, while local motion is encoded by velocities

$\Sigma \in \mathbb{R}^{3\times 3}$2

and local Gaussian-specific velocity embeddings are formed through KNN on template vertices. Multi-scale embeddings are then defined as

$\Sigma \in \mathbb{R}^{3\times 3}$3

The stated purpose of this coarse-to-fine sampling is to provide “unbiased inputs” to the non-rigid network by fusing long-range trends with short-range detail (Xu et al., 2024).

The optimization objective aggregates photometric, perceptual, and mask supervision:

$\Sigma \in \mathbb{R}^{3\times 3}$4

with rendered images produced by splatting observation-space Gaussians. The training pipeline initializes Gaussians at template vertices, estimates SMPL(-X) parameters from multi-view sequences, builds multi-scale temporal windows, computes skeletal residuals and velocity embeddings, predicts non-rigid residuals, refines pose, skins to observation space, renders, and jointly optimizes all components end to end.

Empirically, the paper positions this STGE against two deficient extremes: pose-only avatars, which capture articulation but miss garment-induced non-rigid motion, and pure temporal embeddings for dynamic scenes, which render well but are hard to animate parametrically. On DNA-Rendering and I3D-Human, GAST is reported to outperform concurrent Gaussian-avatar baselines on PSNR and SSIM across the reported sequences, while ablations show incremental gains from adding $\Sigma \in \mathbb{R}^{3\times 3}$5, then $\Sigma \in \mathbb{R}^{3\times 3}$6, then multi-scale sampling (Xu et al., 2024). A direct implication is that, in this setting, STGE is principally a deformation-conditioning mechanism rather than a modification of opacity or color features, since the paper states that there is no explicit sequential modulation of $\Sigma \in \mathbb{R}^{3\times 3}$7 and $\Sigma \in \mathbb{R}^{3\times 3}$8.

3. Spatio-temporal Gaussian kernels as event embeddings in point processes

In spatio-temporal point processes, STGE takes a different form: instead of warping Gaussian primitives in a rendered scene, it encodes how past events influence future events. NEST formulates the conditional intensity as a constant background rate plus an excitation term over prior events,

$\Sigma \in \mathbb{R}^{3\times 3}$9

where the excitation kernel is a mixture of Gaussian diffusion kernels,

$\alpha$0

Each component has a source-location-dependent mean shift $\alpha$1, anisotropic covariance $\alpha$2, and mixture weight $\alpha$3 satisfying $\alpha$4 (Zhu et al., 2019).

The diffusion kernel is

$\alpha$5

This structure combines exponential temporal decay with spatial Gaussian diffusion whose spread depends on diffusion time. The covariance matrix is parameterized as

$\alpha$6

which allows anisotropy and directional dependence.

The neural-embedding aspect appears through a latent spatial embedding

$\alpha$7

produced by a multilayer fully connected network. From this embedding, the model decodes shift, scale, anisotropy, and mixture weights. For example,

$\alpha$8

$\alpha$9

and

$sh$0

These parameters jointly define an interpretable Gaussian embedding of event influence (Zhu et al., 2019).

A central property of this formulation is tractability. The spatio-temporal likelihood involves the standard point-process term

$sh$1

and the integral is approximately available in closed form because the kernel is Gaussian in space and exponential in time. This tractability supports both maximum-likelihood training and the paper’s imitation-learning formulation, in which the model policy is explicitly related to intensity and optimized through an MMD-based worst-case reward in an RKHS (Zhu et al., 2019).

The empirical positioning of this STGE differs from that of avatar and video models. Here, the principal claims concern distributional fidelity, robustness, and interpretability. The paper reports that NEST+IL consistently improves one-step prediction MSE and MMD relative to ETAS and often relative to NEST+MLE on both synthetic and real datasets, while parameter maps of $sh$2, $sh$3, and $sh$4 are presented as interpretable summaries of diffusion geometry. This points to a second major interpretation of STGE: not as a scene representation, but as a learned geometry of influence propagation over space and time.

4. Feed-forward STGE for generative video tokenization

In the Gaussian Video Transformer, STGE is the explicit feed-forward module that maps latent “rigid” video features into a set of time-indexed 2D Gaussian primitives (Chen et al., 15 Aug 2025). The problem setting is distinct from both avatar reconstruction and point-process modeling: fixed-grid, patch-wise tokenizers over-encode low-information regions spatially and fail to separate static from dynamic content temporally. STGE addresses this by generating explicit Gaussian tokens with learned positions, orientations, scales, and coefficients in a single pass, without per-video optimization.

Given an encoder output $sh$5, STGE maps the latent tensor to Gaussian sets

$sh$6

Each 2D Gaussian is parameterized by mean $sh$7, covariance $sh$8, and coefficient vector $sh$9. The covariance is factorized through orientation $\Sigma = R S S^\top R^\top,$0 and anisotropic scales $\Sigma = R S S^\top R^\top,$1:

$\Sigma = R S S^\top R^\top,$2

ensuring positive definiteness for $\Sigma = R S S^\top R^\top,$3 (Chen et al., 15 Aug 2025).

The Gaussian density is standard, but rasterization uses the exponential term as an unnormalized radiance weight,

$\Sigma = R S S^\top R^\top,$4

At a latent token site $\Sigma = R S S^\top R^\top,$5, the rasterized token is

$\Sigma = R S S^\top R^\top,$6

There is no alpha compositing or occlusion ordering; rasterization is a linear latent-space blend. Spatial adaptability follows from the learned positions and covariances, which concentrate Gaussian energy in informative regions and assign lower weight to uniform areas (Chen et al., 15 Aug 2025).

The generative mapping from latent features to Gaussian parameters is organized through two modules. Spatio-Temporal Attention alternates spatial self-attention within each frame and temporal self-attention across corresponding spatial sites. Deformable Spatio-Temporal Fusion performs deformable cross-attention from latent rigid features onto Gaussian queries anchored at Gaussian positions:

$\Sigma = R S S^\top R^\top,$7

where offsets and attention weights are produced from the queries, and $\Sigma = R S S^\top R^\top,$8 denotes bilinear sampling. Residual MLPs then update Gaussian parameters,

$\Sigma = R S S^\top R^\top,$9

and after $R$0 fusion iterations, an MLP head regresses final parameters $R$1 (Chen et al., 15 Aug 2025).

Temporal compactness is handled through Gaussian Set Partitioning rather than through the Gaussian parameterization alone. A learned binary mask separates static and dynamic Gaussian indices using a straight-through estimator:

$R$2

Static Gaussians are kept only at $R$3 and reused for later frames; dynamic Gaussians remain time-indexed. The corresponding regularizer,

$R$4

encourages compact dynamic subsets (Chen et al., 15 Aug 2025).

The training objective is

$R$5

with coefficient vectors vector-quantized using a $R$6 codebook. Optimization uses Adam with learning rate $R$7, pretraining on ImageNet-1K and fine-tuning on UCF101, K600, and DAVIS-2017.

The paper reports that GVT achieves lower rFVD than MAGVIT-v2 on UCF101 and K600, improves action-recognition accuracy in VideoMAE evaluations, and reaches SSIM $R$8 and LPIPS $R$9 on DAVIS-2017-480p with feed-forward inference. The ablation most directly isolating the spatio-temporal value of STGE shows that adding STA to a spatial-only attention variant reduces rFVD from $S$0 to $S$1 on UCF101, while the learned GSP reduces tokens from $S$2 to $S$3 with rFVD $S$4; fixed partitioning severely degrades to rFVD $S$5 (Chen et al., 15 Aug 2025). This indicates that, in this formulation, STGE is both a token generator and a content-adaptive allocation mechanism.

5. Gaussian temporal embeddings, uncertainty, and the proposed spatial extension in networks

TGNE addresses continuous-time temporal graphs by embedding each node as a time-varying Gaussian distribution in latent space,

$S$6

with trajectories defined by piece-wise linear interpolation between critical times $S$7 and a mean-field variational posterior over critical points

$S$8

Within an interval $S$9, if $\Sigma' = J W \Sigma W^\top J^\top,$0, then

$\Sigma' = J W \Sigma W^\top J^\top,$1

and, from independence under the variational posterior,

$\Sigma' = J W \Sigma W^\top J^\top,$2

The work states that this covariance formula is implied rather than explicitly written in the paper (Romero et al., 2024).

The decoder adopts a continuous-time latent-position point-process model. For the Euclidean-distance version,

$\Sigma' = J W \Sigma W^\top J^\top,$3

and the log-likelihood decomposes over dyads and intervals through cumulative rates $\Sigma' = J W \Sigma W^\top J^\top,$4. For piece-wise linear trajectories, the paper derives a closed-form cumulative rate for the Euclidean decoder; when such closed form is unavailable, a Riemann approximation is used (Romero et al., 2024).

Inference proceeds by minimizing the negative ELBO,

$\Sigma' = J W \Sigma W^\top J^\top,$5

with a Gaussian random walk prior over critical-point trajectories and a reparameterized Monte Carlo estimate of the expected log-likelihood. To scale to sparse temporal graphs, TGNE uses negative sampling and node-batching; the implementation is in Pyro with Adam, using learning rates $\Sigma' = J W \Sigma W^\top J^\top,$6 for variational parameters and $\Sigma' = J W \Sigma W^\top J^\top,$7 for the bias $\Sigma' = J W \Sigma W^\top J^\top,$8 (Romero et al., 2024).

The central contribution of TGNE to the broader STGE discussion is uncertainty. Node-level uncertainty is summarized by

$\Sigma' = J W \Sigma W^\top J^\top,$9

and is reported to correlate with local sparsity and temporal degree. Edge-level posterior predictive uncertainty is estimated by Monte Carlo variability of cumulative rates. The reported findings are that uncertainty decreases with observed interaction count and that the strength of this relationship depends on prior scale $$C = \sum_{i \in N} c_i \alpha'_i \prod_{j=1}^{i-1}(1-\alpha'_j).$$0, illustrating a bias-variance trade-off (Romero et al., 2024).

The paper then formulates STGE as a spatio-temporal extension of TGNE. The proposed augmented latent variable is

$$C = \sum_{i \in N} c_i \alpha'_i \prod_{j=1}^{i-1}(1-\alpha'_j).$$1

with

$$C = \sum_{i \in N} c_i \alpha'_i \prod_{j=1}^{i-1}(1-\alpha'_j).$$2

If the spatial component is latent, it can follow the same piece-wise linear Gaussian trajectory construction; if observed, it may be treated as a noisy observation with fixed measurement covariance. The proposed spatially informed decoder becomes

$$C = \sum_{i \in N} c_i \alpha'_i \prod_{j=1}^{i-1}(1-\alpha'_j).$$3

or, in anisotropic form,

$$C = \sum_{i \in N} c_i \alpha'_i \prod_{j=1}^{i-1}(1-\alpha'_j).$$4

with $$C = \sum_{i \in N} c_i \alpha'_i \prod_{j=1}^{i-1}(1-\alpha'_j).$$5 and $$C = \sum_{i \in N} c_i \alpha'_i \prod_{j=1}^{i-1}(1-\alpha'_j).$$6 required to be SPD (Romero et al., 2024).

This is a proposal rather than an experimentally validated model in the paper’s main results. Accordingly, the defensible interpretation is that TGNE provides a probabilistic template for STGE in relational data: Gaussian embeddings become spatio-temporal once explicit spatial features are inserted into the latent state and the event-intensity function.

6. Comparative structure, misconceptions, and limitations

Across these works, four recurrent components define the practical content of STGE. First, spatial structure is made explicit through Gaussian means and covariances, whether in 3D avatar geometry, 2D image-plane splats, spatial event kernels, or node uncertainty ellipsoids. Second, temporal information is not merely appended as a scalar timestamp; it modulates the Gaussian representation through motion sequences, exponential decay, piece-wise linear interpolation, or alternating temporal attention. Third, the Gaussian parameterization is usually chosen for tractability: it supports rasterization in GAST and GVT, closed-form or approximately closed-form integrals in NEST and TGNE, and interpretable decomposition into position, scale, orientation, and uncertainty (Xu et al., 2024, Zhu et al., 2019, Chen et al., 15 Aug 2025, Romero et al., 2024). Fourth, all four works emphasize explicit geometry rather than purely opaque embeddings.

A common misconception is that STGE necessarily implies a standardized acronym or a single benchmarked model class. The cited papers show the opposite. GAST and NEST instantiate the concept without naming it STGE, GVT uses the term directly for a feed-forward tokenizer, and TGNE presents a related Gaussian-trajectory framework plus a proposed spatial extension. A second misconception is that “temporal Gaussian embedding” refers only to temporal smoothing. In the avatar and video settings, temporal structure changes the Gaussian parameters or their conditioning in a way that directly affects rendering and animation; in NEST, it changes the influence kernel itself through diffusion time and exponential decay; in TGNE, it determines both latent position evolution and posterior uncertainty.

The limitations are likewise domain-specific. In GAST, multi-scale sequence encoding and KNN velocity embeddings increase computation and memory relative to pose-only Gaussian avatars, and fast topologically complex dynamics such as extreme garment flutter remain difficult; the paper also notes needle-like artifacts typical of 3DGS, which can hurt LPIPS relative to NeRF-based methods such as Dyco (Xu et al., 2024). In NEST, efficiency depends on tractable Gaussian integrals and thinning-based sampling, while increasing the number of Gaussian components raises training cost; NEST+MLE is generally faster than NEST+IL because sampling is the bottleneck in imitation learning (Zhu et al., 2019). In GVT, severe motion or occlusion can force many dynamic Gaussians, increasing cost, and some compression cases remain worse than H.264; the work also notes the broader limitation that the 2DGS ecosystem lacks large pretrained models and alignment for downstream text-to-video applications (Chen et al., 15 Aug 2025). In TGNE, the limitations include piece-wise linear trajectories, isotropic diagonal covariances, transductivity, and potential rotational drift at high prior scales; the proposed STGE extension may also lose the closed-form cumulative rate when spatial terms are non-Euclidean (Romero et al., 2024).

Taken together, these works suggest that STGE is most valuable where three requirements coincide: explicit spatial geometry, nontrivial temporal dependence, and a need for interpretable or computationally manageable parameterizations. The exact role of the Gaussian object varies—from rendered primitive to diffusion kernel to latent posterior—but in each case the Gaussian form is the device through which space and time are fused into a single operational representation.