Decoupled ST-GS: Spatio-Temporal Gaussian Models

• Decoupled ST-GS is a framework that explicitly separates static and dynamic Gaussian modeling in spatio-temporal data.
• It applies to high-dynamic scene reconstruction by using clustering and event-based tracking to optimize the modeling of moving objects and static backgrounds.
• The approach improves rendering accuracy and temporal continuity through specialized loss functions and a fusion strategy that balances static and dynamic contributions.

Decoupled ST-GS refers to a set of frameworks and methodologies centered on the separation—or decoupling—of spatio-temporal components in Gaussian-based generative or learning models. This decoupling arises from the need to address fundamental mismatches and inefficiencies that emerge when treating heterogeneous phenomena (e.g., static and dynamic scene elements, forward and backward optimization flows, or discrete and continuous processes) in a unified or entangled fashion. Across domains such as high-dynamic scene reconstruction, deep discrete optimization, and quantum spin chains, "Decoupled ST-GS" denotes architectures and algorithms explicitly designed to disentangle and separately process spatial and temporal variables, thereby improving fidelity, interpretability, optimization stability, and physical plausibility.

1. Principles of Spatio-Temporal Decoupling in Gaussian Models

Decoupled ST-GS frameworks focus on representing complex scenes or statistical processes as collections of Gaussian primitives, with explicit separation of spatial and temporal attributes. In high-dynamic scene reconstruction, this takes the form of representing static backgrounds and moving objects with distinct Gaussian components—static Gaussians encode rigid, temporally invariant structure, while dynamic (4D) Gaussians track non-rigid, temporally varying elements via means and covariances defined over spacetime $(x, y, z, t)$ (Zhou et al., 29 Jun 2025). Dissimilar image features (spatial) and event-based motion cues (temporal) are extracted and clustered to assign Gaussians to their appropriate static or dynamic role. This separation enables the design of pipelines where background and foreground are handled by specialized modules, reducing the incidence of spatio-temporal mismatches and temporal aliasing in novel view synthesis.

In mathematical terms, the scene density and radiance fields are modeled as: $$\rho(X,t) = \sum_{i=1}^{N_s} \sigma^s_i \,\exp\left(-\tfrac12 (X-\mu^s_i)^\top (\Sigma^s_i)^{-1}(X-\mu^s_i)\right) + \sum_{j=1}^{N_d} \sigma^d_j \,\exp\left(-\tfrac12 \left([X,t]-\mu^d_j\right)^\top (\Sigma^d_j)^{-1}([X,t]-\mu^d_j)\right),$$ where static Gaussians $(\mu^s_i, \Sigma^s_i)$ and dynamic Gaussians $(\mu^d_j, \Sigma^d_j)$ are handled separately at all pipeline stages (Zhou et al., 29 Jun 2025).

2. Decoupled ST-GS in Dynamic Scene Reconstruction

High-dynamic scene reconstruction is a primary domain for Decoupled ST-GS frameworks. Unified representations are prone to errors because frame-based imaging is ill-suited for dynamic objects that exhibit motion discontinuities, while backgrounds require rigid, temporally stable modeling. Decoupling is operationalized by:

• Extracting spatial (e.g., superpixel, CNN) and temporal (event-based) features.
• Clustering these features to segment the scene into static and dynamic regions (e.g., via K-Means, ClusterFormer).
• Tracking dynamic object patches through time using event-based correlation and Extended Kalman Filters, enabling continuous representations of object trajectories.
• Regularizing object Gaussians so that their color and motion derivatives are tightly coupled to event-derived appearance and flow statistics, enforcing physical and temporal plausibility via

$$\mathcal L_{consis} = \sum_{j=1}^{N_d} \|c^d_j - L_{ev,j}\|_1 + \|\dot\mu^d_j - F_{ev,j}\|_1$$

where $L_{ev,j}$ and $F_{ev,j}$ are event-derived color and velocity proxies (Zhou et al., 29 Jun 2025).

This decoupling, followed by fusion at the rendering stage via shadow-aware blending and color fusion, yields sharp, temporally continuous renderings and supports arbitrary time interpolation, outperforming unified or undisentangled baselines by 2–6 dB PSNR on challenging datasets (Zhou et al., 29 Jun 2025).

3. Clustering, Modeling, and Fusion Strategies

Scene decoupling is anchored in robust multimodal clustering and continuous model assignment:

• Clustering is performed in a joint spatial-temporal feature space, merging per-patch appearance descriptors with windowed event-based motion representations.
• This produces pseudo-labels (e.g., $y_p \in \{0,1\}$) indicating static or dynamic roles.
• The model maintains two Gaussian sets: static Gaussians with constant spatial parameters, and dynamic Gaussians whose parameters evolve through time-dependent functions, regularly updated via event-informed tracking.
• Rendering fuses the contributions via shadow overlap and color mixing: $w_{\rm fused}(p) = (1-\rho)w^s(p) + \rho w^d(p)$

$$c_{\rm fused}(p) = \frac{\sigma^s}{\sigma^s + \sigma^d}(1-\rho)c^s + \frac{\sigma^d}{\sigma^s + \sigma^d}\rho c^d$$

enabling accurate reconstruction of scenes with strong foreground motion and stationary backgrounds (Zhou et al., 29 Jun 2025).

4. Loss Functions and Training Regimes

Decoupled ST-GS approaches are characterized by targeted loss decompositions:

• Reconstruction loss (per-pixel color and structure similarity): $$\rho(X,t) = \sum_{i=1}^{N_s} \sigma^s_i \,\exp\left(-\tfrac12 (X-\mu^s_i)^\top (\Sigma^s_i)^{-1}(X-\mu^s_i)\right) + \sum_{j=1}^{N_d} \sigma^d_j \,\exp\left(-\tfrac12 \left([X,t]-\mu^d_j\right)^\top (\Sigma^d_j)^{-1}([X,t]-\mu^d_j)\right),$$0
• Clustering consistency loss: $$\rho(X,t) = \sum_{i=1}^{N_s} \sigma^s_i \,\exp\left(-\tfrac12 (X-\mu^s_i)^\top (\Sigma^s_i)^{-1}(X-\mu^s_i)\right) + \sum_{j=1}^{N_d} \sigma^d_j \,\exp\left(-\tfrac12 \left([X,t]-\mu^d_j\right)^\top (\Sigma^d_j)^{-1}([X,t]-\mu^d_j)\right),$$1 to enforce patch assignment consistency and intra-class feature compactness.
• Event-based spatio-temporal consistency: $$\rho(X,t) = \sum_{i=1}^{N_s} \sigma^s_i \,\exp\left(-\tfrac12 (X-\mu^s_i)^\top (\Sigma^s_i)^{-1}(X-\mu^s_i)\right) + \sum_{j=1}^{N_d} \sigma^d_j \,\exp\left(-\tfrac12 \left([X,t]-\mu^d_j\right)^\top (\Sigma^d_j)^{-1}([X,t]-\mu^d_j)\right),$$2
• The total loss in fine-tuning is thus: $$\rho(X,t) = \sum_{i=1}^{N_s} \sigma^s_i \,\exp\left(-\tfrac12 (X-\mu^s_i)^\top (\Sigma^s_i)^{-1}(X-\mu^s_i)\right) + \sum_{j=1}^{N_d} \sigma^d_j \,\exp\left(-\tfrac12 \left([X,t]-\mu^d_j\right)^\top (\Sigma^d_j)^{-1}([X,t]-\mu^d_j)\right),$$3 A two-stage training protocol starts with pure reconstruction to initialize the model, then incorporates disentanglement and consistency regularization to refine dynamic object trajectories and appearance (Zhou et al., 29 Jun 2025).

Empirical ablations demonstrate that each component—scene-level disentanglement, object-based tracking, cluster-based assignment, and event-guided consistency—contributes measurable performance improvements, with full decoupling plus fusion attaining the highest quantitative and qualitative scores.

5. Architecture and Implementation Considerations

A typical pipeline includes:

• Frame extraction and event accumulation.
• Preprocessing to extract spatial/temporal features and perform K-Means clustering.
• Gaussian parameterization, with object-level dynamic Gaussians receiving continuous-time updates from EKF or learned motion networks.
• Differentiable rendering for both static and dynamic Gaussians, with alpha-compositing, color and shadow-aware blending.
• Training executed in two phases on GPU, with all major modules (feature extraction, clustering, motion tracking, Gaussian projection/splatting, loss computation) GPU-accelerated for real-time throughput.

A table summarizing Gaussian role separation appears below:

Component	Static Gaussians	Dynamic Gaussians
Parameterization	$$\rho(X,t) = \sum_{i=1}^{N_s} \sigma^s_i \,\exp\left(-\tfrac12 (X-\mu^s_i)^\top (\Sigma^s_i)^{-1}(X-\mu^s_i)\right) + \sum_{j=1}^{N_d} \sigma^d_j \,\exp\left(-\tfrac12 \left([X,t]-\mu^d_j\right)^\top (\Sigma^d_j)^{-1}([X,t]-\mu^d_j)\right),$$4	$$\rho(X,t) = \sum_{i=1}^{N_s} \sigma^s_i \,\exp\left(-\tfrac12 (X-\mu^s_i)^\top (\Sigma^s_i)^{-1}(X-\mu^s_i)\right) + \sum_{j=1}^{N_d} \sigma^d_j \,\exp\left(-\tfrac12 \left([X,t]-\mu^d_j\right)^\top (\Sigma^d_j)^{-1}([X,t]-\mu^d_j)\right),$$5
Source features	Frame-based spatial cues	Event-based temporal features
Assignment	K-Means/ClusterFormer clustering	K-Means/ClusterFormer clustering
Tracking	None	EKF over event correlations
Rendering fusion	Shadow/color mixing	Shadow/color mixing

This modularity enables flexible adaptation to arbitrary camera viewpoints and continuous time, supporting sharp, coherent renderings in complex dynamic environments.

6. Empirical Validation and Impact

Experimental results on synthetic and real-world datasets, including Event-HyperNeRF and CoFED, establish that Decoupled ST-GS:

• Achieves PSNR gains of 2–6 dB over unified/non-decoupled baselines.
• Outperforms prior multimodal and nonparametric baselines in both static background recovery and dynamic object rendering.
• Yields temporally continuous, ghosting-free, and accurately colored reconstructions across high-dynamic scenes—demonstrated via frame-wise and time-interpolated novel view renderings.

Loss-based ablations confirm that spatio-temporal disentanglement and event consistency regularization are the dominant contributors to these improvements (Zhou et al., 29 Jun 2025).

7. Limitations and Future Directions

The efficacy of Decoupled ST-GS is influenced by several factors:

• Feature extraction and clustering quality: spurious or ambiguous assignments can occur when event data is noisy or scene elements move subtly.
• Event-based tracking is limited by event density, rate, and sensor coverage, potentially missing low-movement dynamics.
• Blending and fusion at the rendering stage may require further physical modeling for highly complex illumination conditions.

Open directions include the integration of more sophisticated appearance-flow coupling, neural rendering hybrids, and improved physical priors for both spatial and temporal regularization. A plausible implication is that such frameworks could provide unified representations for arbitrarily complex dynamic environments, but achieving robustness across diverse real-world conditions may necessitate further methodological advances (Zhou et al., 29 Jun 2025).