GS-DMSR: Dynamic 3D Gaussian Splatting

• GS-DMSR is a dynamic scene reconstruction framework that uses adaptive per-Gaussian gradient focusing to optimize rapid convergence and high fidelity rendering.
• It integrates a multi-scale manifold enhancement combining an explicit coarse deformation field with an implicit MLP-based decoder for fine-scale corrections.
• Empirical results demonstrate improved PSNR, lower storage overhead, and real-time rendering performance on both synthetic and real dynamic datasets.

The GS-DMSR (Dynamic Sensitive Multi-scale Manifold Enhancement for Accelerated High-Quality 3D Gaussian Splatting) method is a framework for dynamic scene reconstruction that addresses the challenge of balancing rapid model convergence with high-fidelity rendering, particularly in scenes exhibiting complex, non-rigid motions. GS-DMSR introduces adaptive per-Gaussian training focus and a multi-scale deformation manifold, yielding fast convergence, low storage overhead, and efficient rendering at real-time rates, as empirically validated on both synthetic and real dynamic datasets (Lu et al., 9 Jan 2026).

1. Pipeline Structure and Key Components

GS-DMSR is structured around two principal innovations:

• Dynamic-Sensitive Gradient Focusing (MS-DGO): An adaptive mechanism that quantifies and classifies the "motion saliency" of each Gaussian according to the temporal evolution of its parameters, focusing updates and computational resources only on those Gaussians undergoing significant change.
• Multi-Scale Manifold Enhancement: A composite deformation pipeline leveraging both an explicit low-rank deformation field for coarse motion and an implicit MLP-based nonlinear decoder for detailed corrections.

The iterative pipeline is outlined as follows:

1. Initialization: The model commences with a static 3D Gaussian cloud $G = \{G_i\}$, each $G_i$ parameterized by position $x_i \in \mathbb{R}^3$, scale $s_i \in \mathbb{R}^3$, orientation quaternion $r_i \in \mathbb{R}^4$, opacity $\alpha_i \in \mathbb{R}$, and color coefficients $c_i \in \mathbb{R}^k$.
2. Per-Iteration Update:
   • Extract spatiotemporal features $f_i^{c}(t)$, $f_i^{f}(t)$ for each Gaussian.
   • Apply the MS-DGO module: compute motion-saliency score $S_i(t)$ and classify each $G_i$.
   • Update Gaussian parameters using explicit coarse deformation and implicit fine-scale corrections.
   • Render the current frame via differentiable Gaussian splatting and compute loss for optimization.
   • Only high-saliency Gaussians receive full deformation updates; low-saliency Gaussians receive reduced learning rates or are frozen.
3. Convergence: Training proceeds until convergence, with multi-scale collaboration and parameter sharing between decoder heads.

2. Adaptive Gradient Focusing: MS-DGO

The MS-DGO module adaptively allocates computational effort by identifying which Gaussians continue to participate dynamically in scene evolution.

• State Variables: Each Gaussian $G_i$ has a state vector

$$a_i(t) = [x_i(t); r_i(t); s_i(t); c_i(t)] \in \mathbb{R}^D$$

where $D = 3 + 4 + 3 + k$.

• Motion-Saliency Score:

$$S_i(t) = \|W \Delta a_i(t)\|_1 = \sum_{d=1}^D w_d\,|a_i^{(d)}(t) - a_i^{(d)}(t-1)|$$

with $W = \mathrm{diag}(w_1, ..., w_D)$ normalizing the contributions from different components.

• Saliency Categorization: A threshold $\tau$ is imposed. If $S_i(t) > \tau$, $G_i$ is high-saliency; otherwise, low-saliency.
• Learning Rate Allocation: Gradient updates per $G_i$ proceed at rate

$$\eta_i(t) = \begin{cases} \eta_H & \text{if}\ S_i(t) > \tau\ \eta_L \ll \eta_H & \text{otherwise} \end{cases}$$

In practice, $\eta_H \approx 10^{-3}$, $\eta_L \approx 10^{-5}$ or $\eta_L = 0$.

This mechanism selectively allocates optimization budget, improving convergence rate by suppressing wasteful updates on near-static or converged Gaussians (Lu et al., 9 Jan 2026).

3. Multi-Scale Manifold Enhancement Architecture

The deformation of each Gaussian is modeled hierarchically:

• Coarse-Scale (Explicit Field): A low-rank spatio-temporal basis estimates coarse offsets $\Delta x_i^{c}(t)$ via a feature extractor $E_c$ and linear decoder $D_c$:

$$f_i^{c}(t) = E_c(x_i, t) \ \Delta x_i^{c}(t) = D_c(f_i^c(t))$$

This branch models broad deformations but omits fine detail and color.

• Fine-Scale (Implicit MLP): An MLP $\phi$ (with shared trunk, multi-head for different geometric attributes) accepts fine-grained features $f_i^{f}(t)$ and predicts corrections for position, rotation, and scale:

$$(\Delta x_i^{f}, \Delta r_i^{f}, \Delta s_i^{f}) = (\phi_x, \phi_r, \phi_s)(f_i^f(t))$$

The merged Gaussian parameters are:

$$x_i'(t) = x_i(t) + \Delta x_i^c(t) + \Delta x_i^f(t) \ r_i'(t) = r_i(t) + \Delta r_i^f(t) \ s_i'(t) = s_i(t) + \Delta s_i^f(t)$$

• Multi-Scale Training Schedule:
  • For the first $N_1$ iterations (e.g., 2,000), only the explicit field is optimized ($\phi$ is frozen).
  • Subsequently, $\phi$ is unfrozen and both branches are updated in tandem.
• Loss Functions:
  • Reconstruction loss: $$L_{\text{rec}} = \sum_{t,p} \| \hat{C}_p(t) - C^*_p(t) \|_1$$
  • Deformation regularization: $$L_{\text{reg}} = \lambda_{\text{reg}} \sum_{i,t} \|\Delta x_i^c(t)\|^2$$
  • Decoder smoothness (optional): $$L_{\text{smooth}} = \lambda_{\text{smooth}} \sum_{i,t} \| \nabla_x \phi(f_i^f(t)) \|^2$$
  • Total loss: $$L = L_{\text{rec}} + L_{\text{reg}} + L_{\text{smooth}}$$

This multi-scale approach ensures both low-frequency motion and high-frequency geometric details are accurately modeled and integrated into the dynamic scene representation.

4. Optimization, Implementation, and Dataset Protocol

• Framework: The method is implemented in PyTorch and optimized for a single RTX3090 GPU.
• Training Details:
  • Mini-batch: 4 time frames × 1024 pixel rays.
  • Learning rates: $\eta_H=1\times10^{-3}$ for high-saliency, $\eta_L=1\times10^{-5}$ (or 0) for low-saliency.
  • Saliency threshold: $\tau=1\times10^{-3}$.
  • Training schedule comprises $8,000$–$10,000$ total iterations, with the coarse-to-fine transition at iteration $2,000$.
• Dataset Handling:
  • Synthetic (D-NeRF): 50–200 frames, random camera poses.
  • Real (HyperNeRF): Structure-from-Motion (SfM) initialization, point cloud to Gaussians.
  • Dynamic Object: 6 synthetic objects with controlled motion.
  • Uniform normalization of images, with intrinsics from SfM.
• Performance Metrics:
  • PSNR, SSIM, LPIPS on held-out frames; training time, runtime FPS, and storage (Gaussians × attributes) are monitored.
• Empirical Comparison:

Method	PSNR (dB)	Training Time	FPS (runtime)
4D-GS	34.05 @ 8 min	8 min	82
GS-DMSR	34.56 @ 8 min	8 min	96

(Lu et al., 9 Jan 2026)

5. Algorithmic Summary

The high-level training and inference loop is summarized as follows:

Algorithm GS-DMSR Input: Images I_p(t), camera poses, initial Gaussians {x_i, r_i, s_i, c_i, α_i} for t = 1 to T_total do Sample mini-batch {t_b}, pixel rays p for each Gaussian i: f_i^c ← CoarseGrid.query(x_i, t) f_i^f ← FineGrid.query(x_i, t) Δa_i ← a_i(t–1) – a_i(t–2) S_i ← ‖W Δa_i‖₁ saliency_flag[i] ← (S_i > τ) for each Gaussian i: Δx_i^c ← D_c(f_i^c) (Δx_i^f, Δr_i^f, Δs_i^f) ← φ(f_i^f) x_i' ← x_i + Δx_i^c + Δx_i^f r_i' ← r_i + Δr_i^f s_i' ← s_i + Δs_i^f Render frame with updated Gaussians Compute losses – L_rec, L_reg, optional L_smooth Backpropagate for each Gaussian i: if saliency_flag[i]: η_i ← η_H else: η_i ← η_L θ_i ← θ_i – η_i ∇_{θ_i} L Store updated a_i(t) end for Output: Optimized dynamic Gaussians and φ decoder

(Lu et al., 9 Jan 2026)

6. Comparative Context and Significance

GS-DMSR builds on 3D Gaussian Splatting frameworks by addressing a critical bottleneck in dynamic scene modeling: the trade-off between rapid convergence and accurate high-resolution rendering in the context of spatially and temporally complex deformations. The motion-saliency-based gradient allocation (MS-DGO) is conceptually related to broader ideas from derivative-driven importance sampling, while the multi-scale manifold strategy extends the paradigm of combining explicit and implicit deformation models for dynamic geometry.

Compared to contemporaneous mesh-oriented extensions such as DyGASR, which leverage adaptive generalized exponentials and explicit surface regularization (Zhao et al., 2024), GS-DMSR remains focused on Gaussian splatting for differentiable volume rendering and prioritizes dynamic motion modeling through adaptive, per-Gaussian optimization scheduling.

Quantitative gains include an increase in test PSNR, substantial reduction in training and inference time (by not wasting budget on static regions), and lower memory usage due to effective parameter freezing. The system attains real-time rendering at 96 FPS and full convergence within 8 minutes on high-resolution scenes, while maintaining top-tier visual reconstruction performance (Lu et al., 9 Jan 2026).

7. Future Directions and Applications

The GS-DMSR method is positioned for applications in real-time novel view synthesis, 3D video, AR/VR dynamic scene rendering, and dynamic object tracking in high-precision datasets. Its adaptive optimization principle suggests potential extensions to broader differentiable graphics tasks where combinatorial sparsity and temporal coherence are present. Integration with mesh extraction and surface-aware regularization, as pioneered in contemporary work on generalized exponential splatting and Poisson reconstruction (Zhao et al., 2024), is a promising future avenue.

A plausible implication is that methods adopting dynamic, saliency-aware optimization schedules and multi-scale deformable manifolds will remain central for scalable, high-fidelity dynamic scene reconstruction in forthcoming research.