Deformable 3D Gaussian Splatting

• Deformable 3D Gaussian Splatting is a technique employing anisotropic 3D Gaussian primitives to efficiently render dynamic scenes and facilitate novel view synthesis.
• It combines cage-based deformation methods and neural deformation fields to maintain high-frequency texture details during both global and local adjustments.
• Recent advances achieve state-of-the-art performance, with over 35 dB PSNR and real-time rendering speeds exceeding 300 FPS by integrating structured embeddings and adaptive temporal models.

Deformable 3D Gaussian Splatting is a volumetric scene representation and deformation technique that synthesizes novel views of complex, potentially dynamic or edited 3D geometry through explicit, anisotropic 3D Gaussian primitives. While static 3D Gaussian Splatting provides highly efficient, real-time novel-view rendering for fixed geometry, extension to deformable settings requires precise alignment of geometry and preservation of high-frequency appearance during global or local deformations. Recent developments—including cage-based control, neural deformation fields, rigid-body rigging, and embedding-based temporal models—enable the seamless adaptation of existing 3DGS models to arbitrary target shapes or time-varying scenarios with little loss of fidelity.

1. Mathematical Foundations of Deformable 3D Gaussian Splatting

A deformable 3DGS scene is represented as a collection of N explicit, parameterized 3D Gaussians: $g_i = (\mu_i, \Sigma_i, c_i, \alpha_i)$ where $\mu_i \in \mathbb{R}^3$ is the center, $\Sigma_i \in \mathbb{R}^{3 \times 3}$ the symmetric positive-definite covariance (typically factored as $\Sigma_i = R_i S_i S_i^\top R_i^\top$ for rotation $R_i$ and axis-aligned scale $S_i$), $c_i$ the (possibly view-dependent) color, and $\alpha_i$ the opacity or density. Rendering projects each Gaussian to the screen, computes per-pixel weights via the analytic projection of ellipsoids, sorts by depth, and composites using alpha-blending:

$$C(p) = \sum_{i=1}^N c_i \, \alpha_i \prod_{j<i} (1-\alpha_j)$$

Deformation is modeled as a mapping from the canonical Gaussians (typically at $t=0$ or a reference shape) to a deformed configuration through one of several parameterizations:

• Cage-based deformation: associates each $\mu_i \in \mathbb{R}^3$0 to a surrounding cage by mean-value/harmonic coordinates $\mu_i \in \mathbb{R}^3$1; as the cage deforms, each $\mu_i \in \mathbb{R}^3$2 is updated via

$\mu_i \in \mathbb{R}^3$3

where $\mu_i \in \mathbb{R}^3$4 are new cage vertex positions. Covariances transform via the local deformation Jacobian $\mu_i \in \mathbb{R}^3$5:

$\mu_i \in \mathbb{R}^3$6

(Tong et al., 17 Apr 2025, Xie et al., 2024)

• Neural deformation fields: learn per-Gaussian deformations as functions of (potentially fused) geometric and temporal embeddings, e.g.,

$\mu_i \in \mathbb{R}^3$7

with $\mu_i \in \mathbb{R}^3$8 a per-Gaussian embedding, $\mu_i \in \mathbb{R}^3$9 a temporal embedding (e.g., positional encoding or grid), and $\Sigma_i \in \mathbb{R}^{3 \times 3}$0 an MLP (Bae et al., 2024, Jiao et al., 21 Mar 2026, Jiao et al., 27 Aug 2025).

• Flexible basis curves: model temporal evolution of each Gaussian attribute (mean, covariance, opacity) by a sum of radial basis functions in time (Yang et al., 2024, Shan et al., 2 Jan 2025).
• Rigid/skinned body rigging: combine canonical-to-pose mapping via skinning (e.g., SMPL/SMPL-X) with per-Gaussian refinement via neural modules for articulated avatars (Jung et al., 2023, Qian et al., 2023).

The color and opacity are typically kept fixed under deformation to preserve high-frequency texture unless explicitly handled for dynamic appearance or view/time dependence.

2. Cage-Based and Hybrid Deformation Control

Cage-based deformation schemes place the 3DGS point cloud inside a "cage" mesh whose deformation induces a volumetric mapping propagated deterministically, thereby enabling:

• Intuitive global and semiglobal control (structural shape editing)
• Preservation of high-frequency textural detail via Jacobian-corrected covariance propagation, avoiding splat stretching or blurring (Tong et al., 17 Apr 2025, Xie et al., 2024)
• Flexible alignment to diverse target modalities: target shapes may be specified as text prompts (via proxy 3DGS), single images (AtlasNet, mesh proxy), point clouds, meshes, or 3DGS models, with shape alignment driven by PointNet-AtlasNet pipelines optimized for physical plausibility, coordinate positivity, and normal consistency.

Integration with neural Jacobian fields provides precise, per-face control of local differential mappings, increasing smoothness and regularity, particularly in user-guided or sketch-based editing scenarios (Xie et al., 2024). However, cage-induced deformations cannot guarantee the preservation of strict invariants such as straight lines or sharp features for CAD-like geometries, and local manipulations may require alternative parameterizations (e.g., direct handle dragging).

3. Neural Deformation Fields and Per-Gaussian Embedding

Most dynamic or temporal 3DGS representations employ neural deformation fields, where each Gaussian's position and shape is predicted by MLPs taking as input per-Gaussian and temporal embeddings. Prominent designs include:

• Joint per-Gaussian and per-frame embeddings for accurate local deformation, decomposed into slow/fast (coarse/fine) pathways for modeling both gradual and abrupt motion (Bae et al., 2024, Jiao et al., 21 Mar 2026, Jiao et al., 27 Aug 2025).
• Geometry-aware deformation, in which features derived from local 3D neighborhoods (via sparse 3D U-Nets or PointNet) enrich prediction and enforce spatial consistency (Lu et al., 2024).
• Partitioning schemes, such as MAPo, that decouple highly dynamic Gaussians from quasi-static components via motion scoring, recursively specializing deformation networks for different spatiotemporal regimes (Jiao et al., 27 Aug 2025).
• Physics-based or rigid-body deformation, e.g., deformable linear objects (ropes/cables) modeled with PBD solvers coupled with GS-based observation filtering (Dinkel et al., 13 May 2025), or mesh-rigged splats for intraoperative navigation with joint photometric and geometric supervision (Fehrentz et al., 23 Sep 2025).

Ablations consistently show that neglecting per-Gaussian adaptation (e.g., using global or purely coordinate-based deformation) produces contamination across static/dynamic regions and loss of geometric detail (Jiao et al., 27 Aug 2025, Bae et al., 2024).

4. Advanced Deformation Modeling: Temporal, View, and Hierarchical

Emerging techniques further refine deformable 3DGS models by addressing dynamic scene complexities:

• Temporal basis or RBF expansions for per-Gaussian attribute curves, supporting both reversible (elastic) and irreversible (shearing, topological change) deformations (Shan et al., 2 Jan 2025, Nam et al., 25 May 2025).
• Adaptive hierarchical motion masks, distinguishing static from deformable regions by evaluating deformation magnitude or rendering-loss differences, restricting neural inference and deformation updates to relevant Gaussians and maximizing rendering speed (Shan et al., 2 Jan 2025).
• View-dependent deformable fields: augment deformation conditioning with view/camera direction, enabling modeling of subtle appearance and emission changes, notably in domains with challenging radiometric phenomena (e.g., thermal-infrared) (Nam et al., 25 May 2025).
• Group trajectory analysis and rigid-group factorization (e.g., GroupFlow in SpeeDe3DGS), which clusters Gaussians by similar motion, replacing redundant per-Gaussian neural inference with compact per-group rigid transforms and dramatically accelerating rendering and reducing memory (Tu et al., 9 Jun 2025).

Quantitative evaluations indicate that these strategies yield state-of-the-art quality (e.g., PSNR $\Sigma_i \in \mathbb{R}^{3 \times 3}$1 35 dB, real-time rendering $\Sigma_i \in \mathbb{R}^{3 \times 3}$2 300 FPS) across diverse real and synthetic benchmarks while addressing scalability (Shan et al., 2 Jan 2025, Tu et al., 9 Jun 2025).

5. Applications: Editing, Medical Imaging, Tracking, and Animation

Deformable 3D Gaussian Splatting underpins several lines of application:

• Interactive 3D geometric editing and animation: cage-based, sketch-driven workflows enable scene-level deformation or animation through explicit or diffusion-prior guided systems, supporting modalities ranging from text and images to arbitrary sketch keyframes (Tong et al., 17 Apr 2025, Xie et al., 2024).
• Dynamic scene synthesis: per-Gaussian embedding fields, temporal decomposition, and deformation hierarchy enable photorealistic, real-time reconstruction and free-viewpoint rendering in scenes with large, non-rigid motion (e.g., articulated avatars, human-body performance capture, dynamic objects) (Jung et al., 2023, Qian et al., 2023, Bae et al., 2024, Jiao et al., 27 Aug 2025).
• Real-time 3D medical reconstruction and navigation: explicit and flexible deformation modeling of tissue and organs, combined with rapid point-cloud initialization and segregation of static/deformable components, addresses intraoperative speed and fidelity requirements in endoscopic and surgical applications (Chen et al., 2024, Zhu et al., 2024, Yang et al., 2024, Shan et al., 2 Jan 2025, Fehrentz et al., 23 Sep 2025).
• Physics-guided and object-specific tracking: hybrid pipelines combine physics-based dynamics (e.g., PBD) with differentiable GS rendering, enabling robust 3D tracking under occlusion in robotics or manipulation scenarios (Dinkel et al., 13 May 2025).
• Dynamic transient and occlusion handling: deformable transient fields and memory-efficient masking allow high-quality synthesis in settings with occlusions, dynamic distractors, or uncontrolled capture (Park et al., 8 Mar 2025).

6. Quantitative Evaluation, Limitations, and Future Directions

Benchmarks systematically report geometry and appearance fidelity (Chamfer distance, PSNR, SSIM, LPIPS), perceptual metrics (user study votes, DINO cosine), and practical speed/memory (FPS, training time, parameter count). For example, CAGE-GS achieves leading Chamfer and DINO scores on ShapeNet, and quantitative PSNR increases of 1–2 dB have been attributed to geometry-aware or embedding-based deformation compared to earlier coordinate-only or global models (Tong et al., 17 Apr 2025, Lu et al., 2024, Jiao et al., 27 Aug 2025, Jiao et al., 21 Mar 2026).

Key ablation findings:

• Omission of Jacobian-based covariance update produces severe blurring or texture drift (Tong et al., 17 Apr 2025).
• Single deformation network for the entire sequence induces over-averaging and blurry renderings in fast dynamic regions unless partitioned (MAPo, GroupFlow) (Jiao et al., 27 Aug 2025, Tu et al., 9 Jun 2025).
• Explicit per-Gaussian embedding is essential to prevent leakage of fast motion into neighboring static geometry (Bae et al., 2024).

Limitations include:

• Incomplete preservation of strict geometric invariants (lines, planes) under global cage-based methods (Tong et al., 17 Apr 2025).
• Trade-off between per-Gaussian neural inference overhead and real-time constraint; group/rank pruning and trajectory factorization mitigate this (Tu et al., 9 Jun 2025).
• Residual temporal blur or artifacts in extreme motion regions or highly stochastic texture domains (Bae et al., 2024, Nam et al., 25 May 2025).

Future directions cited include:

• End-to-end learning of Jacobian fields and deformation priors.
• Interactive/VR deformation and editing interfaces.
• Deformation-aware neural surface coupling and mutual supervision for guaranteed geometry correspondence (Li et al., 2024).
• Spatio-temporal deformation and frequency modeling in domains such as thermal or multispectral data (Nam et al., 25 May 2025).
• Extension to bounded-support kernels and new functional bases (e.g., deformable Beta splatting for adaptive kernel shape and improved densification) (Liu et al., 27 Jan 2025).

---

References:

• (Tong et al., 17 Apr 2025) "CAGE-GS: High-fidelity Cage Based 3D Gaussian Splatting Deformation"
• (Xie et al., 2024) "Sketch-guided Cage-based 3D Gaussian Splatting Deformation"
• (Lu et al., 2024) "3D Geometry-aware Deformable Gaussian Splatting for Dynamic View Synthesis"
• (Jiao et al., 21 Mar 2026) "Fast and Robust Deformable 3D Gaussian Splatting"
• (Tu et al., 9 Jun 2025) "Speedy Deformable 3D Gaussian Splatting: Fast Rendering and Compression of Dynamic Scenes"
• (Jiao et al., 27 Aug 2025) "MAPo : Motion-Aware Partitioning of Deformable 3D Gaussian Splatting for High-Fidelity Dynamic Scene Reconstruction"
• (Shan et al., 2 Jan 2025) "Deformable Gaussian Splatting for Efficient and High-Fidelity Reconstruction of Surgical Scenes"
• (Yang et al., 2024) "Deform3DGS: Flexible Deformation for Fast Surgical Scene Reconstruction with Gaussian Splatting"
• (Bae et al., 2024) "Per-Gaussian Embedding-Based Deformation for Deformable 3D Gaussian Splatting"
• (Dinkel et al., 13 May 2025) "DLO-Splatting: Tracking Deformable Linear Objects Using 3D Gaussian Splatting"
• (Jung et al., 2023) "Deformable 3D Gaussian Splatting for Animatable Human Avatars"
• (Qian et al., 2023) "3DGS-Avatar: Animatable Avatars via Deformable 3D Gaussian Splatting"
• (Li et al., 2024) "DGNS: Deformable Gaussian Splatting and Dynamic Neural Surface for Monocular Dynamic 3D Reconstruction"
• (Nam et al., 25 May 2025) "Veta-GS: View-dependent deformable 3D Gaussian Splatting for thermal infrared Novel-view Synthesis"
• (Liu et al., 27 Jan 2025) "Deformable Beta Splatting"
• (Chen et al., 2024, Zhu et al., 2024, Fehrentz et al., 23 Sep 2025, Park et al., 8 Mar 2025) (medical/tracking/editing/transient)