Deformable Gaussian Mask in 3D Reconstruction

• Deformable Gaussian mask is a technique that segments 3D scenes using time-varying Gaussian primitives to distinctly model dynamic and static regions.
• It employs hybrid loss strategies and foreground-background disentanglement to enhance reconstruction quality for applications like surgical imaging and visual effects.
• The method leverages efficient GPU-based implementations to enable real-time rendering and robust handling of occlusions and sparsely observed regions.

A deformable Gaussian mask is a methodological component in 3D dynamic scene reconstruction frameworks that separates, trains, or supervises parts of a scene using time-varying or semantically-aware partitions on a set of spatially distributed Gaussian primitives. This mechanism has become foundational in recent 3D Gaussian Splatting (GS) methods for both surgical scene reconstruction and entertainment-oriented dynamic view synthesis, providing explicit, differentiable control over dynamic and static scene regions. By constructing masks in the spatial or spatio-temporal domain, these approaches enable hybrid loss strategies, foreground/background disentanglement, artifact suppression, and view-consistent reconstruction in the presence of occluders or sparsely observed regions.

1. Mathematical Foundation for 3D Gaussians and Splat-Based Rendering

A 3D Gaussian in these frameworks is parameterized by a mean $\mu_i \in \mathbb{R}^3$, a positive semi-definite covariance $\Sigma_i \in \mathbb{R}^{3\times 3}$, an opacity $\alpha_i \in [0,1]$, and an appearance or color vector $\mathbf{c}_i \in \mathbb{R}^F$, often with $F=3$ for RGB or higher for harmonics (Xie et al., 6 Jul 2024, Azzarelli et al., 7 Nov 2025). The covariance is typically factored as

$\Sigma_i = R_i S_i S_i^T R_i^T,$

where $R_i$ is a rotation matrix (represented via quaternion $q_i$) and $S_i=\mathrm{diag}(s_i)$ is a scale. Under camera parameters and an image Jacobian $(V,J)$, each 3D Gaussian projects into a 2D elliptical footprint on the image plane:

$\Sigma_i' = J V \Sigma_i V^T J^T.$

The rendered pixel color employs alpha compositing in either front-to-back or back-to-front order. For SurgicalGaussian (Xie et al., 6 Jul 2024), the composited pixel color and depth at position $\mathbf{r}$ are:

$$\hat{C}(\mathbf{r}) = \sum_{i=1}^{N} \left( \alpha_i' \prod_{j<i}(1-\alpha_j') \right) c_i, \quad \hat{D}(\mathbf{r}) = \sum_{i=1}^{N} \left( \alpha_i' \prod_{j<i}(1-\alpha_j') \right) d_i,$$

where $\alpha_i' = \alpha_i G'_i(\mathbf{r})$ is the effective opacity and $G'_i$ is the projected Gaussian density.

2. Deformable Gaussians: Temporal and Spatial Parameterization

Deformation fields are central to dynamic GS models. Each canonical Gaussian is anchored in a reference (“canonical”) space and deformed over time.

• In SurgicalGaussian, a forward mapping deformation MLP $F_\theta$ predicts offsets $$(\delta \mathbf{x},\delta \mathbf{s},\delta \mathbf{q})$$ conditioned on both the canonical center $\mathbf{x}_c$ and time $t\in[0,1]$:

$$(\delta \mathbf{x}, \delta \mathbf{s}, \delta \mathbf{q}) = F_\theta(\gamma(\mathbf{x}_c), \gamma(t)),$$

with final parameters

$$\mathbf{x}_o = \mathbf{x}_c + \delta \mathbf{x}, \quad \mathbf{s}_o = \mathbf{s}_c \odot \exp(\delta \mathbf{s}), \quad \mathbf{q}_o = \mathbf{q}_c \otimes \delta \mathbf{q}$$

per Equations 3–4 in (Xie et al., 6 Jul 2024).

• For Splatography, deformation per Gaussian $D_i(t)$ is produced by a hex-plane network with MLP decoding, yielding temporal offsets for position, rotation, and color:

$$G_i(t) = (\mu_i + \Delta\mu_i(t),\, \Sigma_i + \Delta\Sigma_i(t),\, c_i + \Delta c_i(t),\, \alpha_i(t))$$

(Azzarelli et al., 7 Nov 2025).

3. Construction and Utilization of Deformable Gaussian Masks

The mask-based partitioning or supervision mechanism is used to focus learning or rendering resources on specific subregions.

• Tool-Mask Guidance in Surgery: In SurgicalGaussian, binary masks $M_i$ ($1=$ instrument, $0=$ tissue) are used to ensure that reconstruction losses ignore instrument pixels, enforcing that only visible tissue is modeled. The mask multiplicatively occludes the loss:

$$L_{\rm color} = \sum_i \| (I_i-\hat C_i)\odot(1-M_i) \|_1,$$

$$L_{\rm depth} = \sum_i \| (D_i-\hat D_i)\odot(1-M_i) \|_1$$

(Xie et al., 6 Jul 2024). Initialization excludes instruments via $M^* = \bigcap_{i=0}^T M_i$, creating a seed point cloud only from consistently unoccluded tissue pixels.

• Foreground–Background Split in Filmmaking: Splatography introduces a deformable Gaussian mask for explicit splitting. Sparse binary masks $M^*_v$ at $t=0$ for each view are used to back-project 3D Gaussian centers $\mu_i$:

$m_i = \prod_{v=1}^V M^*_v(\Pi_v(\mu_i))$

yielding sets $\mathcal{G}_f$ (foreground: $m_i=1$) and $\mathcal{G}_b$ (background: $m_i=0$). Each set is optimized with custom losses and, in the dynamic stage, distinct deformation fields—full position/rotation/color changes for $\mathcal{G}_f$, position-only for $\mathcal{G}_b$ (Azzarelli et al., 7 Nov 2025).

4. Canonical and Dynamic Training Objectives with Masked Supervision

The mask enables distinct training objectives for specific subregions:

• Canonical Stage: For Splatography, $\mathcal{G}_f$ and $\mathcal{G}_b$ are pretrained separately. The foreground uses a blended-color loss with a random background to prevent overfitting to training backgrounds, while the background adopts an edge-aware loss utilizing blurred background pixels. The full loss is

$$\mathcal{L}_{\mathrm{canon}} = \mathcal{L}_{\mathrm{fg}} + \mathcal{L}_{\mathrm{bg}}$$

(Azzarelli et al., 7 Nov 2025).

• Dynamic Stage: Once canonical training converges, dynamic fine-tuning is performed, enabling the foreground and background Gaussians to deform differently in time. Regularization penalizes deviations from desired opacity peaks and bandwidths.
• Surgical Scene Regularization: To enforce physically plausible motion, SurgicalGaussian incorporates local deformation consistency losses (position and covariance) between each Gaussian and its neighbors, as well as color smoothness regularization for occluded regions—measured with total variation loss in intersection masks $M^*$ (Xie et al., 6 Jul 2024).

5. Implementation and Computational Considerations

Efficient GPU implementation is central for real-time rendering and scalable training:

System	Gaussian Storage	Deformation Network	Mask Use	Performance (RTX 3090)
SurgicalGaussian	Struct-of-arrays, on GPU	8x256 MLP per Gaussian	Byte-masks, precomputed	4 GB GPU, ~40k iters, 80–200 FPS
Splatography	Canonical + hex-plane	Two-layer MLP decoders	Sparse binary masks (t=0)	6–8 hr per scene training

Mask computation is performed prior to training (via segmentation algorithms or annotation). Splatting and compositing kernels, as well as deformation field evaluations, are GPU-resident for performance. Mask guidance incurs negligible computational overhead in both frameworks (Xie et al., 6 Jul 2024, Azzarelli et al., 7 Nov 2025).

6. Applications and Quantitative Impact

• Surgical Scene Reconstruction: Deformable Gaussian masks (with tool-masks) enable the removal of surgical instruments from both initialization and optimization, yielding higher-fidelity reconstructions of soft tissue despite occlusions and rapid nonrigid deformation. Rendering quality, speed, and GPU efficiency are improved over NeRF-based baselines, with training requiring only 4 GB and inference at up to 200 FPS (Xie et al., 6 Jul 2024). The regularized masks also ensure color smoothness in never-seen tissue regions.
• Filmmaking and Dynamic Effects: Foreground-background decomposition via deformable masks enables Splatography to increase PSNR by up to 3 dB on 3D scenes, halve memory footprint, and produce explicit segmentations of transparent or dynamic actors. Unlike approaches with a single deformation field, the mask-split approach prevents background overfitting and foreground under-segmentation, preserving high-frequency motion and allowing direct downstream use of segmentations (Azzarelli et al., 7 Nov 2025).

A plausible implication is that mask-based partitions empower scene decomposition strategies suited for both artifact reduction and content-aware rendering in occlusion-heavy, sparsely observed, or semantically structured dynamic scenes.

7. Relation to Prior Art and Future Directions

Earlier dynamic GS approaches relied on a singleton canonical scene and a single deformation field, leading to capacity issues under sparse cameras or complex interleaved dynamics, with no explicit mechanism for foreground normalcy or extraction (Azzarelli et al., 7 Nov 2025). Deformable Gaussian mask techniques facilitate more targeted optimization and flexible scene representations. Novel applications continue to emerge in intra-operative reconstruction, transparent/smoke effect modeling, and memory-constrained view synthesis.

Future directions likely include adaptive mask construction, dynamic mask updates, multi-modal mask guidance, and integration with scene semantics for instance-level or part-based dynamic relabeling. This suggests further capability to address challenges in general unconstrained dynamic scene capture and efficient real-time rendering.