Feature-Variance-Guided Hierarchical Densification

• The paper introduces FHD, a novel strategy that modulates gradient-based densification with local feature variance to target high-complexity regions while curbing redundant growth.
• FHD assigns hierarchical levels using quantile thresholds on feature variance, scheduling densification to stabilize coarse geometry before refining details.
• Empirical validations show that FHD improves PSNR, SSIM, and memory efficiency in 3D and 4D Gaussian Splatting, leading to sharper and more stable scene reconstructions.

Feature-variance-guided Hierarchical Densification (FHD) is a data-driven, multi-level densification strategy initially developed for high-fidelity 3D and 4D Gaussian Splatting (3DGS, 4DGS) representations. FHD systematically allocates new scene primitives by modulating standard gradient-based densification heuristics with local feature variance statistics, targeting regions of high spatial or chromatic complexity while suppressing overgrowth in smooth areas. This approach enables precise, scalable, and memory-efficient scene reconstruction, particularly in dynamic or long-range settings (Kwak et al., 10 Dec 2025, Su et al., 20 Apr 2025).

1. Motivation, Problem Setting, and Conceptual Foundation

In 3DGS and its temporal extensions, accurate scene modeling depends on controlled proliferation of Gaussian anchors. Naïve densification—proliferating new primitives wherever the magnitude of the aggregate training gradient is large—tends to overpopulate early-stage reconstructions with redundant anchors in regions of spurious gradient fluctuations, especially around high-frequency details or underconstrained textures. This excessive anchor growth not only increases memory usage but can degrade temporal consistency and cause rendering instability in dynamic scenes (Kwak et al., 10 Dec 2025).

FHD addresses this limitation by augmenting or replacing aggregate gradient magnitude tests with localized variance statistics derived from feature activations or per-pixel gradient signals. By partitioning anchors or Gaussians into frequency-based levels (low, mid, high), and regulating densification eligibility over the course of training, the method ensures that coarse geometric structure is robustly stabilized before introducing additional degrees of freedom to model fine details. This hierarchy yields more efficient anchor utilization, sharper reconstruction of textured or motion-rich content, and substantial reductions in both training and rendering memory footprints (Kwak et al., 10 Dec 2025, Su et al., 20 Apr 2025).

2. Mathematical Formalism and Level Assignment

The feature-variance metric in FHD quantifies local spatial or chromatic complexity at each anchor. In the MoRel framework (Kwak et al., 10 Dec 2025), the gradient variance $\sigma_k^2$ for a global anchor-point $k$ is accumulated over the Global Canonical Anchor (GCA) stage:

$\sigma_k^2 = \operatorname{Var}(\hat f_k)$

where $\hat f_k$ denotes the learned feature vector of anchor $k$. Two quantile thresholds $\tau_1,\tau_2$ (typically at the 33% and 66% quantiles) are computed over all $\sigma_k^2$ values, and each anchor is assigned a frequency level $L_k$:

• $0$ (low-frequency), if $\sigma_k^2 < \tau_1$
• $1$ (mid-frequency), if $\tau_1 \leq \sigma_k^2 < \tau_2$
• $2$ (high-frequency), if $\sigma_k^2 \geq \tau_2$

This quantile-based thresholding can be generalized to arbitrary level granularity. In the Metamon-GS pipeline (Su et al., 20 Apr 2025), the variance is measured directly from per-pixel RGB gradient vectors for each Gaussian, using a running estimator (Welford’s algorithm). The composite densification signal combines this variance $D_k$ with the mean positional gradient norm $\bar g_k$:

$\Gamma_k = \gamma D_k + \bar g_k$

A Gaussian is densified (split) when $\Gamma_k$ exceeds a predefined threshold $\tau_{\mathrm{th}}$.

3. Densification Scheduling and Algorithmic Pipeline

FHD introduces a hierarchical, level-dependent schedule to modulate when and where Gaussian anchors are allowed to densify:

1. Level Assignment: After initial training (GCA or equivalent), feature variances are computed per anchor or Gaussian, quantiles are evaluated, and levels are assigned.
2. Gradient Tracking: During each Key-frame Anchor (KfA) or Piece-Wise Deformation (PWD) training stage (in 4DGS), running sums of primitive-wise gradient magnitudes $g_{k,i}^{(j)}$ are maintained.
3. Level-weighted Criterion: At each densification checkpoint (every $M$ iterations), a weighted statistic $g_{k,i}^\ell(j) = w_{L_k}(j) \cdot g_{k,i}^{(j)}$ is computed, with the schedule

$$w_L(j) = \begin{cases} 1 & \text{if } L = 0 \ \lambda_L + (1-\lambda_L) \frac{j}{J} & \text{if } L \geq 1 \end{cases}$$

where $\lambda_L \in (0,1)$, $j$ is the current iteration, $J$ is the total stage iterations. This causes low-frequency anchors to be densified early, with mid and high-frequency anchors unlocked later.

1. Densification: If $g_{k,i}^\ell(j)$ (or $\Gamma_k$) exceeds the density threshold, a new anchor or child Gaussian is spawned in the neighborhood.
2. Pruning: Optionally, anchors with low opacity or insufficient support are removed to control memory usage.

The schedule refines coarse structure before allocating capacity to high-frequency regions, preventing noisy oversampling and improving final quality.

4. Empirical Validation and Quantitative Analysis

Ablation studies in MoRel (Kwak et al., 10 Dec 2025) and Metamon-GS (Su et al., 20 Apr 2025) evidence the impact of FHD:

• MoRel (Kwak et al., 10 Dec 2025):
  • Adding FHD to ARBB lowers rendering memory from 144 MB to 126 MB (−12.5%), and training memory from ≈6500 MB to 6000 MB, while maintaining or improving PSNR and SSIM with negligible LPIPS change.
  • Increasing level granularity from 1 to 3 levels improves quality and further reduces per-anchor storage.
• Metamon-GS (Su et al., 20 Apr 2025):
  • On Mip-NeRF360, adding FHD (termed "VGD") increases SSIM from 0.870 to 0.876, PSNR from 29.34 to 29.52, and reduces LPIPS from 0.187 to 0.171.
  • Variance-guided densification successfully eliminates persistent high-variance, recovers crisp boundaries and textured details, and prevents needle-like artifacts observed with naïve densification.
• High-variance anchors are targeted for densification precisely where needed, leading to uniformly sharp results and stable convergence.

Method	PSNR	SSIM	LPIPS	Training Mem	Rendering Mem
ARBB w/o FHD (Kwak et al., 10 Dec 2025)	21.07	0.672	0.342	≈6500 MB	144 MB
ARBB + FHD	21.20	0.672	0.348	6000 MB	126 MB
Scaffold-GS (Su et al., 20 Apr 2025)	28.84	0.848	0.220	—	—
+ LHE	29.34	0.870	0.187	—	—
+ LHE + VGD	29.52	0.876	0.171	—	—

5. Integration with Other Representation Components

In advanced pipelines such as Metamon-GS, FHD operates in synergy with high-capacity feature embedding and lighting models:

• Multi-level Hash Grid Lighting Encoder: FHD's densification decisions are informed by a multi-resolution hash grid, which augments the static anchor embedding with learned, view-dependent illumination features. Spawned Gaussians benefit from accurate, view-conditioned color estimation, and the hash-encoding keeps computational and memory costs sublinear in the number of Gaussians (Su et al., 20 Apr 2025).
• Densification-Rendering Feedback Loop: Variance-based scores ($D_k$) are decoupled from the MLP input, ensuring that densification targets regions of feature uncertainty without biasing the color model itself. The hash grid and hierarchical densification collectively maintain sharpness and color fidelity through all stages of training.

6. Limitations and Potential Extensions

FHD assumes that variance statistics computed immediately after initial training (e.g., GCA) are robust proxies for local complexity throughout subsequent optimization. In scenes with evolving frequency content or nonstationary textures, static quantile thresholds may lead to suboptimal allocation. Hyperparameters controlling quantile boundaries and level-weight schedules require dataset-specific tuning. Adaptive schemes that update level assignments dynamically or employ alternative local frequency metrics (e.g., spectral energy) are plausible extensions. Integration of FHD with block-structured or spatially multi-grid approaches (as in Block-NeRF) is a potential avenue for further scalability, especially for very large-scale scenes (Kwak et al., 10 Dec 2025).

7. Context and Impact within Scene Representation Research

FHD exemplifies a class of data-aware, hierarchy-driven densification strategies that directly exploit local statistical structure—feature variance rather than just mean gradient magnitude—to govern primitive proliferation in implicit or semi-implicit scene representations. The method has demonstrated substantial improvements in both computational efficiency and reconstruction fidelity across a broad set of synthetic and real-world benchmarks, including challenging long-range, high-motion, and high-frequency scenarios. Its lightweight algorithmic profile enables deployment in dynamic, memory-constrained applications without the need for monolithic grid structures or offline phase partitioning. The architectural compatibility with advanced feature embedding and lighting systems further enhances its utility in state-of-the-art differentiable rendering pipelines (Kwak et al., 10 Dec 2025, Su et al., 20 Apr 2025).