Coarse-to-Fine Hierarchical Gaussian Splatting

• The paper introduces a hierarchical Gaussian splatting framework that organizes anisotropic 3D Gaussian primitives into a balanced binary tree for adaptive, efficient LOD rendering.
• It employs spatial chunking, moment-matched merging, and smooth attribute blending to handle extreme-scale scenes while preserving high visual fidelity.
• Real-time performance is attained through coarse global initialization, depth regularization, and GPU-optimized parallel processing, enabling applications in VR/AR and urban mapping.

Coarse-to-Fine Hierarchical Gaussian Splatting is a representation and rendering framework designed to enable real-time, scalable, and memory-efficient synthesis of very large 3D scenes. The approach organizes Gaussian primitives into a hierarchy, supporting adaptive level-of-detail (LOD) selection and resource-aware rendering. This methodology addresses the bottlenecks of flat, monolithic 3D Gaussian Splatting in extreme-scale or sparsely observed scenes by introducing spatial chunking, hierarchical merging, and differentiable attribute blending mechanisms (Kerbl et al., 2024).

1. Mathematical Foundations of Hierarchical Gaussian Splatting

The core primitive of hierarchical Gaussian Splatting is an anisotropic 3D Gaussian, parameterized by a mean $\boldsymbol{\mu} \in \mathbb{R}^3$, covariance matrix $\Sigma \in \mathbb{R}^{3 \times 3}$, base opacity $o \in [0,1]$, and color attributes $c$ represented as spherical harmonics coefficients (alternatively, by a small MLP for view-dependent color). The density in world space is given as:

$$f(\boldsymbol{w}) = w \cdot \exp\left(-\tfrac{1}{2}(\boldsymbol{w}-\boldsymbol{\mu})^\top \Sigma^{-1} (\boldsymbol{w}-\boldsymbol{\mu})\right)$$

where $w$ is a normalization factor proportional to $o \cdot \sqrt{(2\pi)^3 |\Sigma|}$.

Rendering involves projecting the 3D Gaussian into 2D image space, yielding a mean $\boldsymbol{\mu}'$ and a projected covariance $\Sigma'$. The 2D Gaussian contribution to a pixel $\boldsymbol{x} \in \mathbb{R}^2$ is:

$\Sigma \in \mathbb{R}^{3 \times 3}$0

Each projected Gaussian contributes to color and alpha channels, and final compositing is performed via standard front-to-back $\Sigma \in \mathbb{R}^{3 \times 3}$1-blending:

$\Sigma \in \mathbb{R}^{3 \times 3}$2

with $\Sigma \in \mathbb{R}^{3 \times 3}$3 the per-pixel opacity for the $\Sigma \in \mathbb{R}^{3 \times 3}$4-th Gaussian.

The essential innovation is to arrange Gaussians in a balanced binary hierarchy over scene regions ("chunks") and compute parent Gaussian parameters by a statistically consistent merging process, which minimizes KL divergence and matches moments: $\Sigma \in \mathbb{R}^{3 \times 3}$5 Weights $\Sigma \in \mathbb{R}^{3 \times 3}$6 are normalized by projected elliptic area and opacity.

2. Divide-and-Conquer Hierarchy Construction

To handle vast scenes, the approach subdivides the scene into overlapping spatial "chunks" (e.g., 50–100 m blocks), each trained independently. After local chunk training, a top-down binary bounding volume hierarchy (BVH) is built for each chunk:

• The AABB of Gaussians is recursively split along the longest axis at the median.
• Aggregated parent nodes represent clusters of their child Gaussians via moment-matched parameter merging.
• The hierarchy is constructed to a specified granularity or leaf budget, supporting efficient culling and LOD traversal.

This structure permits multilevel rendering granularity, minimizing memory and computational overhead by only rendering the subset of Gaussians required for current view parameters or pixel-size targets.

3. Level-of-Detail (LOD) Selection and Smooth Attribute Blending

LOD selection is parameterized by a granularity threshold $\Sigma \in \mathbb{R}^{3 \times 3}$7 (targeted screen-projected Gaussian diameter, in pixels). For each rendered view:

• Hierarchy nodes are scanned to select the LOD "cut": the smallest set where each selected node $\Sigma \in \mathbb{R}^{3 \times 3}$8 has screen granularity $\Sigma \in \mathbb{R}^{3 \times 3}$9.
• The render collects and composites only these nodes, yielding a view-adaptive Gaussian subset.

Transitions between LODs are made continuous via linear interpolation between parent and child attributes,

$o \in [0,1]$0

Ensuring seamless transition without "popping" artifacts, covariance interpolation is carefully handled via scale + rotation decomposition, and alpha blending between levels maintains consistent accumulated transmittance.

4. Training Adaptations for Large and Sparse Captures

For large or sparsely covered scenes, several novel regularization and adaptation steps are introduced:

• Coarse global initialization: All images are covered with a sparse set of Gaussians plus a sky-sphere, optimizing only for opacity and color to serve as a non-redundant global background that remains frozen during chunk-wise optimization.
• Modified densification in chunks: Densification chooses the Gaussians to split by maximum screen-space gradient, avoiding over-growth in poorly constrained/sparse areas.
• Depth regularization: Monocular depth maps (from models like DPT), rescaled by SfM points, provide an auxiliary loss term:

$o \in [0,1]$1

where $o \in [0,1]$2 is a rendered "depth channel."

• Exposure compensation: A per-image affine transformation is learned to align rendered and GT color across varying exposure conditions, jointly optimized with Gaussian attributes.

Post-hierarchy, interior nodes can be further fine-tuned at variable $o \in [0,1]$3 levels, with gradients backpropagated through the interpolation and merge formulas; only non-leaf (internal) nodes are updated to preserve leaf-level accuracy.

5. Efficiency and Quality Trade-Offs

The hierarchical approach allows fine control over trade-offs:

• Deeper hierarchies enable more granular LOD control but increase storage (~2x per Gaussian for parent/meta data).
• Rendering quality is dictated by the number of Gaussians rendered: a small $o \in [0,1]$4 (e.g., 3 px) renders ~75% of all leaves (high visual quality, 30–60 FPS on RTX A6000), while a large $o \in [0,1]$5 (e.g., 15 px) renders only ~10% of leaves (lower quality, up to 150 FPS).
• Dynamic response: The $o \in [0,1]$6 threshold can be continuously adapted to meet a frame-time budget; cut/expand operations are multi-threaded to prefetch Gaussians onto the GPU for real-time requirements.
• The compositing cost scales nearly linearly with the number of rendered Gaussians.

6. Practical Implementation and Differentiability

All hierarchy construction and rendering routines (moment-matched merging, cut selection, blending/interpolation weights, per-chunk and hierarchical losses) are fully differentiable and suitable for parallel implementation on a mix of CPU (for hierarchy traversal/merging) and GPU (rendering, dense batch computation).

Key aspects include:

• Statically freezing leaf node parameters post-chunk training and updating only the hierarchy during hierarchical LOD optimization.
• Supporting chunked streaming or multi-machine processing via oriented chunk divisions.
• Rendering and optimization remain compatible with standard PyTorch/Autodiff frameworks.

7. Impact, Applications, and Scope

Coarse-to-fine Hierarchical Gaussian Splatting substantially extends the scalability of explicit 3DGS models from table-top or room-scale scenes to kilometer-scale urban environments, enabling real-time synthesis with robust support for variable scene density and coverage (Kerbl et al., 2024). By directly adapting the number of rendered Gaussians and providing seamless LOD transitions, the method supports applications in VR/AR, urban mapping, and efficient visualization of extremely large capture datasets.

The hierarchical strategy outlined is directly compatible with semantic label lifting, structure-aware segmentation, and multi-metric optimization, and has already influenced subsequent advances in scalable 3DGS pipelines (Zhang et al., 13 Apr 2026, Xiao et al., 23 May 2025).

---

References:

• "A Hierarchical 3D Gaussian Representation for Real-Time Rendering of Very Large Datasets" (Kerbl et al., 2024)
• "GS4City: Hierarchical Semantic Gaussian Splatting via City-Model Priors" (Zhang et al., 13 Apr 2026)
• "SplatCo: Structure-View Collaborative Gaussian Splatting for Detail-Preserving Rendering of Large-Scale Unbounded Scenes" (Xiao et al., 23 May 2025)