Papers
Topics
Authors
Recent
Search
2000 character limit reached

3DGS Model: Overview, Applications, and Challenges

Updated 9 November 2026
  • 3DGS is a scene-representation framework that uses anisotropic three-dimensional Gaussian primitives to model a scene with spatial centers, covariance, and view-dependent appearance.
  • 3DGS is effective for high-quality rendering and fast rasterization in applications like novel-view synthesis and geometric reconstruction, and have challenges in memory consumption and geometric ambiguity.
  • Research focus on optimizing 3DGS relies on methods such as T-3DGS for transient object removal,making the technology applicable for static and dynamic scenes.

3D Gaussian Splatting (3DGS) is an explicit scene-representation and rendering framework in which a scene is modeled as a collection of anisotropic three-dimensional Gaussian primitives. Each primitive has a spatial center, covariance or equivalent scale-and-rotation parameters, opacity, and view-dependent appearance, commonly represented with spherical-harmonic (SH) coefficients. The primitives are projected into the image plane, depth-ordered, and alpha-composited by a tile-based differentiable rasterizer. 3DGS is used for novel-view synthesis, geometric reconstruction, compression, object detection, transient-object removal, hardware-accelerated rendering, and mesh-to-Gaussian illumination caching. Its principal advantages are high rendering quality and fast rasterization; its principal challenges are memory consumption, geometric ambiguity, dependence on camera poses and initialization, and the difficulty of representing complete, semantically or temporally consistent geometry.

1. Representation and image formation

A 3DGS scene is represented by a set of anisotropic Gaussian primitives. A common parameterization for Gaussian ii is

Gi=(μi,Σi,αi,ci),\mathcal{G}_i=(\boldsymbol{\mu}_i,\boldsymbol{\Sigma}_i,\alpha_i,\mathbf{c}_i),

where μi∈R3\boldsymbol{\mu}_i\in\mathbb{R}^3 is the center, Σi∈R3×3\boldsymbol{\Sigma}_i\in\mathbb{R}^{3\times3} is the covariance, αi\alpha_i is opacity, and ci\mathbf{c}_i is view-dependent color. The spatial Gaussian density is

Gi(x)=exp⁡[−12(x−μi)TΣi−1(x−μi)].G_i(\mathbf{x})= \exp\left[ -\frac{1}{2} (\mathbf{x}-\boldsymbol{\mu}_i)^{\mathsf T} \boldsymbol{\Sigma}_i^{-1} (\mathbf{x}-\boldsymbol{\mu}_i) \right].

The covariance is factorized into scale and rotation:

Σi=RiSiSiTRiT,\boldsymbol{\Sigma}_i = \mathbf{R}_i\mathbf{S}_i\mathbf{S}_i^{\mathsf T}\mathbf{R}_i^{\mathsf T},

where Si\mathbf{S}_i is diagonal and Ri\mathbf{R}_i is obtained from a rotation quaternion. This parameterization produces oriented ellipsoidal primitives rather than isotropic points. In one accounting used for pruning, a fourth-degree SH color representation gives 48 color coefficients; together with position, opacity, scale, and quaternion rotation, this yields 59 parameters per Gaussian (Zhang et al., 2024).

For a camera, the Gaussian is locally projected into the image plane. If W\mathbf{W} is the world-to-camera transformation and J\mathbf{J} is the Jacobian of the local projective approximation, the projected covariance is

Σi2D=JWΣiWTJT.\boldsymbol{\Sigma}^{2D}_i = \mathbf{J}\mathbf{W} \boldsymbol{\Sigma}_i \mathbf{W}^{\mathsf T} \mathbf{J}^{\mathsf T}.

The projected primitive defines an elliptical image-space footprint. Gaussians are approximately sorted from front to back, and their contributions are composited along each pixel or camera ray:

C(p)=∑i∈N(p)ciαi∏j<i(1−αj).\mathbf{C}(\mathbf{p}) = \sum_{i\in\mathcal{N}(\mathbf{p})} \mathbf{c}_i\alpha_i \prod_{j<i}(1-\alpha_j).

Here αi\alpha_i includes the learned opacity and the value of the projected two-dimensional Gaussian at pixel p\mathbf{p}. The product is the accumulated transmittance from nearer primitives. This explicit rasterization avoids repeated multilayer perceptron evaluations along densely sampled rays, which is a central distinction from NeRF-based rendering.

The standard reconstruction objective combines photometric and structural terms, commonly an L1L_1 image loss and an SSIM-derived loss. Adaptive density control begins from Structure-from-Motion (SfM) points and periodically duplicates or splits Gaussians in poorly reconstructed regions while removing sufficiently low-opacity primitives. This process can produce millions of Gaussians, establishing the computational and storage pressures addressed by subsequent research.

2. Optimization, initialization, and geometric structure

Vanilla 3DGS depends on calibrated camera poses and a suitable initial point cloud, commonly supplied by SfM or COLMAP. Image reconstruction losses alone do not uniquely determine three-dimensional structure: multiple Gaussian arrangements can produce similar training images. Consequently, Gaussian centers may be displaced from surfaces, primitives may float or overlap, and the resulting representation may have incomplete or view-dependent geometry.

Several methods modify initialization and optimization to impose geometric structure. GDGS uses an MLP-based geometry-guided initialization that predicts Gaussian centers, covariance matrices, and opacity from sparse geometry, followed by surface-aligned optimization. Its target center is defined from a nearby mesh point and surface normal:

μitarget=μimesh+dNi.\boldsymbol{\mu}^{\mathrm{target}}_i = \boldsymbol{\mu}^{\mathrm{mesh}}_i+d\mathbf{N}_i.

Distance, normal-alignment, and surface-fitting losses encourage Gaussians to remain near surfaces and orient consistently with surface normals. GDGS also introduces region-aware density control based on local complexity, density variance, high-gradient Gaussians, and a Top-20 regional term. On the reported averages, GDGS-30K improves over 3DGS-30K by $0.49$ PSNR on Mip-NeRF360, $1.25$ PSNR on Tanks and Temples, and $0.87$ PSNR on Deep Blending (Wang et al., 1 Jul 2025).

CDGS adds confidence-aware depth regularization. Monocular depth is aligned to sparse SfM depths by a scale-and-shift transformation,

Dtrans(p)=αDinit(p)+β,D_{\mathrm{trans}}(p)=\alpha D_{\mathrm{init}}(p)+\beta,

and its influence is controlled by both local confidence cues and a global alignment loss. The depth-loss weight is

λd(la)=λmax⁡e−kla,\lambda_d(l_a)=\lambda_{\max}e^{-kl_a},

with reported values λmax⁡=0.6\lambda_{\max}=0.6 and k=150k=150. CDGS uses edge, texture, and refined-depth cues to modulate depth supervision, thereby avoiding uniform reliance on uncertain monocular estimates. On Tanks and Temples, it reports an average M3C2 RMSE of $0.120$ m, compared with $0.124$ m for original 3DGS and $0.122$ m for DRGS (Zhang et al., 20 Feb 2025).

DET-GS addresses sparse geometric semantics for object detection rather than surface reconstruction. Boundary Guidance modifies image supervision with category-colored object boundaries, while Box-Focused Sampling projects 2D detections into three-dimensional frustums and preferentially samples Gaussians likely to belong to objects. The selected Gaussian attributes are passed to FCAF3D, a point-cloud-style detector. On ScanNet, the full method reports $59.9$ mAP@0.25[email protected] and $37.8$ mAP@0.5[email protected], compared with $54.3$ and $34.1$ for the basic 3DGS detection pipeline (Cao et al., 2024).

3DGS2^2 changes the optimizer rather than the representation. It exploits localized image-space influence and weak or structured coupling among Gaussian attributes by constructing small Newton systems for position, rotation, scale, opacity, and color. Position updates are reduced to a two-dimensional subspace perpendicular to the viewing direction; opacity uses an interior-point barrier; and nearby camera views are included to reduce stochastic overshoot. The method is described as near second-order because it uses local Hessian information without forming the global Hessian. It reports more than ten times fewer iterations in some cases and training-time reductions of approximately 5.1×5.1\times, 5.3×5.3\times, and 6.4×6.4\times on Mip-NeRF360, Tanks and Temples, and Deep Blending, respectively (Lan et al., 22 Jan 2025).

3. Compression, pruning, and representation reduction

The number of Gaussian primitives affects training memory, checkpoint size, disk storage, data transfer, rasterization, depth sorting, and alpha compositing. Compression methods operate at several levels: pruning, quantization, structured latent representations, entropy coding, and view-aware omission.

LP-3DGS replaces a manually selected pruning ratio or importance threshold with a trainable per-Gaussian mask. Given an importance score SiS_i and mask parameter mim_i, the opacity is modified as

oim=oi gs(miSi),o_{im}=o_i\,gs(m_iS_i),

where gsgs is a differentiable Gumbel-Sigmoid function. An L1L_1 mask regularizer,

Rmask=1N∑i=1N∣mi∣,R_{\mathrm{mask}} = \frac{1}{N}\sum_{i=1}^{N}|m_i|,

encourages sparsity while reconstruction losses preserve quality-critical Gaussians. The mask is learned during a 500-iteration interval within a 30,000-iteration training schedule, after which zero-mask Gaussians are physically removed. Using RadSplat scores, LP-3DGS reports an average pruning ratio of $0.63$ on Mip-NeRF360, with PSNR $27.47$, SSIM $0.8120$, and LPIPS $0.2269$. The final compact model remains compatible with standard 3DGS rendering (Zhang et al., 2024).

CAT-3DGS addresses rate-distortion-optimized transmission rather than only parameter reduction. It builds on ScaffoldGS anchors and uses PCA-oriented multi-scale triplanes as hyperpriors. Spatial autoregressive models exploit inter-anchor correlation in the triplanes, while channel-wise autoregressive models exploit dependencies within each anchor feature. View frequency-aware masking omits primitives rarely used in training-view rendering. The reported rate parameters range from $0.002$ to $0.04$ for most datasets, and the coded bitstream includes anchor positions, features, offsets, scaling factors, masks, triplanes, and decoder weights. On Mip-NeRF360, CAT-3DGS reports an average high-rate size of $12.35$ MB at PSNR $27.77$, compared with $744.7$ MB for vanilla 3DGS and $253.9$ MB for ScaffoldGS (Zhan et al., 1 Mar 2025).

EA-3DGS combines mesh-guided initialization, contribution-aware pruning, structure-aware densification, and vector quantization for outdoor scenes. It constructs an adaptive tetrahedral mesh and places Gaussian components on triangular faces, typically using k=3k=3 Gaussians per face. A contribution score aggregates opacity- and covariance-related effects over training-view rays. Low-curvature regions are subsequently analyzed through local covariance eigenvalues, with curvature estimated as

ρ=λ0λ0+λ1+λ2.\rho= \frac{\lambda_0}{\lambda_0+\lambda_1+\lambda_2}.

Vector quantization is applied separately to color, SH coefficients, scale, and rotation. In a Mill-19 ablation, codebook compression reduces storage from $1.8$ GB to $0.3$ GB, with PSNR decreasing from $23.39$ to $23.07$ and SSIM from $0.754$ to $0.747$ (Guo et al., 16 May 2025).

Dynamic 3DGS compression can be estimator-dependent when it assumes temporal correspondence or motion models. A robust estimator-agnostic approach concatenates groups of frames into one Gaussian set and augments each Gaussian with a frame index. Temporal redundancy is thereby converted into spatial redundancy that static codecs can exploit. Across six N3DV sequences, static codecs achieve reported BD-rate gains of −42.0%-42.0\% to −71.8%-71.8\% on tracked sets; on untracked sets, all evaluated codecs except HGSC remain competitive with per-frame coding (Wang et al., 22 Sep 2026).

4. Reconstruction under sparse, unposed, or contaminated observations

3DGS ordinarily assumes a static scene and known camera poses. Several extensions address violations of these assumptions.

T-3DGS removes transient objects rather than modeling dynamic scenes. Its Transient Mask Predictor is an unsupervised logistic classifier operating on DINOv2 features and reconstruction behavior. A consistency term discourages labeling semantically corresponding regions in both the reference and rendered images as transient:

Lconsist=∥P⊙P^∥1.\mathcal{L}_{\mathrm{consist}} = \|P\odot\hat P\|_1.

The Transient Mask Refiner uses SAM for spatial refinement and SAM2 for forward, backward, and final forward temporal propagation. The resulting masks exclude transient pixels from the 3DGS photometric objective, while depth total variation suppresses floating artifacts. On the T-3DGS dataset, the reported mean performance improves from PSNR $25.49$, SSIM $0.93$ without temporal refinement to PSNR $28.03$, SSIM $0.96$ with refinement (Markin et al., 2024).

D3^3GS targets sparse-view reconstruction with three to nine input views. It combines metric depth, DINO features, and RGB diffusion. MapAnything and bundle adjustment provide sparse metric anchors; a diffusion U-Net completes low-resolution depth; a DPT decoder produces high-resolution metric depth; and the depth is back-projected to initialize Gaussian means. Each Gaussian also stores a three-dimensional PCA-reduced DINO feature, which is rendered and supervised across views. A diffusion model periodically refines novel-view renderings, and the refined images become pseudo-targets for further Gaussian optimization. On DTU, the reported three-, six-, and nine-view PSNR values are $21.34$, $26.04$, and $27.16$, respectively; on LLFF they are $21.47$, $24.89$, and $26.59$ (Gao et al., 19 Sep 2026).

Unposed 3DGS reconstruction addresses the absence of camera poses and initial geometry in large outdoor image sequences. Input images are partitioned into overlapping groups, each processed by a pretrained MVS model. Local point clouds and poses are aligned using a similarity transformation in Sim⁡(3)\operatorname{Sim}(3). Probabilistic Procrustes Mapping assigns soft correspondence weights and uses a dustbin mechanism to reject uncertain matches. After global alignment, confidence-aware anchors initialize Gaussians, and camera poses and Gaussian parameters are jointly refined through differentiable rendering with analytical pose Jacobians. On Waymo, the method reports ATE $1.41$, PSNR $31.53$, SSIM $0.915$, and LPIPS $0.245$, with approximately one minute of mapping time and 14 GB of GPU memory (Cheng et al., 24 Jul 2025).

D3^3GS and T-3DGS have different objectives. D3^3GS attempts to complete sparse observations with metric and generative priors, whereas T-3DGS suppresses transient observations to recover a persistent static scene. Neither changes the fundamental Gaussian rasterization model.

5. Applications and systems

3DGS is used as a representation for novel-view synthesis, object detection, geometric reconstruction, digital twins, heritage preservation, forestry, UAV photogrammetry, and interactive rendering.

For edge rendering, 3DGauCIM co-designs algorithms and hardware for static and dynamic 3DGS. DRAM-access reduction frustum culling discards entire spatial-temporal grids without loading their Gaussian records. Adaptive Tile Grouping improves on-chip reuse by processing strongly connected tiles together. Adaptive Interval Initialization Bucket-Bitonic Sort initializes depth buckets from previous frames to reduce sorting imbalance. A DCIM-friendly computation flow approximates exponentials with base-2 decomposition, a lookup table, and bit-shift operations. The reported end-to-end results are 203 FPS at $0.28$ W for large-scale real-world static scenes and 211 FPS at $0.63$ W for dynamic scenes (Huang et al., 25 Jul 2025).

3DGS3^3 accelerates high-resolution and high-frame-rate rendering without changing the Gaussian representation or splatting kernel. It renders a lower-resolution frame and applies two post-rendering modules: Gradient-Aware Super Sampling (GASS) and Lightweight Temporal Frame Interpolation (LTFI). GASS uses analytical spatial derivatives of the differentiable Gaussian image formation process, depth, normals, and recurrent refinement. LTFI uses pose-based warping, feature fusion, adaptive blending, and a recurrent temporal state. At 4096×21604096\times2160, it reports 96.47 FPS on Mip-NeRF360, compared with 17.35 FPS for direct high-resolution 3DGS; at 2560×14402560\times1440, it reports 226.50 FPS on Deep Blending (Zhao et al., 12 May 2026).

Mesh2GS addresses the inverse conversion from mesh to 3DGS. It uses plenoptic sampling theory to derive camera spacing and Gaussian spacing from focal length, depth range, image resolution, and light-field frequency. The world-space sampling interval is

Δw=zmin⁡Δvf,\Delta w= \frac{z_{\min}\Delta v}{f},

and Gaussian centers are placed slightly inward from the visible mesh surface. Tangential scales are set to Δw\Delta w, while the normal scale is Δw2/(4zmin⁡)\Delta w^2/(4z_{\min}). The method separates albedo from shading and updates Gaussian colors while keeping geometry fixed. A neural illumination enhancement module models non-Lambertian residuals such as specular highlights. In the reported global-illumination experiments, Mesh2GS achieves PSNR $31.24$ on Chair Room, $23.75$ on Bed Room, and $17.12$ on Veach Room (Zhu et al., 20 Jun 2026).

3DGS-DET demonstrates that explicit Gaussian primitives can serve as inputs to point-cloud-style 3D detectors. The Gaussian representation provides direct three-dimensional centers, geometric extents, opacity, and appearance features, while Boundary Guidance and Box-Focused Sampling make the representation more object-centric. The final detector remains FCAF3D rather than a Gaussian-specific detection architecture (Cao et al., 2024).

6. Evaluation, perceptual quality, and limitations

Common 3DGS evaluation metrics include PSNR, SSIM, and LPIPS for novel-view synthesis; precision, recall, F-score, Chamfer distance, and M3C2 distance for geometry; mAP@0.25[email protected] and mAP@0.5[email protected] for object detection; training time, peak GPU memory, rendering FPS, file size, and BD-rate for systems and compression.

3DGS-IEval-15K was introduced to evaluate perceptual quality specifically for compressed 3DGS. It contains 15,200 rendered images from 10 real scenes, six compression algorithms, and 20 viewpoints per scene. Sixty participants supplied 228,000 ratings, producing mean-opinion scores over geometry-only, color-only, and mixed distortions. The dataset explicitly separates training-like viewpoints from challenging novel or extrapolated viewpoints. Among the evaluated objective metrics, HYPERIQA reports the strongest overall correlation with human MOS, with SRCC $0.9407$, PLCC $0.9391$, and KRCC $0.8121$, whereas PSNR reports SRCC $0.6451$ (Xing et al., 17 Jun 2025).

The literature identifies recurring limitations:

  • Memory and model size: uncompressed scenes may contain millions of Gaussians, increasing training memory, checkpoint size, storage, and transfer cost.
  • Geometric ambiguity: photometric reconstruction does not uniquely constrain three-dimensional surfaces, especially under sparse views, occlusions, weak texture, reflections, and view-dependent appearance.
  • Initialization dependence: unreliable SfM or MVS geometry can produce holes, floaters, pose errors, and incomplete surfaces.
  • Scene-dependent compression: pruning thresholds and safe compression levels vary with geometry, texture, coverage, redundancy, and viewpoint distribution. LP-3DGS learns a pruning level but remains dependent on the underlying importance score.
  • External-model dependence: depth-aware, semantic, transient-removal, and diffusion-guided methods depend on estimators such as Depth Anything V2, DINO variants, SAM/SAM2, DPT, MapAnything, or diffusion models.
  • Static-scene assumptions: many methods remove or suppress dynamics rather than reconstructing a temporally coherent dynamic scene.
  • Runtime trade-offs: autoregressive compression improves rate-distortion performance but increases decoding time; post-rendering acceleration shifts cost from rasterization to neural reconstruction; hardware optimizations rely on spatial and temporal coherence.
  • Incomplete geometric guarantees: SDF, depth, mesh, and normal regularization improve geometric behavior but do not eliminate depth-estimation errors, unobserved surfaces, or ambiguities in non-Lambertian and volumetric content.
  • Evaluation gaps: image metrics do not fully measure three-dimensional consistency, continuous camera-path quality, temporal coherence, energy consumption, or perceptual artifacts specific to compressed Gaussian representations.

Taken together, these developments position 3DGS as more than a rendering primitive. It is a flexible explicit representation whose central pipeline—anisotropic Gaussian parameterization, differentiable projection, depth ordering, and alpha compositing—can be augmented by geometric priors, learned optimization, semantic supervision, compression models, hardware scheduling, and generative refinement. The principal research direction is the controlled allocation of Gaussian capacity: deciding where primitives should exist, how they should be organized geometrically and temporally, which attributes should be stored or transmitted, and how their rendered consequences should be evaluated across viewpoints.

Topic to Video (Beta)

No one has generated a video about this topic yet.

Whiteboard

No one has generated a whiteboard explanation for this topic yet.

Follow Topic

Get notified by email when new papers are published related to 3DGS.