3D Gaussian Splatting Applications

• 3DGS is a scene representation method that models environments using anisotropic 3D Gaussians, offering interpretability and rapid, high-fidelity rendering.
• Its explicit representation allows direct geometric and semantic manipulation, facilitating applications like segmentation, editing, and generation with real-time performance.
• Integration with 2D foundation models such as CLIP and SAM enhances semantic control and enables versatile multi-modal workflows in advanced 3D scene processing.

3D Gaussian Splatting (3DGS) is an explicit 3D scene representation technique that models complex scenes as a set of anisotropic 3D Gaussian primitives. Each Gaussian encodes spatial position, covariance (shape and orientation), color (often via spherical harmonics), and opacity. Unlike implicit neural field methods (e.g., NeRF), 3DGS leverages differentiable rasterization to facilitate real-time, high-fidelity photorealistic rendering and interpretable scene manipulation. Its explicit, compact representation supports direct geometric and semantic manipulation, making it suitable for a rapidly expanding range of downstream applications. The following sections systematically review the taxonomy of 3DGS applications, including segmentation, editing, and generation, along with enabling methodologies and supporting evaluation protocols (He et al., 13 Aug 2025).

1. Fundamental Properties and Advantages of 3DGS

3DGS departs from the implicit continuous fields of neural radiance field (NeRF) models, representing a scene as an explicit set of anisotropic Gaussians. Rendering proceeds by rasterizing these primitives, projecting each Gaussian onto the image plane, and accumulating their contributions with alpha blending. This design enables fast rendering, interpretability, and the ability to directly interface with geometric and semantic cues. Photorealistic rendering quality is achieved owing to the fine-grained geometric and appearance details encapsulated within each primitive. Furthermore, the explicit nature of the representation supports real-time applications, as the pipeline circumvents the expensive per-ray volumetric integration characteristic of NeRF-based methods.

Key distinctions between 3DGS and NeRF include:

• Explicit representation vs. implicit neural-valued fields
• Direct supervision and interpretability vs. non-transparent neural encoding
• Rasterization-based rendering for real-time performance

2. Segmentation, Editing, Generation, and Other Applications

3DGS is highly versatile, with applications spanning segmentation, editing, generation, and multiple additional scene understanding tasks.

Segmentation is sub-divided as follows:

• Generic segmentation: Predicts pixel- or primitive-level semantic, instance, or panoptic labels without explicit prompts (e.g., RT-GS2, OmniSeg3D, Unified-Lift).
• Promptable segmentation: Utilizes external prompts (e.g., points, boxes, text, referring expressions) to direct the segmentation process. This includes interactive segmentation (SAGD, Click-Gaussian), open-vocabulary segmentation using category names (LangSplat, Feature-3DGS), and referring segmentation (Ref-LERF). Outputs may be rendered in 2D or produced directly on the Gaussians, enabling multi-modal workflows.

Editing comprises techniques for geometry, appearance, or illumination manipulation. Input modalities include:

• Text-guided editing: Language used to localize and guide edits of scene regions; methods integrate semantic cues from 2D foundation models (e.g., GaussianEditor, GSEditPro).
• Image-guided editing: Reference image(s) or parameter-efficient fine-tuning (e.g., LoRA adaptations) drive appearance modifications (e.g., TIP-Editor, GS-VTON), often focusing on multi-view consistency.

Generation focuses on synthesizing scenes or objects from text, sparse views, or single images:

• Text-to-3D: Guidance via Score Distillation Sampling (SDS) combined with 2D diffusion model priors (DreamGaussian, GaussianDreamer, StableDreamer).
• Image-to-3D: Point cloud or mesh structures serve as intermediate steps before refinement into a full 3D Gaussian set (ScalingGaussian, GS-LRM).
• Outputs occur at both object and scene levels.

Other Tasks include:

• Human avatar modeling: Both body-based (SMPL/SMPL-X) and head-based approaches.
• SLAM: Scene mapping and On-device or web-scale simultaneous localization and mapping using geometry and learned semantics from Gaussians.
• Object detection: Both promptable (open-vocabulary) and generic methods.

3. Semantic Control via 2D Foundation Model Integration

3DGS application pipelines often rely on pre-trained 2D vision foundation models, such as CLIP, SAM, and DINO, to provide semantic features with generalization and open-vocabulary capabilities. The workflow typically consists of:

• Using SAM to generate instance or hierarchical masks per view.
• Extracting region-level semantic embeddings from CLIP.
• Compressing these descriptors with (optionally) a scene-specific autoencoder before injecting them into the learning pipeline for supervising Gaussian-level features.

This approach facilitates open-vocabulary control, referring segmentation, and multi-modal tasks by directly transferring the semantic knowledge from large-scale 2D datasets into the explicit 3DGS scene.

Mathematically, the learning objective for per-scene optimization is given by

$$\tilde{\theta}_s \in \arg\min_{\theta_s} \frac{1}{N} \sum_{n=1}^N \epsilon\left( f_{\theta_s}(x_n^s), x_n^s \right)$$

where $\epsilon$ is a reconstruction or semantic loss, $f_{\theta_s}$ is the scene-specific mapping, and $x_n^s$ are data samples for scene $s$. In feed-forward pipelines, a universal $f_\theta$ is trained over many scenes, supporting single-pass inference on novel targets.

4. Learning Paradigms and Supervision Strategies

3DGS-based applications employ two primary paradigms:

• Per-scene optimization: A mapping $f_{\theta_s}$ is fit for each scene individually, often yielding high-fidelity reconstructions but demanding substantial compute for every new case.
• Feed-forward inference: A universal $f_\theta$ is learned across a sizable scene corpus, enabling direct inference or fine-tuning on new inputs for real-time applications; this approach trades some detail for inference speed and scalability.

Supervision strategies include direct loss-based supervision from 2D semantic features, contrastive learning, codebook-based quantization, and instance clustering. Graph-cut postprocessing, hierarchical attention, and cross-view consistency terms further refine instance boundaries and enforce semantic coherence.

Emerging shared principles:

• Decoupling color, geometry, and semantics explicitly for interpretable and versatile manipulation.
• Hierarchical and multi-scale feature design for capturing both global and local structure.
• Cross-view and 3D-to-2D consistency mechanisms for multi-modal congruence.

5. Representative Datasets and Evaluation Protocols

Standardized benchmarks underpin the comparative evaluation of 3DGS applications:

Task	Common Benchmarks	Key Metrics
Segmentation	ScanNet, Replica, NVOS, SceneSplat	mIoU, F-score, PQ, mAcc, mBIoU
Editing	DTU, Tanks/Temples, GL3D, LLFF	CLIP similarity, FID, SSIM, LPIPS, PSNR
Generation	ShapeNet, ScanNet, Objaverse, G-Objaverse	PSNR, SSIM, FID, LPIPS, KID, user paper

For segmentation, metrics such as mean intersection-over-union (mIoU), panoptic quality (PQ), and F-score predominate, while editing relies on multi-modal similarity (e.g., CLIP, FID) and perceptual/structural metrics. Generation tasks are assessed with image synthesis metrics as well as large-model-driven MLLM scoring frameworks.

Comparative tables in surveys reflect the performance ranking of contemporary methods (e.g., Unified-Lift excelling in instance segmentation, NovelGS for high-quality 3D generation), synthesizing advances across functional categories.

6. Methodological and Developmental Trends

Recent and ongoing trends in 3DGS applications include:

• Shift toward feed-forward pipelines: Iterative optimization is being displaced by models supporting direct inference for practical real-time, interactive, or web-scale deployment.
• Diffusion model integration: 2D or cross-modal diffusion priors are increasingly utilized for both generative and editing tasks.
• Latent-space modeling: Encodings such as codebooks and low-dimensional autoencoders are being developed for geometry, appearance, and semantics compression and manipulation.
• Explicit separation of attributes: Decoupled optimization and representation of geometric, radiometric, and semantic properties ease editing, transfer, and interpretability.
• Multi-modal and multi-task learning: Expansion from canonical NVS (novel view synthesis) to detection, SLAM, avatar modeling, and more, often within a unified backbone leveraging shared representations.

A continually updated repository of current literature, resources, and codebases facilitates ongoing development: https://github.com/heshuting555/Awesome-3DGS-Applications (He et al., 13 Aug 2025).

7. Outlook and Significance

The adoption of 3DGS for explicit, differentiable, and semantically enriched 3D scene representation marks a significant advancement for both computer vision research and downstream applications requiring geometric and semantic understanding. The ability to perform high-quality segmentation, intuitive object-driven editing, and robust 3D synthesis supports a broad spectrum of use cases—from immersive XR and robotics to generative content creation and digital asset management.

By integrating 2D foundation models and developing both per-scene and feed-forward pipelines, the field is moving toward scalable, interactive, and semantically aware 3D perception and manipulation. Continued convergence of geometric, radiometric, and semantic supervision in 3DGS will likely lead to further improvements in rendering quality, model efficiency, and downstream task performance, with significant impact across applied and foundational disciplines.