Label-aware 3D Gaussian Splatting

• Label-aware 3D Gaussian Splatting extends conventional 3D Gaussian methods by integrating label information for improved segmentation and object extraction.
• It employs varied labeling strategies—discrete IDs, semantic logits, dual-opacity, and coordinate transfers—to balance clarity with computational efficiency.
• The approach achieves cross-view consistency and notable performance gains, demonstrating robust novel-view synthesis and efficient scene understanding.

Label-aware 3D Gaussian Splatting (LabelGS) denotes a class of 3D Gaussian Splatting methods that augment the standard Gaussian scene representation with label-bearing, label-rendering, or label-propagation mechanisms so that novel-view synthesis can be coupled to 3D scene segmentation, instance reasoning, panoptic lifting, part-aware editing, or language-grounded object selection. In the literature, this label awareness is realized in several distinct ways: explicit discrete object IDs attached to Gaussians, per-Gaussian semantic or instance logits, view-dependent label fields, dual-opacity occupancy branches, and coordinate-aware annotation transfer around an otherwise geometry-only renderer. The resulting systems retain the explicit, efficient rasterization pipeline of 3DGS while extending it toward segmentation and downstream scene understanding tasks (Zhang et al., 27 Aug 2025, Wang et al., 2024, Xie et al., 14 Oct 2025, Liu et al., 19 May 2026, Huo et al., 2024, Kim et al., 15 Jul 2025, Tan et al., 26 May 2026).

1. Foundational concept and design space

Standard 3D Gaussian Splatting represents a scene as a set of anisotropic Gaussians with geometry, opacity, and radiance parameters, and renders images by projecting Gaussians to screen space and compositing them front-to-back. A representative formulation used across LabelGS-related work writes the rendered pixel color as

$$C(v)=\sum_i c_i \alpha_i T_i,\qquad T_i=\prod_{j<i}(1-\alpha_j),$$

with $\alpha_i$ determined by projected Gaussian density and opacity. Label-aware variants preserve this compositing structure but alter what else is attached to each Gaussian, or how labels are transported around the renderer (Zhang et al., 27 Aug 2025, Liu et al., 19 May 2026, Xie et al., 14 Oct 2025).

The design space is heterogeneous rather than canonical. The paper explicitly titled “LabelGS” uses a discrete object label $B_j$ per Gaussian and defines object extraction by filtering Gaussians with the same label. PLGS instead constructs semantic and instance fields over a structured Gaussian representation and renders panoptic outputs by alpha-compositing class and instance probabilities. UniGS renders semantic logits together with RGB, depth, normals, and a pruning factor in one unified rasterization pass. OP2GS assigns each Gaussian an explicit instance ID together with a second opacity $\sigma^{*}$ dedicated to object-mask rendering. RoMaP introduces spherical-harmonic label coefficients $\mathbf{r}_i$ to obtain view-dependent soft part labels. EGSRAL is label-aware in a different sense: labels are not stored on Gaussians, but are propagated through a learned mapping between coordinate systems so that synthesized views inherit transformed annotations. TrackRef3D further extends the paradigm to open-world referring segmentation by aligning Gaussian features with canonical object descriptions derived from multi-view tracking (Zhang et al., 27 Aug 2025, Wang et al., 2024, Xie et al., 14 Oct 2025, Liu et al., 19 May 2026, Kim et al., 15 Jul 2025, Huo et al., 2024, Tan et al., 26 May 2026).

System	Label carrier	Distinctive mechanism
LabelGS	discrete label $B_j$	OAM, GPF, random region sampling
PLGS	semantic logits $\mathbf{s}_i$, instance logits $\mathbf{h}_i$	semantic anchors, cluster loss, self-training
UniGS	semantic logits $\mathbf{o}$	unified multimodal rasterization
OP2GS	instance label $l$, instance opacity $\alpha_i$0	dual-opacity object occupancy
RoMaP	SH label coefficients $\alpha_i$1	3D-GALP, masked local editing
EGSRAL	transformed annotations, not per-Gaussian labels	OWCS–EWCS adaptor
TrackRef3D	referring features plus trajectory-level identities	track-then-label consensus

A recurrent misconception is that LabelGS necessarily means “semantic labels stored directly on Gaussians.” The literature does not support that restriction. EGSRAL demonstrates a coordinate-aware formulation in which the renderer remains geometry-only while the rendering process becomes label-aware through consistent pose and annotation transformation; conversely, OP2GS, UniGS, PLGS, RoMaP, and the discrete-label LabelGS paper all embed label information more directly into the Gaussian representation (Huo et al., 2024, Liu et al., 19 May 2026, Xie et al., 14 Oct 2025, Wang et al., 2024, Kim et al., 15 Jul 2025, Zhang et al., 27 Aug 2025).

2. Label representation inside Gaussian primitives

The most direct formulation is the discrete-label representation of LabelGS, where a label-aware scene is written as

$\alpha_i$2

with $\alpha_i$3 denoting an object or instance ID. Object extraction is then implemented by

$\alpha_i$4

This design makes segmentation and object rendering a filtering operation on Gaussians rather than a feature-decoding problem, and the paper explicitly contrasts this with high-dimensional feature-based approaches such as Feature-3DGS and LangSplat (Zhang et al., 27 Aug 2025).

A second family stores dense semantic or instance predictions per Gaussian. PLGS assigns each Gaussian semantic logits $\alpha_i$5 and instance logits $\alpha_i$6, with the corresponding per-pixel distributions produced by alpha-compositing the Gaussian probabilities. UniGS similarly stores semantic logits $\alpha_i$7 per Gaussian and renders per-pixel logits by the same compositing rule used for color, but within a geometry-aware multimodal rasterizer that simultaneously outputs RGB, depth, normals, semantic logits, and a pruning factor. These formulations are suited to class-level or panoptic outputs because labels remain differentiable fields rather than fixed IDs (Wang et al., 2024, Xie et al., 14 Oct 2025).

OP2GS makes object identity explicit but separates identity from appearance supervision by adding an integer instance label $\alpha_i$8 and an instance opacity $\alpha_i$9. The original opacity $B_j$0 remains responsible for RGB reconstruction, while $B_j$1 is used only for rendering object masks:

$B_j$2

This decoupling is central to the paper’s account of “label contamination”: Gaussians that are useful for appearance can be made transparent in the object-mask branch without being removed from RGB rendering (Liu et al., 19 May 2026).

RoMaP adopts a different representation appropriate for part-level editing. Each Gaussian is extended from

$B_j$3

to $B_j$4, where $B_j$5 is an SH-parameterized label vector. For a viewing direction $B_j$6, the Gaussian produces a view-dependent label signal $B_j$7, which is then splatted into a soft 2D segmentation map. This representation explicitly models view-dependent label variation and soft-label behavior near part boundaries (Kim et al., 15 Jul 2025).

These alternatives imply different notions of “label.” In discrete-label systems, the label is a hard partition of the Gaussian set. In logit-based systems, the label is a rendered distribution. In SH-based systems, the label is a view-dependent field. In coordinate-aware systems such as EGSRAL, a plausible description is that the label resides in the transformation pipeline rather than in the primitive itself (Zhang et al., 27 Aug 2025, Wang et al., 2024, Kim et al., 15 Jul 2025, Huo et al., 2024).

3. Lifting 2D supervision into 3D and enforcing cross-view consistency

A central problem in LabelGS is that 2D masks are noisy, view-inconsistent, and vulnerable to occlusion. The discrete-label LabelGS paper addresses this by generating cross-view consistent masks with DEVA, and, for sparse-view datasets, by introducing Densification of View (DOV): a vanilla 3DGS is first trained, intermediate pseudo-camera views are rendered, DEVA is run on the dense sequence, and only masks corresponding to real training images are retained. Label assignment then follows the dominant rendering contribution, with the main contributing Gaussian defined by $B_j$8, and assignments applied only when $B_j$9. Two further mechanisms refine this lifting. The Occlusion Analysis Model (OAM) uses DepthAnythingV2 to build occlusion lists

$\sigma^{*}$0

and unoccluded masks

$\sigma^{*}$1

so that label supervision ignores pixels where an object is occluded. The Gaussian Projection Filter (GPF) restricts lifting to pixels whose label matches the label at the projected Gaussian center, thereby reducing boundary conflicts (Zhang et al., 27 Aug 2025).

PLGS tackles a related problem in panoptic lifting by arguing that conventional 3DGS is more susceptible to noisy 2D mask supervision because it lacks NeRF-like smoothness. Its response is a panoptic-aware structured Gaussian model based on Scaffold-GS. For semantics, it constructs consistency-enhanced masks by cross-view warping with depth, projects consistent pixels into 3D, clusters them with K-means, and assigns majority-vote semantic labels to cluster centers. These semantic anchor points initialize and regularize anchor-based Gaussians through a cluster loss. For instances, PLGS back-projects 2D instance masks to 3D, fits oriented bounding boxes, and uses IoU+IoM costs with the Hungarian algorithm to build a globally consistent instance set, which is then reprojected to provide supervision (Wang et al., 2024).

TrackRef3D generalizes consistency enforcement from object masks to open-world language grounding. The pipeline first discovers objects per view using Florence-2 and SAM-2, then links masks across views with DEVA into trajectories. The Trajectory-Aware Semantic Consensus Module performs synonymous clustering using SentenceTransformer embeddings and agglomerative clustering, followed by trajectory-aware voting

$\sigma^{*}$2

to assign a canonical semantic identity to each trajectory. Visibility-aware keyframe selection then chooses the frame used for description generation by maximizing

$\sigma^{*}$3

This produces multi-view consistent object labels and descriptions before any Gaussian-language alignment is learned (Tan et al., 26 May 2026).

EGSRAL shows that 2D-to-3D lifting can also be replaced by coordinate transfer. Its neural adaptor maps camera poses from the Original World Coordinate System (OWCS) to the Estimated World Coordinate System (EWCS), where the 3DGS is trained. Novel views are sampled in OWCS, transformed into EWCS for rendering, and their 3D annotations are transformed in OWCS using the same random affine perturbation. The result is automatic annotation generation without explicit per-Gaussian semantics. The paper is explicit that “EGSRAL does not assign semantic or instance labels directly to Gaussians”; label association is maintained at the level of coordinate frames and transforms (Huo et al., 2024).

Taken together, these works establish cross-view consistency as the decisive difficulty in LabelGS. The technical responses vary—tracking, geometric filtering, structured anchors, object trajectories, or coordinate adaptors—but all are designed to prevent contradictory 2D evidence from fragmenting the 3D representation (Zhang et al., 27 Aug 2025, Wang et al., 2024, Tan et al., 26 May 2026, Huo et al., 2024).

4. Rendering, loss design, and optimization strategies

Most LabelGS formulations preserve 3DGS alpha compositing and add label-specific branches or losses. In the discrete-label LabelGS paper, the core label consistency objective is

$\sigma^{*}$4

and the overall training objective is

$\sigma^{*}$5

Label-aware optimization begins from the 1000-th iteration, and each iteration samples 10 object masks per image for efficiency (Zhang et al., 27 Aug 2025).

PLGS renders semantic and instance distributions by compositing per-Gaussian semantic and instance probabilities:

$\sigma^{*}$6

Its total loss combines reconstruction, semantic, and instance terms,

$\sigma^{*}$7

with $\sigma^{*}$8, $\sigma^{*}$9, $\mathbf{r}_i$0, and $\mathbf{r}_i$1. The semantic branch is regularized by a cluster loss over anchor-associated Gaussians, and the method alternates initial training with projected semantic masks and later self-training with pseudo labels (Wang et al., 2024).

UniGS integrates semantics into a geometry-aware rasterizer. Each Gaussian stores semantic logits $\mathbf{r}_i$2 and the rasterizer outputs per-pixel logits

$\mathbf{r}_i$3

together with RGB, depth, normals, and a pruning factor. The method differs from center-depth formulations by rendering depth from analytic ray–ellipsoid intersections, allowing depth and normal losses to update rotation and scale as well as position. Semantic supervision is standard cross-entropy on rendered logits, but the optimization is explicitly joint:

$\mathbf{r}_i$4

This makes geometry supervision part of the semantic training mechanism rather than a separate auxiliary objective (Xie et al., 14 Oct 2025).

OP2GS introduces a separate optimization path for object occupancy. Stage 1 performs standard 3DGS training with

$\mathbf{r}_i$5

then Stage 2 fixes per-Gaussian labels from multi-view majority voting and learns $\mathbf{r}_i$6 with

$\mathbf{r}_i$7

The object branch uses a random object loss in which only $\mathbf{r}_i$8 object IDs are sampled per iteration, and the mask gradients act on $\mathbf{r}_i$9 and geometry but not on the RGB opacity $B_j$0 or color. This dual-branch separation is the paper’s main answer to visibility ambiguities and mislabeled Gaussians (Liu et al., 19 May 2026).

RoMaP turns label rendering into the control variable for local 3D editing. The rendered label map $B_j$1 is trained against a soft pseudo 2D mask using an $B_j$2 loss, while 3D-GALP imposes a neighbor-consistency regularizer

$B_j$3

The learned 3D mask then gates a regularized SDS objective,

$B_j$4

so that only target-part Gaussians are updated during editing (Kim et al., 15 Jul 2025).

The common pattern is that label-aware rendering is rarely an isolated head. It is usually entangled with visibility modeling, geometric regularization, or supervision scheduling. This suggests that, in practice, LabelGS is an optimization strategy as much as a representational extension (Zhang et al., 27 Aug 2025, Xie et al., 14 Oct 2025, Liu et al., 19 May 2026, Kim et al., 15 Jul 2025).

5. Downstream uses and reported empirical performance

Label-aware 3DGS has been evaluated in segmentation, panoptic lifting, open-vocabulary retrieval, multimodal reconstruction, synthetic data generation, and localized editing. On 3D scene segmentation, the discrete-label LabelGS paper reports on 3D-OVS that Feature-3DGS achieves mIoU 0.22 and PSNR 19.7, Gaussian Grouping achieves mIoU 0.57 and PSNR 23.0, SAGS achieves mIoU 0.76 and PSNR 30.4, and full LabelGS achieves mIoU 0.925 and PSNR 34.26. On LLFF, LabelGS reports mIoU 0.851 and PSNR 28.02; on LERF-MASK, mIoU reaches 0.75 with PSNR 32.47 in the w/o GPF variant and 0.80/0.82 in ablations; on Mip-NeRF360, full LabelGS reports mIoU 0.75 and PSNR 32.42, with ablations reaching mIoU 0.79 and PSNR 32.73. At $B_j$5 for 15k iterations on the sofa scene, Feature-3DGS requires 443 minutes, Gaussian Grouping 28 minutes, LangSplat 22 minutes, and LabelGS 20 minutes, which the paper summarizes as a 22× speedup over Feature-3DGS (Zhang et al., 27 Aug 2025).

For panoptic lifting, PLGS reports that NeRF-based Panoptic Lifting, DM-NeRF, and Contrastive Lift require ~22–24 hours per scene and render at 0.1–0.7 FPS, whereas PLGS trains in ~2 hours per scene and renders at ~20.8 FPS. The paper also reports average gains of +5.8 mIoU and +10.1 PQ$B_j$6 over naive 3DGS baselines, with additional ablations attributing +2.4 mIoU and +2.2 PQ$B_j$7 to the structured representation, +1.1 mIoU and +2.2 PQ$B_j$8 to robust semantic initialization, +2.5 mIoU to self-training, and +5.3 PQ$B_j$9 to consistent instance guidance (Wang et al., 2024).

On object-aware open-vocabulary segmentation, OP2GS reports mean mIoU 97.1% on 3DOVS, compared with 96.4% for ObjectGS and 89–96% for other baselines. On LERF-Mask, it reports mIoU/mBIoU pairs of 92.3/89.1 for figurines, 91.0/84.6 for ramen, and 87.0/85.4 for teatime. Runtime is reported as 121.7 FPS at 1.6M Gaussians, compared with 43.6 FPS for Gaussian Grouping and 78.6 FPS for ObjectGS. Image quality on LERF-Mask is reported as PSNR 25.21, SSIM 0.890, and LPIPS 0.187, versus 25.87, 0.867, and 0.211 for 3DGS (Liu et al., 19 May 2026).

UniGS evaluates label-aware multimodal rendering rather than pure segmentation. On Replica Room0, SGS-SLAM reports PSNR 36.28, mIoU 0.976, and 96.4 FPS, while UniGS reports 36.66, 0.987, and 184.2 FPS. The paper states that similar improvements hold on Room2 and Office2, with mIoU improving from approximately 0.944–0.978 to 0.975–0.984, and argues that a unified representation is preferable to task-specific semantic Gaussians (Xie et al., 14 Oct 2025).

EGSRAL uses label-aware synthetic data generation for driving scenes. On nuScenes, it reports PSNR 29.04 and shows that auto-labeled synthetic images improve downstream detectors. For Co-DETR on the sample set, mAP rises from 23.7 to 25.9 to 26.8 as data scale increases from 1× to 2× to 3×; on the all set, mAP rises from 28.1 to 29.9 to 31.3. For MonoLSS, AP3D on sample increases from 17.21 to 19.78 to 20.31, and on all from 21.69 to 22.15 to 22.87. With a larger base dataset of 34 scenes and 8,497 images, 2D mAP on sample rises from 28.6 to 30.2 to 31.5, and 3D AP3D on sample from 22.16 to 23.36 to 23.89 (Huo et al., 2024).

TrackRef3D evaluates the language-aware extension of LabelGS. On Ref-LERF, average mIoU improves from 29.2 for ReferSplat to 38.8 for TrackRef3D. On the self-collected Laboratory dataset, referring segmentation improves from 37.4 to 48.5, while semantic short-query performance improves from 24.4 to 68.3. On LERF-OVS, average mIoU rises from 55.4 to 63.2, and on 3D-OVS from 94.1 to 95.2. These results are attributed to the combination of TSCM and the Hybrid Training Strategy (Tan et al., 26 May 2026).

RoMaP applies label-aware Gaussian masks to local 3D editing rather than segmentation benchmarks. It reports CLIP 0.277, CLIP$\mathbf{s}_i$0 0.205, BLIP-VQA 0.723, and TIFA 0.674, compared with best baseline values of 0.201, 0.095, 0.497, and 0.571 respectively. For 3D segmentation quality on complex 3D-OVS scenes, it reports average mIoU 0.559 for 3D-GALP, compared with 0.076 for LangSplat and 0.288 for LeGaussian, indicating that the label field is not merely an editing utility but also a strong segmentation mechanism (Kim et al., 15 Jul 2025).

6. Limitations, boundary cases, and future directions

The most immediate limitation across LabelGS methods is dependence on external supervision quality. LabelGS depends on DEVA for cross-view consistent masks, and the paper states that missing small objects, tracking failures, or inaccurate masks directly limit segmentation capability; heavily occluded or very tiny objects remain difficult, and GPF can be too restrictive in dense 360° scenes (Zhang et al., 27 Aug 2025). PLGS is motivated by the observation that conventional 3DGS is more sensitive to noisy 2D labels than NeRF because it lacks a shared parametric structure, so its own structured representation should be read as a mitigation rather than a complete solution to label noise (Wang et al., 2024). OP2GS still relies on SAM or SAM2 pseudo masks and assigns a single label to each Gaussian, which the paper identifies as insufficient for more complex semantics or partial overlaps (Liu et al., 19 May 2026).

A second limitation concerns what is actually labeled. EGSRAL explicitly notes that it has no explicit semantic Gaussians, cannot render a semantic map or instance map directly from the Gaussian set, depends on SfM quality, and does not enforce temporal label consistency. UniGS is class-level rather than instance- or panoptic-aware and focuses on supervised offline reconstruction of static scenes. TrackRef3D relies on the quality of Florence-2, SAM-2, DEVA, and CogVLM-2, and its multi-stage pipeline inherits those models’ failure modes. RoMaP’s part labels are optimized for local editing and depend on text-conditioned attention maps rather than dense human annotation (Huo et al., 2024, Xie et al., 14 Oct 2025, Tan et al., 26 May 2026, Kim et al., 15 Jul 2025).

The literature also makes clear that “label awareness” is not a single architectural choice but a trade-off between explicitness, efficiency, and robustness. Discrete labels give crisp extraction and low memory cost but can struggle with ambiguity and mixed boundaries. Logit fields support differentiable semantics but may require stronger regularization against noise. Dual-opacity designs reduce contamination but keep identity fixed once initialized. Coordinate-aware pipelines preserve a lightweight geometry-only core but do not yield semantic Gaussians. This suggests that future LabelGS systems will likely hybridize these lines: explicit object IDs or instance opacities for clean occupancy, structured semantic fields for smoothness, geometry-aware rasterization for sharper boundaries, and track-then-label or adaptor-based consistency mechanisms for multi-view supervision (Zhang et al., 27 Aug 2025, Liu et al., 19 May 2026, Wang et al., 2024, Xie et al., 14 Oct 2025, Tan et al., 26 May 2026, Huo et al., 2024).

Several papers state these directions explicitly. EGSRAL outlines per-Gaussian semantic and instance labels, joint training with detectors, panoptic or open-vocabulary labeling, label-consistent motion modeling, and multi-output GS as natural extensions. UniGS points to semantic-geometry coupling, uncertainty estimation, and instance- or panoptic-aware splats. OP2GS argues for object-level semantics without high-dimensional per-Gaussian features, while TrackRef3D shows that canonical object identities and multi-positive text supervision can be learned automatically. A plausible synthesis is that mature LabelGS systems will combine explicit object-aware Gaussian structure with unified multimodal rendering and trajectory-level or coordinate-level consistency, thereby narrowing the gap between novel-view synthesis and full 3D scene understanding (Huo et al., 2024, Xie et al., 14 Oct 2025, Liu et al., 19 May 2026, Tan et al., 26 May 2026).