ObjectGS: Object-Aware 3D Gaussian Splatting

• ObjectGS is defined as an object-aware formulation of 3D Gaussian Splatting that decomposes scenes into distinct objects using local anchors with one-hot ID encoding.
• It introduces a novel anchor mechanism where each anchor produces multiple neural Gaussians with learnable offsets, integrating discrete semantic labels directly into the rendering process.
• The approach enables high-fidelity novel-view synthesis, interactive scene editing, and robust segmentation, addressing key challenges in occlusion and object compositionality.

Searching arXiv for the primary ObjectGS paper and closely related object-centric Gaussian Splatting works. 3D Gaussian Splatting has become a standard explicit representation for high-fidelity scene reconstruction and real-time novel-view synthesis, but its primitives are ordinarily devoid of object-level identity. ObjectGS denotes an object-aware formulation of Gaussian Splatting in which scenes are reconstructed and understood as collections of objects rather than as undifferentiated sets of primitives. In the specific formulation introduced in “ObjectGS: Object-aware Scene Reconstruction and Scene Understanding via Gaussian Splatting,” individual objects are modeled as local anchors that generate neural Gaussians and share object IDs, while a one-hot ID encoding and a classification loss impose discrete semantic structure during reconstruction (Zhu et al., 21 Jul 2025). Within the broader literature, “ObjectGS” also functions as a paradigm label for object-centric or object-compositional Gaussian Splatting, encompassing systems for open-vocabulary segmentation, compositional reconstruction, mesh extraction, editing, relighting, and physics-aware interaction (Liu et al., 2024).

1. Definition and problem setting

ObjectGS addresses a central limitation of classical 3D Gaussian Splatting: although standard 3DGS excels at rendering quality and efficiency, it lacks native object-level semantics. Prior 3DGS-based scene-understanding methods typically either decouple reconstruction and segmentation or regress continuous per-Gaussian semantic features that are alpha-blended during rendering. The former loses mutual benefits between geometry and semantics, whereas the latter conflicts with the discrete nature of semantic labels and produces ambiguity near boundaries and under compositing (Zhu et al., 21 Jul 2025).

The object-aware formulation replaces the notion of a scene as a unified whole with a decomposition into semantically meaningful object entities. In the original ObjectGS system, anchors are the minimal modeling unit. Each anchor carries a local feature, scale, $k$ learnable offsets, and a discrete object ID; all Gaussians generated by an anchor inherit that ID. This makes objects first-class entities during reconstruction rather than labels attached after the fact (Zhu et al., 21 Jul 2025).

The broader object-centric interpretation is wider than the single paper. Gaussian Object Carver states explicitly that ObjectGS refers to object-centric or object-compositional Gaussian Splatting: representing, reconstructing, and editing a scene as a set of separable objects, each parameterized by 3D Gaussians. In that view, the representation is motivated by editability, compositional flexibility, and interactive manipulation, especially in contrast to monolithic radiance fields or implicit surfaces (Liu et al., 2024).

A recurrent concern in this literature is occlusion-aware semantics. In ObjectGS, semantic labels are splatted once at scene level rather than rendered object by object, because independently supervising per-object renders suffers from occlusion errors when 2D masks miss occluded instances. This design treats semantic ambiguity as a rendering problem rather than only a recognition problem (Zhu et al., 21 Jul 2025).

2. Representation and mathematical formulation

ObjectGS uses object-aware anchors to generate Gaussian primitives. Anchor initialization proceeds from an object-labeled 3D point cloud: the labeled point cloud is voxelized, one anchor is placed at the center of each occupied voxel, and the voxel’s majority object ID is attached to the anchor. Each anchor produces $k$ Gaussians, with $k = 10$ in all experiments reported for the original method (Zhu et al., 21 Jul 2025).

For Gaussian generation, positions are defined by learnable offsets:

$$\{\mu_0,\ldots,\mu_{k-1}\} = x + \{o_0,\ldots,o_{k-1}\}\cdot l,$$

where $x$ and $l$ are the anchor center and scale. Appearance is predicted conditionally on anchor feature $f$ and view variables $(\delta, d)$ through an MLP. Semantics are discrete rather than learned as continuous embeddings: every generated Gaussian carries the anchor’s fixed one-hot ID encoding (Zhu et al., 21 Jul 2025).

The Gaussian parameterization follows the standard 3DGS form. A Gaussian $G_i$ is parameterized by mean $\mu_i \in \mathbb{R}^3$, covariance $k$0, opacity $k$1, and view-dependent color $k$2. Covariance is written as

$k$3

with rotation induced by a quaternion and diagonal scales. Projection into image space uses the Jacobian of the camera projection, yielding a screen-space covariance

$k$4

Color compositing then follows standard front-to-back alpha blending:

$k$5

These equations are retained from 3DGS, but semantics are added as parallel channels (Zhu et al., 21 Jul 2025).

The core semantic equation in ObjectGS is

$k$6

where $k$7 is the one-hot object vector for Gaussian $k$8, and $k$9 is the per-pixel class-probability vector. The predicted object ID is $k = 10$0. This replaces continuous semantic-feature blending with discrete, scene-level one-hot accumulation, which is intended to be both occlusion-aware and unambiguous (Zhu et al., 21 Jul 2025).

A useful contrast appears in OP2GS, which augments each primitive with an explicit integer instance ID $k = 10$1 and a second opacity $k = 10$2 for object-mask rendering while retaining $k = 10$3 for appearance. That work defines ObjectGS’s single semantic field as more tightly coupled to appearance and introduces a dual-opacity formulation to decouple visual existence from instance occupancy (Liu et al., 19 May 2026). This suggests that, within object-aware Gaussian representations, the design of the opacity-semantic interface is a principal axis of methodological variation.

3. Supervision, optimization, and training pipeline

ObjectGS begins with 2D-to-3D object ID lifting. Per-view object masks with consistent IDs are extracted using DEVA for tracking and Grounded-SAM for open-vocabulary, text, or click prompts. These 2D IDs are then lifted to a sparse COLMAP point cloud by majority voting, probability voting, or correspondence-based voting; majority voting is reported as the default and best overall in practice for the original method (Zhu et al., 21 Jul 2025).

Training jointly optimizes geometry, appearance, and semantics. The loss is

$k = 10$4

where the semantic term is cross-entropy on pixels with pseudo 2D labels. The reported hyperparameters are $k = 10$5, $k = 10$6 for LERF-Mask and $k = 10$7 for other datasets, and dataset-specific $k = 10$8 values. Gradients update anchor features, offsets, scales, rotations, per-Gaussian opacity, and the appearance MLP, while semantic one-hot vectors remain fixed (Zhu et al., 21 Jul 2025).

A distinctive optimization mechanism is dynamic anchor growth and pruning, inherited from Scaffold-GS in spirit. Only one anchor is allowed per voxel to preserve semantic exclusivity. Growth replicates the parent’s object ID; pruning removes anchors with weak contribution. The method characterizes growth as arising in voxels with high residual or visibility deficit and pruning as targeting anchors with persistently low opacity or gradient contribution, although exact thresholds are not specified (Zhu et al., 21 Jul 2025).

Related ObjectGS-style systems broaden the training picture. Gaussian Object Carver supplements GS optimization with monocular geometry priors and multi-view geometry regularization. Its total loss includes depth supervision $k = 10$9, normal supervision $$\{\mu_0,\ldots,\mu_{k-1}\} = x + \{o_0,\ldots,o_{k-1}\}\cdot l,$$0, photometric reprojection $$\{\mu_0,\ldots,\mu_{k-1}\} = x + \{o_0,\ldots,o_{k-1}\}\cdot l,$$1, geometric reprojection consistency $$\{\mu_0,\ldots,\mu_{k-1}\} = x + \{o_0,\ldots,o_{k-1}\}\cdot l,$$2, and segmentation loss $$\{\mu_0,\ldots,\mu_{k-1}\} = x + \{o_0,\ldots,o_{k-1}\}\cdot l,$$3, with weights $$\{\mu_0,\ldots,\mu_{k-1}\} = x + \{o_0,\ldots,o_{k-1}\}\cdot l,$$4, $$\{\mu_0,\ldots,\mu_{k-1}\} = x + \{o_0,\ldots,o_{k-1}\}\cdot l,$$5, $$\{\mu_0,\ldots,\mu_{k-1}\} = x + \{o_0,\ldots,o_{k-1}\}\cdot l,$$6, $$\{\mu_0,\ldots,\mu_{k-1}\} = x + \{o_0,\ldots,o_{k-1}\}\cdot l,$$7, and $$\{\mu_0,\ldots,\mu_{k-1}\} = x + \{o_0,\ldots,o_{k-1}\}\cdot l,$$8 (Liu et al., 2024). By contrast, OP2GS trains a two-stage pipeline in which vanilla 3DGS is first optimized to convergence of densification and opacity, after which a second opacity $$\{\mu_0,\ldots,\mu_{k-1}\} = x + \{o_0,\ldots,o_{k-1}\}\cdot l,$$9 is introduced and optimized by a random object loss with $x$0 sampled instances per step, $x$1, and stage-2 start around iteration $x$2 of a total $x$3 iterations (Liu et al., 19 May 2026).

These variants show that ObjectGS methods differ not only in semantic parameterization but also in how they allocate optimization capacity between appearance, geometry, and object occupancy. A plausible implication is that “ObjectGS” is best understood as a family of coupled rendering-segmentation objectives rather than a single canonical loss design.

4. Scene understanding and open-vocabulary behavior

The original ObjectGS targets both panoptic segmentation and open-vocabulary segmentation. Because IDs are explicit and consistent across views, 3D instance sets are obtained simply by grouping anchors or Gaussians by stored object ID. At render time, the $x$4-channel tensor $x$5 directly yields per-object masks or hard panoptic assignments via $x$6 (Zhu et al., 21 Jul 2025).

On open-vocabulary segmentation benchmarks, the reported results are strong. On LERF-Mask, ObjectGS reports mIoU/mBIoU of $x$7 on figurines, $x$8 on ramen, and $x$9 on teatime. On 3DOVS, IoU is reported as $l$0 for bed, $l$1 for bench, $l$2 for room, $l$3 for lawn, and $l$4 for sofa, for a mean of $l$5 (Zhu et al., 21 Jul 2025).

On panoptic segmentation and reconstruction, ObjectGS reports on Replica: PSNR $l$6, SSIM $l$7, LPIPS $l$8, IoU $l$9, Dice $f$0, and Acc $f$1. On ScanNet++, it reports PSNR $f$2, SSIM $f$3, LPIPS $f$4, IoU $f$5, Dice $f$6, and Acc $f$7. For 3D instance segmentation on ScanNet++, the reported Chamfer distance is $f$8, with Precision $f$9, Recall $(\delta, d)$0, and F1 $(\delta, d)$1 (Zhu et al., 21 Jul 2025).

A central ablation isolates the semantic formulation. On LERF-Mask figurines, learnable semantics yield $(\delta, d)$2 mIoU, object-independent rendering yields $(\delta, d)$3, and one-hot ID encoding yields $(\delta, d)$4. The same study reports better photometrics for one-hot IDs, with PSNR $(\delta, d)$5 versus $(\delta, d)$6–$(\delta, d)$7, associating discrete labels with both sharper boundaries and strong rendering quality (Zhu et al., 21 Jul 2025).

Later work reframes the same problem. OP2GS reports mean mIoU $(\delta, d)$8 on 3DOVS, exceeding ObjectGS’s $(\delta, d)$9, and reports LERF-Mask mIoU $G_i$0 on figurines and $G_i$1 on ramen, surpassing ObjectGS’s $G_i$2 and $G_i$3 respectively, while teatime reaches $G_i$4 (Liu et al., 19 May 2026). OP2GS attributes this to dual-opacity decoupling and object-level CLIP aggregation. This does not invalidate ObjectGS’s one-hot formulation; rather, it identifies label contamination from appearance-oriented floaters as a residual failure mode of single-mask-field approaches.

A separate line of work, “Scenes as Objects, Not Primitives: Instance-Structured 3D Tokenization from Unposed Views,” reaches AP $G_i$5, AP50 $G_i$6, and AP25 $G_i$7 on 8-view class-agnostic instance segmentation on ScanNet, while ObjectGS in that comparison attains AP $G_i$8, AP50 $G_i$9, and AP25 $\mu_i \in \mathbb{R}^3$0 (Yoo et al., 28 Jun 2026). That feed-forward tokenizer makes objects the native interface through instance-structured token groups rather than per-scene optimization. This suggests that the ObjectGS agenda has expanded from object-aware optimization to object-native scene parameterization.

5. Editing, extraction, composition, and interaction

One of the principal motivations for ObjectGS is that explicit object identity makes downstream manipulation direct. In the original method, object deletion is implemented by removing anchors with a desired ID, object transformation by applying rigid transforms to those anchors and their Gaussians, and recoloring by editing appearance MLP outputs for that subset. The same explicit grouping enables per-object mask rendering and scene editing without post hoc clustering (Zhu et al., 21 Jul 2025).

Mesh extraction is another standard downstream operation. ObjectGS uses a 2DGS variant for high-quality surfaces: keeping anchors with a target ID produces an object-only 2DGS whose rendered depth maps are fused into a TSDF, followed by Marching Cubes (Zhu et al., 21 Jul 2025). Gaussian Object Carver places stronger emphasis on this route, extracting a scene mesh from fused 3DGS depth and then producing watertight object meshes via a zero-shot Object Surface Completion model trained on approximately $\mu_i \in \mathbb{R}^3$1k meshes from ShapeNet Core v2 and Objaverse (Liu et al., 2024). In that formulation, ObjectGS becomes a pipeline from segmented Gaussians to separable, watertight object assets.

Object-centric Gaussian representations have also been adapted to composition. ComGS identifies four ObjectGS tasks: relightable object reconstruction, scene lighting estimation at the object’s placement, realistic shadows, and object insertion and composition. Its pipeline reconstructs an object as relightable Gaussian surfels with intrinsic materials, estimates a $\mu_i \in \mathbb{R}^3$2 HDR environment map at the insertion location, precomputes object-induced occlusion with Surface Octahedral Probes, and performs deferred PBR plus depth compositing. The reported runtime is approximately $\mu_i \in \mathbb{R}^3$3 FPS and editing time approximately $\mu_i \in \mathbb{R}^3$4 seconds (Gao et al., 9 Oct 2025). This shows that object identity alone is insufficient for physically coherent insertion; relightability and localized scene illumination must also be encoded.

In virtual reality, GS-Verse specializes the object-centric idea for interaction by coupling each object’s Gaussian set to an explicit surface mesh. Gaussians are anchored directly to mesh faces via barycentric coordinates and face-aligned covariances, while mesh deformations from rigid-body, mass-spring, FEM, or PBD solvers update the Gaussian parameters frame by frame (Pechko et al., 13 Oct 2025). In a comparative user study with $\mu_i \in \mathbb{R}^3$5 participants against VR-GS, GS-Verse is reported as statistically significantly better for physics-aware stretching manipulation, with Wilcoxon $\mu_i \in \mathbb{R}^3$6, $\mu_i \in \mathbb{R}^3$7, and effect size $\mu_i \in \mathbb{R}^3$8 (Pechko et al., 13 Oct 2025). Within the ObjectGS paradigm, this shifts emphasis from semantic segmentation to the “what you see is what you simulate” principle.

Object-centricity also appears in generative settings. DirectTriGS treats each object as a bounded, object-centric Gaussian asset encoded by triplanes and decoded through a differentiable TriRenderer into a surface-aligned GS. It reports CLIP scores of $\mu_i \in \mathbb{R}^3$9 and $k$00 under two evaluation settings, with inference runtime $k$01 s, $k$02M parameters, and $k$03 GB GPU memory on RTX3090 (Ju et al., 10 Mar 2025). Although this work concerns generation rather than scene understanding, it retains the ObjectGS premise that a manipulable object is the fundamental Gaussian unit.

6. Limitations, debates, and research directions

A persistent limitation across ObjectGS systems is dependence on 2D supervision quality. In the original method, severe failures in SAM, Grounded-SAM, or DEVA can propagate to 3D despite anchor evolution, and prompt-based open-vocabulary selection may occasionally group semantically similar items (Zhu et al., 21 Jul 2025). OP2GS inherits the same general dependency on pseudo-label quality, noting that poor masks can initialize incorrect labels and that extreme ambiguities may persist even when $k$04 suppresses many errors (Liu et al., 19 May 2026).

Another limitation is ambiguity around thin, overlapping, or highly interleaved structures. ObjectGS reports that one-anchor-per-voxel helps but cannot fully resolve all ambiguities (Zhu et al., 21 Jul 2025). OP2GS likewise notes residual mask leakage in thin structures, transparency, or reflection-heavy regions despite transmittance-weighted training (Liu et al., 19 May 2026). These observations indicate that object-centric GS is still constrained by the expressive limits of semi-transparent splat compositing.

Granularity is also unresolved. OP2GS states that each Gaussian carries a single label, so fine-grained part segmentation would require additional grouping or hierarchical labels (Liu et al., 19 May 2026). The feed-forward tokenization work similarly notes that heavy occlusions, near-contact objects, and a fixed maximum number of groups can impair grouping, especially in very cluttered scenes (Yoo et al., 28 Jun 2026). A plausible implication is that future ObjectGS systems will need hierarchical or adaptive object structures rather than flat instance IDs.

The relationship between semantic storage and runtime remains an active fault line. ObjectGS uses fixed one-hot IDs with a variable-length feature rasterizer that splats RGB and ID channels in one pass (Zhu et al., 21 Jul 2025). OP2GS avoids per-Gaussian feature fields and per-scene decoders by storing only an additional scalar $k$05 and an integer label, reporting $k$06 ms/frame ($k$07 FPS) at $k$08M Gaussians versus ObjectGS’s $k$09 ms/frame ($k$10 FPS) in the figurines runtime comparison (Liu et al., 19 May 2026). The tokenized feed-forward method reduces semantic storage further to approximately $k$11K scalars through instance-level embeddings and anchor residuals, compared with millions of per-primitive semantic scalars (Yoo et al., 28 Jun 2026). This makes compact object-level semantics a major trend in post-ObjectGS research.

Finally, the scope of “ObjectGS” has broadened beyond the original anchor-and-one-hot design. It now includes object-compositional reconstruction with watertight completion, local relighting and shadow-aware insertion, mesh-coupled physics interaction, object-centric generation, and feed-forward tokenization from unposed views (Liu et al., 2024). The common denominator is not a single architecture but a commitment to treating objects, rather than primitives alone, as the operative unit of 3D Gaussian representations.