InstaScene: Instance-Aware 3D Reconstruction

• InstaScene is a framework for instance-aware 3D scene reconstruction that decomposes cluttered scenes using 2D Gaussian splatting and class-agnostic masks.
• It employs spatial contrastive learning and cross-view consistency to accurately segment and label instances despite occlusions and noise.
• The framework completes occluded regions through in-situ diffusion-based generative methods, ensuring reconstructed objects remain aligned with the original scene.

InstaScene is a framework for instance-aware 3D scene reconstruction in cluttered environments. Built on top of 2D Gaussian Splatting, it targets two coupled objectives: precise instance decomposition in scenes with adjacent, repeated, or heavily occluded objects, and complete reconstruction of each decomposed instance, including unseen regions, while remaining consistent with the original scene in scale, pose, visible appearance, and world alignment (Yang et al., 11 Jul 2025). The framework combines Gaussian-based scene reconstruction, Gaussian-level instance feature learning, and diffusion-based in-situ generation, and is positioned as a step from scene-as-a-whole reconstruction toward object-centric 3D perception.

1. Problem setting and output

InstaScene assumes a static scene observed through posed RGB images with known camera intrinsics and extrinsics. From these images, it first reconstructs the scene with 2D Gaussian Splatting. During decomposition it additionally uses class-agnostic 2D instance masks from EntitySeg, and during completion it uses the decomposed instance-specific Gaussians and their renderings (Yang et al., 11 Jul 2025).

The output has two levels. First, InstaScene assigns an instance label to each Gaussian, or equivalently partitions the Gaussian set into instances, enabling instance masks to be rendered from arbitrary views. Second, for each decomposed instance it produces a standalone 2DGS model that matches the visible geometry and appearance in the original scene, completes occluded or unseen parts using a generative prior, and remains in the correct metric scale and pose so that it can be placed back in situ.

The formulation is explicitly open-set and category-agnostic. The target is not closed-vocabulary semantic labeling but decomposition of arbitrary instances under clutter and occlusion. The method also assumes sufficient image coverage to reconstruct a scene-level 2DGS, even though individual objects may be heavily incomplete.

2. Gaussian representation and instance decomposition pipeline

InstaScene uses 2D Gaussian Splatting as its core scene representation. Each Gaussian $k$ is parameterized by a center $\mathbf{p}_k \in \mathbb{R}^3$, tangent-plane basis vectors $\mathbf{t}_u,\mathbf{t}_v \in \mathbb{R}^3$, tangent scales $\mathbf{s}_u,\mathbf{s}_v$, opacity $\alpha_k$, and view-dependent color $\mathbf{c}_k$ parameterized by spherical harmonics. Its local surface parameterization is

$$P(u,v) = \mathbf{p}_k + \mathbf{s}_u \mathbf{t}_u u + \mathbf{s}_v \mathbf{t}_v v,$$

with 2D Gaussian density

$$\mathcal{G}(\mathbf{u}) = \exp\left(-\frac{u^2+v^2}{2}\right).$$

A rendered pixel color is obtained by depth-ordered alpha blending of these projected Gaussians. InstaScene augments each Gaussian with a learnable feature vector $\mathbf{f}_i^{3d} \in \mathbb{R}^{16}$, while freezing all other Gaussian attributes during feature learning. Per-pixel feature rendering follows the same compositing structure as color rendering (Yang et al., 11 Jul 2025).

The decomposition pipeline begins from 2D instance masks. For each mask $m_{i,j}$ in image $\mathbf{p}_k \in \mathbb{R}^3$0, InstaScene constructs a Gaussian tracker $\mathbf{p}_k \in \mathbb{R}^3$1: the set of Gaussians that significantly contribute to the rasterization of that mask, using transmittance $\mathbf{p}_k \in \mathbb{R}^3$2. These trackers lift 2D mask evidence into 3D.

Cross-view association is then defined by a view-consensus criterion. A tracker is considered visible in a frame if at least $\mathbf{p}_k \in \mathbb{R}^3$3 of its points contribute to that frame’s rasterization, and contained in another tracker if at least $\mathbf{p}_k \in \mathbb{R}^3$4 of its points appear in that tracker. The view-consensus rate between trackers $\mathbf{p}_k \in \mathbb{R}^3$5 and $\mathbf{p}_k \in \mathbb{R}^3$6 is

$\mathbf{p}_k \in \mathbb{R}^3$7

If $\mathbf{p}_k \in \mathbb{R}^3$8, the two masks are considered to belong to the same instance. Clustering over all such matches yields cross-view 2D mask clusters $\mathbf{p}_k \in \mathbb{R}^3$9.

Under-segmented masks are filtered before 3D consolidation. A mask is treated as under-segmentation when its tracker intersects multiple trackers in the same frame and remains consistently present in their visible frames. After that, all tracker points in a cluster are merged and DBSCAN is used to remove floaters, producing a sparse but reliable 3D instance mask $\mathbf{t}_u,\mathbf{t}_v \in \mathbb{R}^3$0.

3. Spatial contrastive learning

The central decomposition mechanism is spatial contrastive learning, introduced to solve a specific failure mode: 2D masks alone are noisy and cross-view inconsistent, while 3D mask clusters alone are sparse and discard valid Gaussians. InstaScene therefore uses mutual guidance between 2D and 3D supervision (Yang et al., 11 Jul 2025).

Let $\mathbf{t}_u,\mathbf{t}_v \in \mathbb{R}^3$1 denote labeled features, where $\mathbf{t}_u,\mathbf{t}_v \in \mathbb{R}^3$2 indexes instances and $\mathbf{t}_u,\mathbf{t}_v \in \mathbb{R}^3$3 samples within an instance. For each instance $\mathbf{t}_u,\mathbf{t}_v \in \mathbb{R}^3$4, define the prototype $\mathbf{t}_u,\mathbf{t}_v \in \mathbb{R}^3$5 as the mean of its features, and let $\mathbf{t}_u,\mathbf{t}_v \in \mathbb{R}^3$6 be an instance-specific temperature. The contrastive objective is

$\mathbf{t}_u,\mathbf{t}_v \in \mathbb{R}^3$7

This is applied at three levels. The first term uses rendered 2D features in a single view, enforcing intra-view separability. The second uses neighboring views, enforcing cross-view consistency. The third uses Gaussian features associated with reliable 3D masks, anchoring the representation to cleaner 3D structure. The full feature loss is

$\mathbf{t}_u,\mathbf{t}_v \in \mathbb{R}^3$8

The resulting feature field is dense and instance-distinguishable. After training, InstaScene computes a coarse 3D prototype $\mathbf{t}_u,\mathbf{t}_v \in \mathbb{R}^3$9 for each instance from the Gaussians in $\mathbf{s}_u,\mathbf{s}_v$0. Each Gaussian is then assigned to the instance with highest cosine similarity if that similarity exceeds the segmentation threshold $\mathbf{s}_u,\mathbf{s}_v$1. A plausible implication is that the method treats instance decomposition not as direct clustering in geometry alone, but as feature learning over a rasterization-aware Gaussian scene.

4. In-situ generation and complete instance reconstruction

Once an object instance has been segmented, its Gaussians define an instance-level 2DGS model. This representation is faithful to the observed scene but incomplete in occluded or unseen regions. InstaScene addresses this with in-situ generation, which uses a 3D diffusion-based generative model similar to MVD-Fusion, conditioned on rendered views, depth, and geometry-aware warped latent features (Yang et al., 11 Jul 2025).

For an instance $\mathbf{s}_u,\mathbf{s}_v$2, InstaScene selects conditioning views $\mathbf{s}_u,\mathbf{s}_v$3 rendered from camera poses $\mathbf{s}_u,\mathbf{s}_v$4 where the object is least occluded, along with corresponding depth maps $\mathbf{s}_u,\mathbf{s}_v$5. It then targets unseen views $\mathbf{s}_u,\mathbf{s}_v$6 at camera poses $\mathbf{s}_u,\mathbf{s}_v$7, with relative poses $\mathbf{s}_u,\mathbf{s}_v$8 from each conditioning view to each target view. The generative objective is to model

$\mathbf{s}_u,\mathbf{s}_v$9

A distinctive component is alternated view conditioning. Instead of conditioning diffusion on one image, InstaScene alternates the conditioning view over timesteps and averages the predicted noise across all conditioning views:

$\alpha_k$0

This multi-view conditioning is designed to impose consistency across generated target views.

The second component is geometry-aware feature warping. Using depth maps, the method back-projects latent features from source views into 3D, re-projects them into target views, and filters back-facing points using surface normals from the fused 2DGS mesh. Visible regions in target views therefore receive warped latent features derived from real observations, while invisible regions remain denoised from noise. This tightens alignment between generative completion and scene geometry, reducing domain-gap-induced drift.

Viewpoint selection is also explicit. Around each instance center, InstaScene samples 16 canonical viewpoints, evaluates their occlusion using the full scene’s 2DGS, selects the least occluded viewpoints as conditioning views, and treats the rest as target views to be generated. Background pixels in the conditioning views are removed by rendering the instance masks from the learned feature field.

After generation, the instance-level 2DGS is jointly refined using both original source observations and generated supplementary views. The known views preserve consistency with the observed object, while the generated views supervise previously missing surfaces. The completed instance remains scene-aligned because it inherits the original scene’s pose and scale.

5. Empirical performance

InstaScene is evaluated on both decomposition and completion. For decomposition, the main quantitative benchmark is LERF-Mask, with mean Intersection-over-Union over predicted instance masks. For completion, Replica-CAD provides ground-truth CAD geometry, enabling both rendering and geometry evaluation; real ZipNeRF scenes are used qualitatively (Yang et al., 11 Jul 2025).

Before summarizing the implications, the key quantitative results are:

Evaluation	Baseline range	InstaScene
LERF-Mask average mIoU	58.1–60.6	85.6
Replica-CAD PSNR (known / unknown)	17.19/17.46 to 25.09/23.03	32.57 / 29.02
Replica-CAD geometry (CD / F1 / Vol IoU)	0.037–0.081 / 0.150–0.406 / 0.531–0.590	0.016 / 0.767 / 0.716

On LERF-Mask, InstaScene reports $\alpha_k$1 for Figurines, $\alpha_k$2 for Teatime, $\alpha_k$3 for Kitchen, and $\alpha_k$4 average, compared with LangSplat at $\alpha_k$5 average and GSGrouping at $\alpha_k$6 average. The strongest relative gain appears in Kitchen, a cluttered scene where repeated and adjacent objects are particularly difficult.

The ablation study on decomposition clarifies the role of supervision. Using noisy 2D masks alone yields $\alpha_k$7 average mIoU; using only 3D masks yields $\alpha_k$8; adding filtered 2D intra-view masks gives $\alpha_k$9; and the full method with cross-view 2D masks reaches $\mathbf{c}_k$0. This supports the claim that neither 2D masks nor 3D masks alone are sufficient, and that mutual guidance is central to the method.

For complete reconstruction on Replica-CAD, InstaScene is compared with MVD-Fusion, InstantMesh, and SparP. It reports PSNR $\mathbf{c}_k$1, SSIM $\mathbf{c}_k$2, and LPIPS $\mathbf{c}_k$3 on known and unknown views respectively. The original scene 2DGS reconstruction reports PSNR $\mathbf{c}_k$4, SSIM $\mathbf{c}_k$5, and LPIPS $\mathbf{c}_k$6, so InstaScene slightly surpasses the original reconstruction on known views and improves unknown-view rendering. On geometry, it reports Chamfer Distance $\mathbf{c}_k$7, F1 $\mathbf{c}_k$8, and Volume IoU $\mathbf{c}_k$9, outperforming MVD-Fusion, InstantMesh, and SparP.

Qualitatively, the paper attributes baseline failures to different causes. GSGrouping is described as vulnerable to mask noise and tracking failures; LangSplat struggles with repeated objects because similar objects obtain similar language-grounded features; single-view generative baselines miss unseen backsides or distort object geometry; and generic multi-view methods can become misaligned under domain gap. By contrast, InstaScene’s completions are presented as scale-consistent, visually intact, and suitable for re-insertion and manipulation.

6. Broader context, interpretation, and limitations

Within the specific 2025 formulation, InstaScene is an object-centric Gaussian framework for decomposing cluttered scenes into complete, reusable instances. A broader reading of adjacent work suggests that the term also participates in a larger family of instant or sparse-input scene systems, though those systems solve different problems. Single-panorama reconstruction for robotic manipulation backgrounds uses cube-map decomposition, panoramic depth fusion, and training-free depth injection to produce a 3D Gaussian scene in seconds (Li et al., 8 Apr 2026). Feed-forward 3D editing from sparse unposed images uses a pose-free large reconstruction model and asymmetric edited/raw inputs to produce instruction-aligned Gaussian scenes in a single forward pass (Liu et al., 31 Dec 2025). Reference-based 3D-scene inpainting from a 2D proposal reaches $$P(u,v) = \mathbf{p}_k + \mathbf{s}_u \mathbf{t}_u u + \mathbf{s}_v \mathbf{t}_v v,$$0 seconds with a masked large reconstruction model (You et al., 12 Jun 2025). Native 4D Gaussian reconstruction from casual monocular video reconstructs a single video within 10 minutes and renders at hundreds of FPS (Luo et al., 1 Oct 2025). Instant stylization for radiance fields uses dual content/style branches with feature-space AdaIN and produces stylized novel views in less than 10 minutes (Li et al., 2023). An earlier line of work on incremental scene synthesis models a learned spatial memory that can incorporate observations, hallucinate unobserved regions, and converges to solving SLAM in the limit of observing real data at each point (Planche et al., 2018). This suggests that “InstaScene” can denote not only the specific framework of (Yang et al., 11 Jul 2025), but also an emerging design pattern centered on fast scene construction, editing, or completion from limited evidence.

The framework nevertheless has explicit limits. It assumes static scenes and accurate camera poses. It does not handle dynamic, non-rigid, transparent, or highly reflective objects. It depends on the quality of class-agnostic 2D masks from EntitySeg, so severe under-segmentation or poor masks can propagate into clustering and feature learning. It is also a multi-stage pipeline: scene reconstruction, tracker construction, feature learning, diffusion-based generation, and per-instance 2DGS refinement. These constraints mark a boundary between object-centric completion in static clutter and more general scene understanding.

A common misconception would be to treat InstaScene as generic image-to-3D generation applied to objects cropped from scenes. The method is not formulated that way. Its completion stage is explicitly in situ: conditioning views are selected by occlusion, depth and normals are used for feature warping, and the completed instance is refined in the coordinate frame of the original scene. Another misconception would be to treat decomposition as language-grounded open-vocabulary segmentation. In fact, the method uses class-agnostic masks, Gaussian trackers, and spatial contrastive learning, with instance identity arising from geometric-rasterization consistency rather than a vocabulary.

In that sense, InstaScene occupies a distinct position in 3D vision. It does not merely reconstruct a scene, and it does not merely generate isolated objects. It decomposes arbitrary instances from cluttered real or synthetic scenes and reconstructs them as complete 3D assets that remain physically aligned with the source environment (Yang et al., 11 Jul 2025).