InstDrive: Instance-Aware 3D Reconstruction

• InstDrive is an instance-aware 3D Gaussian Splatting framework that reconstructs dynamic driving scenes using explicit, editable 3D instance representations.
• It integrates SAM-generated masks and a two-stage training strategy combining continuous and quantized instance feature learning for robust segmentation.
• The framework outperforms traditional methods on datasets like PandaSet, handling sparse viewpoints and occlusions without complex tracking.

InstDrive is an instance-aware 3D Gaussian Splatting framework tailored for the interactive reconstruction of dynamic driving scene from dashcam videos. It augments standard 3DGS with learned instance features, uses masks generated by SAM as pseudo ground-truth to guide 2D feature learning via contrastive loss and pseudo-supervised objectives, introduces 3D regularization through a voxel-based loss, and employs a lightweight static codebook to bridge continuous features and discrete identities without data pre-processing or complex optimization (Liu et al., 16 Aug 2025). In the provided literature, the exact name InstDrive denotes this reconstruction framework; by contrast, InDRiVE denotes an intrinsic disagreement-based model-based reinforcement learning method for vehicle exploration (Khanzada et al., 7 Mar 2025), and InstaDrive denotes an instance-aware driving world model for realistic and consistent video generation (Yang et al., 3 Feb 2026).

1. Definition and nomenclature

InstDrive addresses the lack of explicit, consistent 3D instance representations in dynamic, open-world driving scenes under long sequences, sparse viewpoints, multiple moving objects, strong occlusions, and no ground-truth 3D instance labels (Liu et al., 16 Aug 2025). Its immediate target is not policy learning or video generation, but reconstructing driving scenes as sets of explicit, editable 3D instances such as vehicles, pedestrians, and road elements.

Several nearby names in the literature refer to different problems. In the InDRiVE paper, the term “InstDrive” does not appear anywhere, and the text states that “InstDrive” is almost certainly a misspelling or informal shorthand referring to InDRiVE rather than a distinct algorithm or component (Khanzada et al., 7 Mar 2025). InstaDrive, in turn, is a separate framework for controllable, instance-aware driving video generation with an Instance Flow Guider and a Spatial Geometric Aligner (Yang et al., 3 Feb 2026). A plausible implication is that searches for “InstDrive” can conflate reconstruction, world modeling, and reinforcement-learning lines of work unless the exact paper title is specified.

2. Problem setting and limitations of prior approaches

The motivating claim of InstDrive is that most 3D reconstruction methods for driving scenes either treat the scene as a single unified representation, provide semantic labels without separating instances, or rely on complex pipelines such as tracking, ID propagation, and clustering to obtain instance identities (Liu et al., 16 Aug 2025). This is presented as inadequate for autonomous driving, where object-level behavior prediction, controllable simulation, and interactive editing of specific actors and infrastructure are required.

The paper organizes prior approaches into three categories. The first is continuous feature learning with post-hoc clustering, exemplified by feature-augmented 3DGS methods in which per-Gaussian features are clustered using cosine similarity or thresholds. The stated limitations are sensitivity to hyperparameters, instability in large outdoor scenes with sparse views and occlusions, and the lack of global, consistent instance IDs. The second category is pre-processing-based ID propagation, such as pipelines combining SAM with tracking. These are described as highly dependent on tracking quality and brittle under long sequences, multi-camera setups, and viewpoint changes. The third category is codebook-based vector quantization with learned codebooks, where learned centers can be unevenly distributed, overlapping, or redundant, and scene-specific clustering introduces additional complexity and instability (Liu et al., 16 Aug 2025).

The outdoor driving setting further sharpens these limitations. The paper emphasizes sparse and limited viewpoints along a trajectory, long temporal sequences, many moving objects and strong occlusions, and large, unbounded environments. This motivates a design that avoids tracking and ID propagation, avoids scene-specific clustering or dynamic codebook optimization, works directly with SAM masks as pseudo 2D instance supervision, and scales to open-world driving data with multiple cameras and long sequences (Liu et al., 16 Aug 2025).

3. Representation, supervision, and two-stage learning

InstDrive uses a standard 3DGS scene representation in which the scene is a set of Gaussians

$\mathcal{G}=\{g_i\}_{i=1}^{N},$

with each Gaussian parameterized as

$g_i=(p_i,s_i,q_i,\alpha_i,c_i),$

where $p_i\in\mathbb{R}^3$ is 3D position, $s_i\in\mathbb{R}^3$ is scale, $q_i\in\mathbb{R}^4$ is orientation, $\alpha_i\in[0,1]$ is opacity, and $c_i$ denotes spherical-harmonic color coefficients (Liu et al., 16 Aug 2025). InstDrive augments each Gaussian with an instance feature vector $f_i\in\mathbb{R}^d$, and the paper reports that $d$ is typically $8$.

The input consists of dashcam-like videos or multi-view imagery from PandaSet, camera poses and calibration, LiDAR points, and 2D masks from SAM. For each training image, SAM produces a set of masks $g_i=(p_i,s_i,q_i,\alpha_i,c_i),$0, and these masks are treated as pseudo ground-truth instances with no semantic labels and no cross-frame ID matching (Liu et al., 16 Aug 2025).

The training strategy has two interwoven stages. The first is continuous instance feature learning, whose goal is to obtain discriminative, coherent continuous features using 2D contrastive supervision and 3D voxel-based consistency. The second is quantized instance feature learning, whose goal is to convert continuous features to discrete, globally consistent instance IDs using a static binarized codebook and instance-consistent pseudo-supervision (Liu et al., 16 Aug 2025).

The 2D supervision is defined through a unified contrastive loss. If $g_i=(p_i,s_i,q_i,\alpha_i,c_i),$1 is the rendered feature at pixel $g_i=(p_i,s_i,q_i,\alpha_i,c_i),$2 and $g_i=(p_i,s_i,q_i,\alpha_i,c_i),$3 is the prototype of mask $g_i=(p_i,s_i,q_i,\alpha_i,c_i),$4, the loss is

$g_i=(p_i,s_i,q_i,\alpha_i,c_i),$5

The first term enforces intra-mask consistency, and the second separates different masks in feature space (Liu et al., 16 Aug 2025).

Because 2D supervision alone can be satisfied by multiple Gaussians along a ray without coherent 3D embeddings, InstDrive adds a voxel-based consistency loss. After partitioning the 3D space into a regular voxel grid and computing a mean feature $g_i=(p_i,s_i,q_i,\alpha_i,c_i),$6 per voxel, the loss is

$g_i=(p_i,s_i,q_i,\alpha_i,c_i),$7

The grid is randomly shifted within one voxel length along each axis, which is stated to make the supervision robust near rigid boundaries (Liu et al., 16 Aug 2025).

4. Static codebook, discrete identities, and editable instances

A central design choice is the use of a static binarized codebook

$g_i=(p_i,s_i,q_i,\alpha_i,c_i),$8

The paper’s rationale is that for category-free instance segmentation, instance features need only be distinct identifiers rather than semantic descriptors, so the codebook can be fixed, uniformly distributed, binary, and scene-agnostic (Liu et al., 16 Aug 2025).

A continuous feature $g_i=(p_i,s_i,q_i,\alpha_i,c_i),$9 is first bounded by

$p_i\in\mathbb{R}^3$0

and is then assigned a discrete identity by nearest-neighbor quantization:

$p_i\in\mathbb{R}^3$1

This provides a discrete instance ID without clustering or learned codebook parameters (Liu et al., 16 Aug 2025).

To stabilize this mapping, InstDrive adds instance-consistent pseudo-supervision. Within each SAM mask, the rendered feature map is quantized to codebook IDs, the majority ID inside the mask is determined, and the corresponding codeword is assigned to all pixels in that mask. The pseudo-supervision loss is

$p_i\in\mathbb{R}^3$2

The overall training objective is described as a weighted sum of reconstruction, contrastive, voxel, and pseudo-supervision losses, with $p_i\in\mathbb{R}^3$3 (Liu et al., 16 Aug 2025).

The optimization details are explicit. Each scene is trained for 30,000 iterations on a single NVIDIA RTX 4090 GPU. The feature dimension is $p_i\in\mathbb{R}^3$4. The codebook module is activated from iteration 10,000 onward. The voxel loss is computed every five iterations, and the voxel grid size is 0.5 meters (Liu et al., 16 Aug 2025).

At test time, each Gaussian has a feature $p_i\in\mathbb{R}^3$5, this feature is quantized to an ID, and the ID is mapped to a unique color through a 2D–3D color map. This enables instance-colored rendering and point-and-click editing. The operations described are deletion, translation of Gaussian positions, orientation adjustment, appearance editing through spherical-harmonic colors, and opacity adjustment (Liu et al., 16 Aug 2025). A plausible implication is that InstDrive is designed not only for reconstruction quality but also for interactive scene manipulation in simulation and analysis workflows.

5. Experimental evaluation on PandaSet

InstDrive is evaluated on PandaSet, described as a large-scale, multimodal autonomous driving dataset with synchronized LiDAR and multiple cameras, 103 scenes, and diverse traffic conditions (Liu et al., 16 Aug 2025). The experiments consider 1Cam, 3Cam, and 6Cam configurations. The reported baselines are GSGroup, which uses SAM plus tracking and only reports results for 1Cam, and OpenGaussian (OpenGS), a codebook-based method with learned vectors (Liu et al., 16 Aug 2025).

The quantitative metrics are mIoU and mAcc, computed on 2D projected masks rather than full 3D metrics. The authors explicitly note that 2D metrics do not fully reflect 3D instance segmentation quality, so qualitative 3D visualizations are also emphasized (Liu et al., 16 Aug 2025).

The main numbers are as follows. For 1Cam, GSGroup reports mIoU 0.7694 and mAcc 0.8606, OpenGS reports mIoU 0.6505 and mAcc 0.8124, and InstDrive reports mIoU 0.7535 and mAcc 0.9022. For 3Cam, OpenGS reports mIoU 0.6318 and mAcc 0.7833, while InstDrive reports mIoU 0.7410 and mAcc 0.8853. For 6Cam, OpenGS reports mIoU 0.5665 and mAcc 0.7274, while InstDrive reports mIoU 0.7091 and mAcc 0.8577 (Liu et al., 16 Aug 2025).

These numbers support three claims made in the paper: InstDrive consistently outperforms OpenGS in all setups, achieves mIoU close to GSGroup but higher mAcc under 1Cam without using tracking or ID propagation, and works robustly in 3Cam and 6Cam configurations where GSGroup fails (Liu et al., 16 Aug 2025).

The ablations refine this picture. In the loss ablation, the full model reports mIoU 0.7091 and mAcc 0.8577, w/o Voxel reports mIoU 0.7117 and mAcc 0.8602, and w/o Pseudo reports mIoU 0.6724 and mAcc 0.8173. The paper interprets this as showing that removing voxel loss can slightly improve 2D metrics while degrading 3D consistency, whereas removing pseudo-supervision significantly reduces both mIoU and mAcc (Liu et al., 16 Aug 2025). In the feature-dimension ablation, $p_i\in\mathbb{R}^3$6 gives mIoU 0.6826 and mAcc 0.8448, $p_i\in\mathbb{R}^3$7 gives mIoU 0.7091 and mAcc 0.8577, and $p_i\in\mathbb{R}^3$8 gives mIoU 0.7192 and mAcc 0.8634; the chosen value $p_i\in\mathbb{R}^3$9 is presented as a good trade-off because the codebook size grows as $s_i\in\mathbb{R}^3$0 (Liu et al., 16 Aug 2025).

6. Contributions, limitations, and significance

The paper claims five principal contributions. It presents a first end-to-end framework for instance-aware 3DGS in dynamic driving scenes; introduces a two-stage training strategy combining continuous features with 2D–3D consistency and efficient instance encoding through a static binary codebook and pseudo-supervision; eliminates pre-/post-processing, clustering, and scene-specific codebook training; supports real-time interactive 3D instance editing; and demonstrates effectiveness on PandaSet through strong quantitative and qualitative results (Liu et al., 16 Aug 2025). The abstract adds that, to the best of the authors’ knowledge, it is the first framework to achieve 3D instance segmentation in dynamic, open-world driving scenes (Liu et al., 16 Aug 2025).

The limitations are also explicit. The fixed codebook capacity can be too small, causing multiple instances to share codes, or too large, causing potential over-segmentation and computational overhead. The current instance features encode identity only and not semantic class labels. Future directions named in the paper are dynamic codebook mechanisms and semantic distillation from pre-trained 2D or 3D models such as CLIP, DINO, and point-cloud methods to enable open-vocabulary querying and semantic-aware editing (Liu et al., 16 Aug 2025).

Several further challenges are identified from context rather than formal benchmarking: extremely crowded scenes or distant small objects may be difficult to separate; heavy occlusions can destabilize features; inaccurate SAM masks propagate errors into instance features; and pose errors can harm voxel assignments and 2D–3D alignment (Liu et al., 16 Aug 2025). This suggests that InstDrive’s current strengths lie in category-free instance coherence and editability rather than full semantic scene understanding.

Within the broader autonomous-driving literature, InstDrive occupies the reconstruction-and-editing end of the design space. It differs from InDRiVE, which uses ensemble disagreement in a Dreamer-based world model to learn task-agnostic latent representations for driving control (Khanzada et al., 7 Mar 2025), and from InstaDrive, which injects instance-aware temporal and geometric control into a diffusion-transformer world model for multi-view driving video synthesis (Yang et al., 3 Feb 2026). InstDrive therefore addresses a distinct problem: turning dynamic driving scenes into explicit, editable, globally instance-consistent 3D Gaussian representations.