---
title: 'InstDrive: Instance-Aware 3D Reconstruction'
url: https://www.emergentmind.com/topics/instdrive
type: topic
---

# InstDrive: Instance-Aware 3D Reconstruction

InstDrive is an instance-aware 3D Gaussian Splatting framework tailored for the interactive reconstruction of dynamic driving scene from dashcam videos. It augments standard 3DGS with learned instance features, uses masks generated by SAM as pseudo ground-truth to guide 2D feature learning via contrastive loss and pseudo-supervised objectives, introduces 3D regularization through a voxel-based loss, and employs a lightweight static codebook to bridge continuous features and discrete identities without data pre-processing or complex optimization [2508.12015]. In the provided literature, the exact name **InstDrive** denotes this reconstruction framework; by contrast, **InDRiVE** denotes an intrinsic disagreement-based model-based reinforcement learning method for vehicle exploration [2503.05573], and **InstaDrive** denotes an instance-aware driving world model for realistic and consistent video generation [2602.03242].

## 1. Definition and nomenclature

InstDrive addresses the lack of explicit, consistent 3D instance representations in dynamic, open-world driving scenes under long sequences, sparse viewpoints, multiple moving objects, strong occlusions, and no ground-truth 3D instance labels [2508.12015]. Its immediate target is not policy learning or video generation, but reconstructing driving scenes as sets of explicit, editable 3D instances such as vehicles, pedestrians, and road elements.

Several nearby names in the literature refer to different problems. In the InDRiVE paper, the term “InstDrive” does not appear anywhere, and the text states that “InstDrive” is almost certainly a misspelling or informal shorthand referring to InDRiVE rather than a distinct algorithm or component [2503.05573]. InstaDrive, in turn, is a separate framework for controllable, instance-aware driving video generation with an Instance Flow Guider and a Spatial Geometric Aligner [2602.03242]. A plausible implication is that searches for “InstDrive” can conflate reconstruction, world modeling, and reinforcement-learning lines of work unless the exact paper title is specified.

## 2. Problem setting and limitations of prior approaches

The motivating claim of InstDrive is that most 3D reconstruction methods for driving scenes either treat the scene as a single unified representation, provide semantic labels without separating instances, or rely on complex pipelines such as tracking, ID propagation, and clustering to obtain instance identities [2508.12015]. This is presented as inadequate for autonomous driving, where object-level behavior prediction, controllable simulation, and interactive editing of specific actors and infrastructure are required.

The paper organizes prior approaches into three categories. The first is continuous feature learning with post-hoc clustering, exemplified by feature-augmented 3DGS methods in which per-Gaussian features are clustered using cosine similarity or thresholds. The stated limitations are sensitivity to hyperparameters, instability in large outdoor scenes with sparse views and occlusions, and the lack of global, consistent instance IDs. The second category is pre-processing-based ID propagation, such as pipelines combining SAM with tracking. These are described as highly dependent on tracking quality and brittle under long sequences, multi-camera setups, and viewpoint changes. The third category is codebook-based vector quantization with learned codebooks, where learned centers can be unevenly distributed, overlapping, or redundant, and scene-specific clustering introduces additional complexity and instability [2508.12015].

The outdoor driving setting further sharpens these limitations. The paper emphasizes sparse and limited viewpoints along a trajectory, long temporal sequences, many moving objects and strong occlusions, and large, unbounded environments. This motivates a design that avoids tracking and ID propagation, avoids scene-specific clustering or dynamic codebook optimization, works directly with SAM masks as pseudo 2D instance supervision, and scales to open-world driving data with multiple cameras and long sequences [2508.12015].

## 3. Representation, supervision, and two-stage learning

InstDrive uses a standard 3DGS scene representation in which the scene is a set of Gaussians
$$
\mathcal{G}=\{g_i\}_{i=1}^{N},
$$
with each Gaussian parameterized as
$$
g_i=(p_i,s_i,q_i,\alpha_i,c_i),
$$
where $p_i\in\mathbb{R}^3$ is 3D position, $s_i\in\mathbb{R}^3$ is scale, $q_i\in\mathbb{R}^4$ is orientation, $\alpha_i\in[0,1]$ is opacity, and $c_i$ denotes spherical-harmonic color coefficients [2508.12015]. InstDrive augments each Gaussian with an instance feature vector $f_i\in\mathbb{R}^d$, and the paper reports that $d$ is typically $8$.

The input consists of dashcam-like videos or multi-view imagery from PandaSet, camera poses and calibration, LiDAR points, and 2D masks from SAM. For each training image, SAM produces a set of masks $\{M^1,\dots,M^N\}$, and these masks are treated as pseudo ground-truth instances with no semantic labels and no cross-frame ID matching [2508.12015].

The training strategy has two interwoven stages. The first is continuous instance feature learning, whose goal is to obtain discriminative, coherent continuous features using 2D contrastive supervision and 3D voxel-based consistency. The second is quantized instance feature learning, whose goal is to convert continuous features to discrete, globally consistent instance IDs using a static binarized codebook and instance-consistent pseudo-supervision [2508.12015].

The 2D supervision is defined through a unified contrastive loss. If $\mathbf{f}_u$ is the rendered feature at pixel $u$ and $\bar{\mathbf{f}}^i$ is the prototype of mask $M^i$, the loss is
$$
\mathcal{L}_{\text{contra}}
=
\frac{1}{N}\sum_{i=1}^{N}\frac{1}{|M^i|}\sum_{u\in M^i}\left\|\mathbf{f}_u-\bar{\mathbf{f}}^i\right\|^2
+
\frac{1}{N(N-1)}\sum_{i=1}^{N}\sum_{j=1,j\neq i}^{N}
\frac{1}{\left\|\bar{\mathbf{f}}^i-\bar{\mathbf{f}}^j\right\|^2}.
$$
The first term enforces intra-mask consistency, and the second separates different masks in feature space [2508.12015].

Because 2D supervision alone can be satisfied by multiple Gaussians along a ray without coherent 3D embeddings, InstDrive adds a voxel-based consistency loss. After partitioning the 3D space into a regular voxel grid and computing a mean feature $\bar{\mathbf{f}}_v$ per voxel, the loss is
$$
\mathcal{L}_{\text{voxel}}
=
\frac{1}{|V|}\sum_{v\in V}\frac{1}{|\mathcal{G}_v|}\sum_{g\in\mathcal{G}_v}
\left\|\mathbf{f}_g-\bar{\mathbf{f}}_v\right\|^2.
$$
The grid is randomly shifted within one voxel length along each axis, which is stated to make the supervision robust near rigid boundaries [2508.12015].

## 4. Static codebook, discrete identities, and editable instances

A central design choice is the use of a static binarized codebook
$$
\mathcal{C}=\{\mathbf{c}_1,\dots,\mathbf{c}_{2^d}\},\qquad \mathbf{c}_i\in\{-1,1\}^d.
$$
The paper’s rationale is that for category-free instance segmentation, instance features need only be distinct identifiers rather than semantic descriptors, so the codebook can be fixed, uniformly distributed, binary, and scene-agnostic [2508.12015].

A continuous feature $\mathbf{f}$ is first bounded by
$$
\hat{\mathbf{f}}=\tanh(\mathbf{f}),
$$
and is then assigned a discrete identity by nearest-neighbor quantization:
$$
\operatorname{ID}(\mathbf{f})=\arg\min_i \left\|\hat{\mathbf{f}}-\mathbf{c}_i\right\|_2^2.
$$
This provides a discrete instance ID without clustering or learned codebook parameters [2508.12015].

To stabilize this mapping, InstDrive adds instance-consistent pseudo-supervision. Within each SAM mask, the rendered feature map is quantized to codebook IDs, the majority ID inside the mask is determined, and the corresponding codeword is assigned to all pixels in that mask. The pseudo-supervision loss is
$$
\mathcal{L}_{\text{pseudo}}
=
\frac{1}{|\Omega|}\sum_{u\in\Omega}
\left\|\mathbf{f}_u-\mathbf{f}_u^{\text{pseudo}}\right\|^2.
$$
The overall training objective is described as a weighted sum of reconstruction, contrastive, voxel, and pseudo-supervision losses, with $\lambda_{\text{voxel}}=\lambda_{\text{contra}}=\lambda_{\text{pseudo}}=0.1$ [2508.12015].

The optimization details are explicit. Each scene is trained for **30,000 iterations** on a **single NVIDIA RTX 4090 GPU**. The **feature dimension** is **$d=8$**. The **codebook module** is activated from iteration **10,000** onward. The **voxel loss** is computed every **five iterations**, and the **voxel grid size** is **0.5 meters** [2508.12015].

At test time, each Gaussian has a feature $f_i$, this feature is quantized to an ID, and the ID is mapped to a unique color through a 2D–3D color map. This enables instance-colored rendering and point-and-click editing. The operations described are deletion, translation of Gaussian positions, orientation adjustment, appearance editing through spherical-harmonic colors, and opacity adjustment [2508.12015]. A plausible implication is that InstDrive is designed not only for reconstruction quality but also for interactive scene manipulation in simulation and analysis workflows.

## 5. Experimental evaluation on PandaSet

InstDrive is evaluated on PandaSet, described as a large-scale, multimodal autonomous driving dataset with synchronized LiDAR and multiple cameras, **103 scenes**, and diverse traffic conditions [2508.12015]. The experiments consider **1Cam**, **3Cam**, and **6Cam** configurations. The reported baselines are **GSGroup**, which uses SAM plus tracking and only reports results for 1Cam, and **OpenGaussian (OpenGS)**, a codebook-based method with learned vectors [2508.12015].

The quantitative metrics are **mIoU** and **mAcc**, computed on **2D projected masks** rather than full 3D metrics. The authors explicitly note that 2D metrics do not fully reflect 3D instance segmentation quality, so qualitative 3D visualizations are also emphasized [2508.12015].

The main numbers are as follows. For **1Cam**, **GSGroup** reports **mIoU 0.7694** and **mAcc 0.8606**, **OpenGS** reports **mIoU 0.6505** and **mAcc 0.8124**, and **InstDrive** reports **mIoU 0.7535** and **mAcc 0.9022**. For **3Cam**, **OpenGS** reports **mIoU 0.6318** and **mAcc 0.7833**, while **InstDrive** reports **mIoU 0.7410** and **mAcc 0.8853**. For **6Cam**, **OpenGS** reports **mIoU 0.5665** and **mAcc 0.7274**, while **InstDrive** reports **mIoU 0.7091** and **mAcc 0.8577** [2508.12015].

These numbers support three claims made in the paper: InstDrive consistently outperforms OpenGS in all setups, achieves mIoU close to GSGroup but higher mAcc under 1Cam without using tracking or ID propagation, and works robustly in 3Cam and 6Cam configurations where GSGroup fails [2508.12015].

The ablations refine this picture. In the loss ablation, the **full** model reports **mIoU 0.7091** and **mAcc 0.8577**, **w/o Voxel** reports **mIoU 0.7117** and **mAcc 0.8602**, and **w/o Pseudo** reports **mIoU 0.6724** and **mAcc 0.8173**. The paper interprets this as showing that removing voxel loss can slightly improve 2D metrics while degrading 3D consistency, whereas removing pseudo-supervision significantly reduces both mIoU and mAcc [2508.12015]. In the feature-dimension ablation, **$d=6$** gives **mIoU 0.6826** and **mAcc 0.8448**, **$d=8$** gives **mIoU 0.7091** and **mAcc 0.8577**, and **$d=10$** gives **mIoU 0.7192** and **mAcc 0.8634**; the chosen value **$d=8$** is presented as a good trade-off because the codebook size grows as $2^d$ [2508.12015].

## 6. Contributions, limitations, and significance

The paper claims five principal contributions. It presents a first end-to-end framework for instance-aware 3DGS in dynamic driving scenes; introduces a two-stage training strategy combining continuous features with 2D–3D consistency and efficient instance encoding through a static binary codebook and pseudo-supervision; eliminates pre-/post-processing, clustering, and scene-specific codebook training; supports real-time interactive 3D instance editing; and demonstrates effectiveness on PandaSet through strong quantitative and qualitative results [2508.12015]. The abstract adds that, to the best of the authors’ knowledge, it is the first framework to achieve 3D instance segmentation in dynamic, open-world driving scenes [2508.12015].

The limitations are also explicit. The fixed codebook capacity can be too small, causing multiple instances to share codes, or too large, causing potential over-segmentation and computational overhead. The current instance features encode identity only and not semantic class labels. Future directions named in the paper are dynamic codebook mechanisms and semantic distillation from pre-trained 2D or 3D models such as CLIP, DINO, and point-cloud methods to enable open-vocabulary querying and semantic-aware editing [2508.12015].

Several further challenges are identified from context rather than formal benchmarking: extremely crowded scenes or distant small objects may be difficult to separate; heavy occlusions can destabilize features; inaccurate SAM masks propagate errors into instance features; and pose errors can harm voxel assignments and 2D–3D alignment [2508.12015]. This suggests that InstDrive’s current strengths lie in category-free instance coherence and editability rather than full semantic scene understanding.

Within the broader autonomous-driving literature, InstDrive occupies the reconstruction-and-editing end of the design space. It differs from InDRiVE, which uses ensemble disagreement in a Dreamer-based world model to learn task-agnostic latent representations for driving control [2503.05573], and from InstaDrive, which injects instance-aware temporal and geometric control into a diffusion-transformer world model for multi-view driving video synthesis [2602.03242]. InstDrive therefore addresses a distinct problem: turning dynamic driving scenes into explicit, editable, globally instance-consistent 3D Gaussian representations.

Source: https://www.emergentmind.com/topics/instdrive