Instance-Grounded Geometry Transformer (IGGT)

• The paper introduces IGGT, an end-to-end transformer that jointly learns geometric structure and instance-level semantics for coherent 3D scene reconstruction.
• It employs a 3D-consistent contrastive learning framework and windowed cross-modal fusion to align multi-view features and cluster object instances via unsupervised tokenization.
• Experimental results on the InsScene-15K dataset demonstrate state-of-the-art performance in depth reconstruction, instance tracking, and segmentation, while also noting scalability challenges.

The Instance-Grounded Geometry Transformer (IGGT) is an end-to-end large unified transformer designed for semantic 3D reconstruction by integrating geometric structure prediction with instance-level contextual understanding. IGGT introduces a 3D-consistent contrastive learning framework to encode joint representations of geometry and object instances directly from 2D visual inputs. This unlocks consistent lifting of multi-view 2D images into a coherent 3D scene containing explicitly distinct object instances, demonstrated on the new InsScene-15K dataset comprising high-quality RGB images, calibrated poses, depth maps, and cross-view-consistent instance masks (Li et al., 26 Oct 2025).

1. Architecture

IGGT consists of three principal components: a transformer backbone, geometry and instance heads, and an unsupervised instance-level tokenization mechanism.

• Backbone: Given $N$ RGB input images $$\{I_i \in \mathbb{R}^{H \times W \times 3} \}_{i=1}^N$$, tokens are extracted using a pretrained DINOv2 ViT (patch size $P$), yielding per-view patch tokens $T_i^0 \in \mathbb{R}^{M \times D}$ with $M = (H \cdot W)/P^2$. A learnable "camera token" $c_i \in \mathbb{R}^D$ prepended to each sequence encodes view identity. Sine-cosine 2D positional encodings are added for patch tokens; camera tokens have dedicated positional embeddings.
• Attention Mechanisms: Stacked $L=24$ transformer blocks alternate between (a) per-view intra-view multi-head self-attention on $T_i$ and (b) global cross-view attention over concatenated tokens $[T_1;\ldots;T_N]$, enabling both local and global context aggregation.
• Geometry Head ($\Phi_{pt}$): A DPT-style decoder upsamples tokens into multi-scale geometric features $$\{I_i \in \mathbb{R}^{H \times W \times 3} \}_{i=1}^N$$0. Subheads regress:
  1. Camera parameters $$\{I_i \in \mathbb{R}^{H \times W \times 3} \}_{i=1}^N$$1 (extrinsics and intrinsics) from the camera token.
  2. Depth map $$\{I_i \in \mathbb{R}^{H \times W \times 3} \}_{i=1}^N$$2.
  3. Point map $$\{I_i \in \mathbb{R}^{H \times W \times 3} \}_{i=1}^N$$3.

• Instance Head ($$\{I_i \in \mathbb{R}^{H \times W \times 3} \}_{i=1}^N$$4): Parallels $$\{I_i \in \mathbb{R}^{H \times W \times 3} \}_{i=1}^N$$5 to generate multi-scale instance features $$\{I_i \in \mathbb{R}^{H \times W \times 3} \}_{i=1}^N$$6. Cross-modal fusion is realized via windowed attention—given $$\{I_i \in \mathbb{R}^{H \times W \times 3} \}_{i=1}^N$$7, $$\{I_i \in \mathbb{R}^{H \times W \times 3} \}_{i=1}^N$$8, the window attention operator $$\{I_i \in \mathbb{R}^{H \times W \times 3} \}_{i=1}^N$$9 (operating locally within spatial windows) fuses geometric and instance features efficiently.

• Instance Tokenization: Rather than fixed class queries, IGGT learns dense instance descriptors $P$0 per pixel, supervised via a multi-view contrastive loss to enforce clustering by instance ID.

2. 3D-Consistent Contrastive Learning

Central to IGGT is a contrastive objective that aligns pixel descriptors within the same object instance across views while separating those of different instances. For sampled pixel set $P$1, descriptors $P$2, and instance labels $P$3, the loss is:

$P$4

where $P$5, with default $P$6, $P$7, and push margin $P$8.

The full training loss aggregates contrastive terms with auxiliary supervision for geometry:

$P$9

where $T_i^0 \in \mathbb{R}^{M \times D}$0 (camera: L1), $T_i^0 \in \mathbb{R}^{M \times D}$1 (depth: L1/Berhu), and $T_i^0 \in \mathbb{R}^{M \times D}$2 (points: L1) supervise respective regression heads.

3. Instance-Clustering and Mask Generation

At inference, all per-pixel instance descriptors $T_i^0 \in \mathbb{R}^{M \times D}$3 across views are clustered using HDBSCAN, a hierarchical density-based algorithm, in $T_i^0 \in \mathbb{R}^{M \times D}$4. Each resulting cluster $T_i^0 \in \mathbb{R}^{M \times D}$5 corresponds to one explicit object instance. Reprojecting clusters to each image yields view-consistent 2D instance masks $T_i^0 \in \mathbb{R}^{M \times D}$6, maintaining consistent IDs across multi-view reconstructions. The contrastive loss margin $T_i^0 \in \mathbb{R}^{M \times D}$7 and HDBSCAN’s density separation guarantee reliable inter-object distinction in descriptor space.

4. InsScene-15K Dataset and Data Pipeline

InsScene-15K is a large-scale dataset curated for training and evaluating IGGT, supporting fine-grained 3D-consistent instance mask prediction.

• Sources: Includes synthetic data (Aria, Infinigen; perfect ground truth), video-captured scenes (RE10K), and RGBD scans (ScanNet++). The synthetic pipeline directly renders RGB, depth, pose, and object masks. For video data, a two-stage approach involving SAM1 mask proposals and SAM2 video mask propagation is used, iterating over keyframes for complete coverage. RGBD scan curation fuses coarse 3D GT projection, fine SAM2 proposals, and ID-alignment via IoU matching.
• Statistics: 15,000 indoor scenes; 10–12 views and 8–10 object instances per scene, yielding approximately 150,000 RGB frames with dense geometric and instance annotations.

5. Experimental Results and Ablation Analyses

On standard benchmarks (ScanNet, ScanNet++), IGGT demonstrates state-of-the-art performance for instance tracking, depth reconstruction, and 2D/3D open-vocabulary segmentation. The following table summarizes ScanNet and ScanNet++ results (columns: T-mIoU, T-SR, Abs Rel, $T_i^0 \in \mathbb{R}^{M \times D}$8(1.03), 2D mIoU, 2D mAcc, 3D mIoU):

Model	T-mIoU (%)	T-SR (%)	Abs. Rel	τ(1.03)	2D mIoU (%)	2D mAcc (%)	3D mIoU (%)
Feature-3DGS	—	—	6.48	41.63	57.69	63.26	23.42
LSM (multi)	—	—	3.17	64.81	53.40	59.50	35.37
VGGT	—	—	1.84	83.60	—	—	—
IGGT (Ours)	69.41	98.66	1.90	83.71	60.46	81.84	39.68

Ablations indicate that windowed cross-modal fusion significantly accelerates instance separation, and removal of the contrastive term causes $T_i^0 \in \mathbb{R}^{M \times D}$9 drop in 2D mIoU and loss of cross-view consistency ($M = (H \cdot W)/P^2$080\%). IGGT also demonstrates plug-and-play integration with different VLMs (e.g., LSeg, CLIP, OpenSeg) for varying trade-offs between global context and concept alignment.

Qualitative analysis shows robust instance tracking across viewpoint changes, with distinct clustering of 8D descriptors visible in PCA projections. 3D point cloud segmentations exhibit clean object boundaries, and instance masks allow downstream applications such as complex scene queries with LLMs (e.g., Gemini 2.5 Pro).

6. Training Protocol and Implementation

IGGT is initialized from a VGGT (1B parameter) checkpoint. Each batch comprises 24 images (2 scenes × 12 frames), resized to $M = (H \cdot W)/P^2$1, with data augmentation including random horizontal flips and slight color jitter. Training is conducted for approximately two days on 8 NVIDIA A800 GPUs using the AdamW optimizer (weight decay $M = (H \cdot W)/P^2$2), with learning rates of $M = (H \cdot W)/P^2$3 (backbone) and $M = (H \cdot W)/P^2$4 (geometry/instance heads). The total model capacity is $M = (H \cdot W)/P^2$51.05B parameters, requiring approximately 210 GFLOPs per forward pass.

7. Limitations and Prospects

Instance mask boundary precision in IGGT is constrained by the unsupervised clustering paradigm and does not attain the granularity of specialized 2D segmentation architectures (e.g., SAM2). The instance head architecture currently relies on DPT decoders; future advances may leverage DETR-style instance decoding for direct mask prediction. Dataset bias towards indoor, rigid scenes limits applicability to dynamic, outdoor, or deformable objects. The scale and computational demands (1B+ parameters, multi-GPU for training) limit real-time deployment; research into model compression and lightweight variants remains open. IGGT is the first unified transformer architecture to jointly learn geometry and instance-wise semantics, establishing a new standard in multi-view 3D understanding via the InsScene-15K dataset and a novel 3D-consistent contrastive loss (Li et al., 26 Oct 2025).