InsScene-15K Dataset

• InsScene-15K is a large-scale multi-modal dataset with 15,000 indoor scenes, each providing high-quality RGB, depth, camera poses, and instance segmentation masks.
• The dataset integrates synthetic, video-captured, and RGBD-scanned data, leveraging SAM and SAM2 for efficient, automated, and consistent mask annotation.
• It enables unified 3D spatial reconstruction, instance tracking, and open-vocabulary segmentation, supporting robust benchmarking for advanced scene understanding models.

InsScene-15K is a large-scale, multi-modal dataset designed to advance unified 3D spatial reconstruction and instance-level scene understanding from multi-view imagery. Developed in support of the Instance-Grounded Geometry Transformer (IGGT) framework, InsScene-15K provides 15,000 indoor scenes annotated with high-quality RGB images, camera poses, per-pixel metric depth, and 3D-consistent instance-level segmentation masks. Its construction leverages synthetic simulations, real-world video sequences, and RGBD scans, and employs recent advances in automated segmentation, notably leveraging Meta’s Segment Anything Model (SAM) and SAM2 for scalable, high-fidelity mask annotation (Li et al., 26 Oct 2025).

1. Dataset Composition and Organization

InsScene-15K comprises 15,000 indoor scenes, each captured from multiple RGB viewpoints. The exact number of images per scene varies: for model training, 1–12 views are sampled per scene; for downstream evaluation, 8–10 views are used to ensure adequate spatial coverage. Each scene is labeled with four aligned modalities:

• High-resolution RGB images (color photographs)
• Camera pose metadata (intrinsics and extrinsics)
• Per-pixel metric depth maps
• 3D-consistent instance masks, providing a unique object instance identifier (ID) per object across all views

The data release structure consists of folders per scene, each with zero-padded file naming (e.g., scene_00001/rgb_000.png, depth_000.png, pose_000.txt, mask_000.png). Typical image resolution is approximately 1000×1000 pixels, and depth ranges from 0.5 m to 10 m. The dataset focuses exclusively on indoor environments, encompassing living rooms, kitchens, offices, and similar domains. The paper does not provide explicit train/val/test splits for InsScene-15K itself; instead, the full dataset is used for IGGT training, while separate benchmarks such as ScanNet and ScanNet++ are employed for evaluation (Li et al., 26 Oct 2025).

2. Data Curation Process and Modalities

The curation of InsScene-15K involves integrating three scene sources, each processed via specific acquisition and annotation protocols:

1. Synthetic Scenes (renderings from Aria and Infinigen): Chosen for diverse and photorealistic layouts. Ground-truth masks are exported directly from the simulation, resulting in pixel-perfect 2D object segmentation without further post-processing.
2. Video-captured Scenes (RE10K dataset): Dynamic indoor scenes are acquired via video, with RGB, depth (if available), and poses obtained using SLAM/SfM backends. Instance mask generation involves initial dense proposals via SAM, temporal propagation using the SAM2 video segmenter, keyframe re-discovery when regions are uncovered, and a final bi-directional propagation pass to ensure temporal mask consistency.
3. RGBD-scanned Scenes (ScanNet++): Static scans with coarse 3D annotations. The 3D instance labels are projected to each 2D frame for initial mask proposals; SAM2 generates fine-grained 2D masks per view, which are aligned with projections by IoU matching. Mask merging ensures multi-view instance ID consistency, and the process iterates until all pixels are assigned.

Automated quality control includes visual inspections, reporting of mean Intersection over Union (mIoU) between original and refined masks (see Fig. 3 in the paper), and manual spot-checks for challenging cases. The annotation process across all real data relies on a human-in-the-loop protocol for keyframe selection, but all mask propagation and merging are automated upon keyframe identification (no full manual annotation per view) (Li et al., 26 Oct 2025).

3. Annotation Protocol and Quality Assurance

Instance mask annotation throughout InsScene-15K is driven by automated models:

• Synthetic data delivers perfect masks directly by simulator export.
• Real data (video/RGBD): Initial mask proposals per scene or frame are generated by SAM, with temporal and spatial consistency enforced by SAM2 and IoU-based matching against projected 3D annotations (for ScanNet++).

Mask quality is assessed through:

• mIoU improvements quantified relative to coarse ground-truth labels (ScanNet++ benchmark).
• Manual spot-checks targeting challenging or ambiguous frames (Appendix Fig. A.1).
• Human intervention solely for keyframe selection when automatic propagation leaves regions uncovered.

A plausible implication is that this scalable, semi-automated workflow enables efficient annotation of large datasets with reduced bias compared to manual-only protocols, while achieving high segmentation fidelity across both real and synthetic domains (Li et al., 26 Oct 2025).

4. Object and Scene Diversity

Object category coverage spans furniture, appliances, decor, and generic indoor scene elements, including but not limited to chairs, tables, sinks, and cabinets. Synthetic scenes can include additional props based on simulator resources (Aria, Infinigen). Instances per scene vary widely, observed from as few as 2–3 objects up to more than 20 in densely populated environments. Lighting variability is present across the two main domains: synthetic sources employ controlled studio lighting, while video and RGBD scans exhibit natural or fluorescent indoor illumination. The dataset is homogeneous with respect to environment type, focusing strictly on indoor scenes (Li et al., 26 Oct 2025). Quantitative object or category distributions are not specified.

5. Data Storage, Formats, and Access

File formats, though not exhaustively detailed in the source, are consistent with established conventions:

• RGB images: PNG or JPG
• Depth maps: 16-bit PNG or 32-bit EXR
• Camera poses: plain-text formats (.txt or .json), with per-view intrinsics/extrinsics
• Instance masks: 8-bit PNG, pixel values indicating instance IDs

Each scene is organized within a dedicated directory indexed by scene and view, supporting efficient programmatic access and sorting. The available code and model weights for IGGT are linked at https://github.com/lifuguan/IGGT_official. The paper does not explicitly detail dataset licensing terms or a direct download URL; prospective users are directed to the project repository or to contact the authors for potential access (Li et al., 26 Oct 2025).

6. Downstream Tasks, Benchmarks, and Loss Functions

InsScene-15K is constructed specifically to enable unified benchmarking across a variety of downstream 2D and 3D perception tasks, as detailed in IGGT experiments:

• Instance spatial tracking: Multi-view consistent segmentation of object instances.
• Open-vocabulary 2D and 3D semantic segmentation: Evaluation on ScanNet and ScanNet++.
• QA-style scene grounding: Zero-shot scene querying using large multimodal models (e.g., GPT-4o, Qwen2.5-VL, Gemini).

IGGT models trained on InsScene-15K achieve the following (benchmarked on ScanNet/ScanNet++):

Task	Metric	ScanNet	ScanNet++
Spatial Tracking	T-mIoU	69.41%	73.02%
	T-SR	≈ 98.7%	≈ 98.7%
3D Reconstruction	Abs Rel	≈ 1.90%	≈ 1.90%
	Inlier Ratio τ	≈ 83.7%	≈ 83.7%
2D Open-vocab Segmentation	mIoU, mAcc	60.5%, 81.8%	31.3%, 70.8%
3D Open-vocab Segmentation	mIoU	39.7%	20.1%

Losses for IGGT include a 3D-consistent contrastive loss on instance features, formulated as

$$L_{mvc} = \lambda_{pull} \sum_{same\,id} d(f_i, f_j) + \lambda_{push} \sum_{diff\,id} \max(0, M - d(f_i, f_j)),$$

with $d(\cdot, \cdot)$ representing L2 distance between normalized 8-D instance features, $\lambda_{pull} = 2.0$, $\lambda_{push} = 1.0$, and margin $M = 1.0$. The overall multi-task loss used for model training is given by $$L_{overall} = L_{pose} + L_{depth} + L_{pmap} + L_{mvc}$$, where geometric supervision follows the VGGT scheme from prior work (Li et al., 26 Oct 2025).

7. Significance and Research Impact

InsScene-15K introduces a unified resource for multi-view 3D scene understanding where spatial geometry and semantic instance segmentation are fully integrated at the per-pixel and per-instance level. Its coupling of automated, scalable annotation (via SAM and SAM2) with multi-modal, real and synthetic data supports robust evaluation and pretraining of transferrable scene representations. The dataset facilitates research on multi-view consistent segmentation, open-vocabulary semantic segmentation across 2D and 3D, and scene question answering, and enables strong benchmark performance for unified perception models in indoor environments (Li et al., 26 Oct 2025). A plausible implication is that its structured multi-modal coverage and annotation protocol can guide future dataset development for holistic scene understanding.