3D Multi-View ImageNet (MVImgNet)

• 3D Multi-View ImageNet (MVImgNet) is a large-scale real-object multi-view dataset that captures diverse 3D generative and discriminative signals across extensive camera angles.
• The dataset employs uniform frame sampling and a refined annotation pipeline with COLMAP, PixSfM, and advanced segmentation to ensure precise 3D geometry and object consistency.
• MVImgNet drives advances in radiance-field reconstruction, multi-view stereo, and 3D point-cloud classification, providing robust benchmarks and pretraining opportunities for 3D vision tasks.

3D Multi-View ImageNet (MVImgNet) is a large-scale real-object multi-view dataset designed as a scalable 3D-aware counterpart to ImageNet for vision research. MVImgNet provides uniformly sampled, annotated multi-view imagery of hundreds of thousands of real-world objects, explicitly capturing 3D generative and discriminative signals through diverse camera perspectives. It serves both as direct training data for 3D vision tasks and as a foundation for evaluating and pretraining both single-view and multi-view models across the 2D–3D continuum, enabling advances in generalizable 3D reconstruction, multi-view consistency, and robust object understanding (Yu et al., 2023, Han et al., 2 Dec 2024).

1. Dataset Construction and Scaling

MVImgNet’s initial release encompasses 219,188 validated video sequences covering 220,000 unique objects from 238 WordNet-based object categories, collectively yielding 6.5 million frames. Objects are typically recorded in short (∼10 s) videos by crowd workers instructed to orbit a handheld phone camera at least halfway (180°), maintaining ≥15% image occupancy by the object and avoiding blurring or occlusion. Each video is labeled to a single object class, with category balance designed to at least partially overlap, but remain distinct from, the canonical ImageNet taxonomy (≈ 65 shared classes) (Yu et al., 2023).

MVImgNet2.0 expands this construction, adding ∼300,000 new objects for a total of ≈ 520,000 object instances from 515 categories. Critically, in MVImgNet2.0, nearly 77% of new sequences achieve full 360° view coverage per object, substantially addressing self-occlusion and shape completeness limitations present in the original dataset. For both releases, roughly 30 frames are uniformly subsampled from each video, ensuring broad pose sampling (Han et al., 2 Dec 2024).

All data are subject to multi-stage vetting for class consistency, blur, and framing. The datasets, with associated metadata, camera parameters, masks, and dense point clouds, are publicly distributed via https://luyues.github.io/mvimgnet2/ in a category- and instance-oriented file structure.

2. Annotation Pipeline and 3D Geometry

Each MVImgNet frame is annotated with per-pixel foreground object masks, camera extrinsics $(R_j, t_j)$, intrinsics $K_j$, and (for dense reconstructions) depth and normal maps. MVImgNet uses COLMAP for sparse Structure-from-Motion (SfM) reconstruction, estimating camera parameters via classical keypoint detection and bundle adjustment:

$x_{ij} = K_j[ R_j|t_j ] X_i$

where $X_i$ is a homogeneous 3D point in world coordinates, and $x_{ij}$ is its pixel projection in image $j$ (Yu et al., 2023). For dense geometry, COLMAP’s PatchMatch is used to extract per-pixel depths $D_j(u,v)$ and normals $N_j(u,v)$, which are then mask-filtered and back-projected for 3D fusion with normal-guided pruning and manual cleaning.

MVImgNet2.0 adopts a more refined annotation pipeline at each stage:

• Segmentation: CarveKit (in MVImgNet) is replaced by a cascade comprising Grounding-DINO (detection), SAM (segment-anything), and DeAOT (tracking), improving mask accuracy as measured by mask MSE (on 500 images: 0.243 → 0.172) and public benchmarks (ECSSD: 0.143→0.103; DAVIS: 0.195→0.143) (Han et al., 2 Dec 2024).
• SfM: COLMAP is superseded by PixSfM, incorporating a deep-feature-metric loss to reduce pose errors, especially for low-texture/specular surfaces. This improvement is validated by higher downstream reconstruction fidelity with NeRF and 3D Gaussian Splatting models (Instant-NGP PSNR improves by 1.12 dB, 3DGS by 5.75 dB).
• Dense Reconstruction: MVImgNet2.0 uses Instant-Angelo (Neuralangelo + hash grids), yielding denser and more accurate point clouds. TriplaneGaussian models trained with MV2-Anno achieve Chamfer distance (CD×10⁻²) of 0.36, strongly surpassing earlier pipelines (MV1: 0.82, Objaverse: 0.40).

3. Derived 3D Datasets: MVPNet

MVImgNet gives rise to MVPNet, a large fully annotated real-world object point-cloud dataset. MVPNet contains 87,200 manually cleaned point clouds from 150 classes, with each cloud carrying a single class label and averaging 75k points. Point clouds are constructed by fusing all per-frame depth maps, applying mask and normal consistency, and performing subsequent manual cleaning to remove outliers or background fragments.

The reconstruction formula for each pixel in frame $j$ is:

$$X = R_j^T \left( K_j^{-1}(u, v, 1)D_j(u,v) - t_j \right)$$

MVPNet is designed for 3D object classification tasks and is benchmarked under ScanObjectNN’s PB_T50_RS and random-rotation splits, serving both as a challenging pretraining and evaluation resource for 3D point-based models (Yu et al., 2023).

4. Benchmark Tasks and Applications

MVImgNet and MVPNet have been leveraged across a spectrum of supervised and self-supervised 2D/3D tasks. Comprehensive benchmarks demonstrate the value of view diversity, accurate geometry, and annotation quality:

• Radiance-Field Reconstruction (e.g., NeRF, IBRNet): Pretraining on MVImgNet followed by category/domain transfer yields ∼2 dB PSNR improvement and 0.05 LPIPS reduction.
• Multi-View Stereo: Self-supervised methods, when pretrained on 100k MVImgNet videos, achieve up to +1.4% absolute improvement in depth accuracy on DTU (at 4 mm error with 5% data).
• View-Consistent Image Classification: ResNet-50 trained on an MVI-Mix (ImageNet+MVImgNet views) achieves 24.2% higher accuracy and reduced variance under view changes compared to ImageNet-only training.
• Contrastive Learning: Fine-tuning MoCo-v2 with positive pairs from multiple views lowers softmax variance (0.098→0.086) and increases accuracy (70.3%→71.2%).
• Salient Object Detection: Introducing view-consistency loss improves IoU by ∼4.1% on challenging frames.
• 3D Point-Cloud Classification: Supervised and masked-autoencoder pretraining on MVPNet boosts downstream accuracy (CurveNet: 74.3%→83.7%, PointMAE: 77.3%→84.1%). MVPNet is objectively more difficult than ScanObjectNN (test accuracy <50% when models are not pretrained on MVPNet) (Yu et al., 2023, Han et al., 2 Dec 2024).

5. MVImgNet in Model Evaluation and Viewpoint Robustness

A key application of MVImgNet is rigorous benchmarking of foundation models’ 3D spatial understanding, as exemplified by multi-view correspondence analysis (Lilova et al., 12 Dec 2025). Specialized MVImgNet subsets enable controlled evaluation of feature consistency under explicit view shifts:

• Subset construction: 15 classes with at least one instance covering all seven angular bins (0°–90°), each bin representing a sampled camera-object rotation.
• Evaluation paradigms: Four key-query difficulty levels (easy/medium/hard/extreme) are defined by reference bin allocation and unobserved query bins.
• Segmentation protocol: Hummingbird in-context framework with frozen ViT encoders, memory-bank lookup (via FAISS, cosine metric), and cross-attention decoders produces per-pixel segmentation masks (16 classes: 15 objects + background) at 512×512 input resolution.
• Metrics: Mean IoU (mIoU) across classes, breaking-point analysis under extreme viewpoint shifts, and per-bin normalization quantify model robustness to unseen perspectives.

This protocol directly isolates the ability of vision encoders to maintain pixel-wise semantic correspondence under progressive 3D rotations, revealing that DINO-based encoders outperform certain 3D-specific models (e.g., VGGT) in large viewpoint shifts unless specifically tuned for multi-view settings (Lilova et al., 12 Dec 2025).

6. Empirical Insights, Impact, and Open Challenges

MVImgNet and MVImgNet2.0 provide pivotal empirical gains:

• Pretraining with MVImgNet consistently improves model generalization for unseen 3D domains.
• Self-supervised multi-view stereo performance benefits from expansive real-world multi-view data, reducing dependence on synthetic or RGB-D sources.
• Multi-view consistency augmentation in 2D vision (classification, representation learning, SOD) enhances robustness to viewpoint variation.
• The scale, annotation quality, and object diversity in MVImgNet2.0 (520k objects, 515 categories, 77% 360° coverage) enable reconstruction models (LGM, LRM, TriplaneGaussian) to achieve up to +1.90 dB PSNR and ×4 improvement in Chamfer distance relative to prior data (Han et al., 2 Dec 2024).

Several challenge areas remain:

• Expanding object diversity toward nature-centric and fine-grained categories.
• Capturing true scene-level contexts (background clutter, full 360° environments).
• Developing cross-modal, domain-adaptive, and distillation-based training schemes to optimally exploit “3D-awareness” signals for a broader range of vision tasks (Yu et al., 2023, Han et al., 2 Dec 2024).

7. Accessibility and Future Prospects

MVImgNet2.0 is publicly available, encompassing 520,000 multi-view sequences with per-frame camera metadata, foreground masks, and high-quality dense point clouds. Annotation code and pipelines are provided. The dataset stands as the leading real-object multi-view resource bridging the 2D–3D scale gap, supporting the development, pretraining, and benchmarking of vision models in both discriminative and generative paradigms, and is likely to drive progress in robust 3D object understanding and view-invariant modeling (Yu et al., 2023, Han et al., 2 Dec 2024, Lilova et al., 12 Dec 2025).