MVImgNet Dataset for 3D Vision

• MVImgNet is a large-scale dataset of multi-view image sequences captured in real-world settings, featuring 6.5 million frames with detailed annotations such as object masks, camera parameters, and depth maps.
• It enables advances in neural rendering and multi-view stereo by providing robust geometric signals and enforcing view-consistent representations, thereby improving metrics like PSNR, SSIM, and classification accuracy.
• The dataset serves as a foundational resource akin to ImageNet for 2D vision, catalyzing research in 3D reconstruction, universal feature learning, and realistic point cloud processing.

MVImgNet is a large-scale dataset of multi-view images collected "in the wild" by video recording real-world objects. It serves as a soft bridge between 2D image understanding and 3D vision by embedding natural multi-view signals, enabling both 3D reconstruction tasks and the enforcement of view-consistent image representations. MVImgNet provides foundational resources for pretraining and benchmarking in 3D vision, similarly to the role played by ImageNet in 2D computer vision.

1. Composition and Data Acquisition

MVImgNet comprises 6.5 million frames from 219,188 videos spanning 238 object classes, with a human-centric bias and 65 classes overlapping ImageNet. Each video typically covers a broad range of viewpoints (e.g., 180° or 360°) of the target object, inherently encoding multi-view consistency. The dataset's acquisition protocol is highly scalable: objects are captured in routine, unconstrained settings using consumer-grade devices, ensuring representative variability and diversity.

Annotations are systematically extracted for each sample:

• Object masks, using automatic foreground segmentation.
• Camera parameters—both intrinsic and extrinsic—determined by Structure-from-Motion (SfM) via COLMAP.
• Depth maps, generated with multi-view stereo methods and integrated into sparse point clouds.

The multi-view attribute provides robust geometric signals, vital for 3D-aware representation learning and cross-view consistency.

2. Annotation Types and Technical Structure

MVImgNet's technical annotation suite includes discrete mask extraction, precise camera calibration, and point cloud fusion via depth aggregation. Camera intrinsics (e.g., focal length, principal point) and extrinsics (rotation and translation matrices) derived from SfM enable accurate multi-view geometry. Depth maps are computed per frame and fused for global point cloud estimation. This annotation structure supports scene-level geometric reasoning and is compatible with modern pipeline architectures for 3D reconstruction.

Multi-view videos serve as the raw substrate for advanced 3D tasks:

• Object masks facilitate segmentation and foreground extraction.
• Camera parameter and pose annotations underpin geometry-aware learning algorithms.
• Point clouds, reconstructed by fusing multi-view depth, are essential for evaluating 3D model completeness, noise resilience, and partial visibility.

3. Applications in 2D and 3D Visual Tasks

MVImgNet, by virtue of its multi-view signals and scale, serves as a pretraining or evaluation source for diverse tasks:

A. Neural Radiance Field (NeRF) Reconstruction

Pretraining generalizable NeRF architectures (e.g., IBRNet) on MVImgNet yields improvements in rendering metrics (PSNR, SSIM, LPIPS) and qualitative fidelity for novel view synthesis compared to other datasets such as CO3D.

B. Multi-View Stereo (MVS)

Self-supervised MVS models, such as JDACS, benefit from MVImgNet's geometric richness—showing significant gains on standard DTU benchmarks (using thresholded depth accuracy metrics) in low-data regimes.

C. View-Consistent Image Understanding

Hybrid image classification experiments (using sets like “MVI-Mix” containing ImageNet and MVImgNet samples and models such as ResNet-50, DeiT-Tiny) reveal lower softmax confidence variance across views and improved overall accuracy. MoCo-v2 contrastive learning profits from multi-view positives derived from the same video, resulting in better view-consistency. Salient object detection (U2Net) models finetuned with an added view-consistency loss: $$\text{Loss}_{\text{OF}} = \mathcal{M}(f_i) - \mathcal{M}(f_{i-1}) \cdot \mathcal{F}(f_i)$$ and the weighted combination: $$\text{Loss} = \tau \cdot \text{Loss}_{\text{OF}} + (1 - \tau) \cdot \text{Loss}_{\text{SOD}}$$ exhibit increased robustness for challenging views.

4. The Derived MVPNet Dataset

A large-scale, real-world point cloud dataset, MVPNet, is constructed by performing dense multi-view 3D reconstruction on MVImgNet videos. MVPNet comprises 87,200 object point clouds across 150 categories, each sample annotated with a class label. The typical imperfections—noise, incomplete views, texture variation—reflect real-world capture conditions.

Experiments in 3D object classification (e.g., ScanObjectNN benchmark) indicate that models pretrained on MVPNet—both supervised (PointNet, PointNet++, CurveNet) and self-supervised (PointMAE)—achieve higher accuracy and robustness relative to other sources. MVPNet thus provides a challenging benchmark that spurs advances in point cloud processing algorithms.

5. Impact on 3D Vision Benchmarks and Universal Representation Learning

MVImgNet is positioned as the 3D counterpart to ImageNet, with potential to standardize pretraining protocols for 3D vision algorithms and bridge the gap between synthetic and real-world datasets. The dataset facilitates research into multi-view consistency for reconstruction, recognition, segmentation, and tracking under varying observational geometries.

Further, MVPNet's realism—characterized by noisy, partial point clouds—poses new challenges for self-supervised and supervised representation learning methods, driving innovation in robust architecture design. MVImgNet and MVPNet's scale, diversity, and annotation quality make them critical resources for universal 2D/3D representation learning, domain adaptation, and knowledge distillation research.

6. Limitations and Prospective Directions

MVImgNet's initial category set remains human-centric; an expansion to broader, less anthropocentric, and more cluttered scene-level objects is highlighted as a plausible future development. Such enhancement is likely to support:

• Generalization to dense and multi-object environments.
• Development of knowledge distillation schemas and cross-domain adaptation frameworks.
• Creation of universal feature representations linking 2D and 3D domains.

Potential technical improvements can address segmentation accuracy, camera pose estimation error, and scene-level annotation, as reflected in subsequent works (e.g., MVImgNet2.0).

7. Significance and Community Role

MVImgNet provides comprehensive, annotated multi-view imagery supporting a wide range of academic and applied 3D vision tasks. Its demonstrated utility—in neural rendering, multi-view stereo, robust classification, and as the base for MVPNet—indicates its value as a foundational resource. Public availability is intended to inspire community research, with strong empirical improvements as shown in model performance metrics (PSNR, accuracy, view-consistency).

MVImgNet has substantially advanced the state of multi-view 2D/3D computer vision by offering unprecedented scale and richly annotated data, catalyzing progress in both fundamental research and practical application.