Objects365-v2-3D: Web-Scale Pseudo-3D Dataset

• Objects365-v2-3D is a pseudo-3D dataset that lifts 2D images into 3D scenes with inferred depth and camera parameters.
• It fuses relative and metric depth predictions along with predicted intrinsics and extrinsics to achieve scale-calibrated 3D reconstructions.
• The dataset augments original 2D object annotations with 3D point clouds, pseudo-RGBD data, and 3D bounding boxes for diverse indoor and outdoor applications.

Searching arXiv for the main paper and the auxiliary methods mentioned, so the article can cite the relevant arXiv records. First, retrieving the main paper record. Searching for related papers: MoGe, Metric3D v2, WildCamera, PerspectiveFields, and Segment Anything. Objects365-v2-3D is a large-scale pseudo-3D scene dataset obtained by lifting the 2D Objects365-v2 object detection dataset into 3D through the 2D-to-3D data lifting pipeline introduced in "Towards Scalable Spatial Intelligence via 2D-to-3D Data Lifting" (Miao et al., 24 Jul 2025). It is pseudo-3D because its geometry, including depth, point clouds, and camera pose, is inferred from single RGB images by learning-based models rather than captured by RGB-D or LiDAR sensors. The dataset uses the original Objects365-v2 images and bounding-box annotations for 365 object categories, and augments each image with scale-calibrated depth, camera intrinsics and extrinsics, a 3D point cloud in camera and world coordinates, a pseudo-RGBD representation, and 3D object annotations derived from 2D boxes and masks. The paper summarizes Objects365-v2-3D as containing approximately 2M scene instances across indoor and outdoor conditions and positions it as a scale-aware 3D resource intended to support spatial intelligence, 3D perception, and vision-language modeling at web scale (Miao et al., 24 Jul 2025).

1. Dataset identity and source corpus

Objects365-v2-3D inherits its images and category labels directly from Objects365-v2, and therefore retains the original dataset’s 365-category object detection taxonomy while extending each 2D image into a 3D scene representation (Miao et al., 24 Jul 2025). The newly added content comprises a scale-calibrated depth map, camera intrinsics and extrinsics, a 3D point cloud, a pseudo-RGBD representation, and 3D object annotations derived from the original 2D annotations.

The paper describes the dataset as large-scale and diverse. In the appendix, Objects365-v2-3D is summarized as having approximately 2M scene instances and 365 categories, with coverage spanning indoor and outdoor scenes and more diverse conditions than typical RGB-D datasets (Miao et al., 24 Jul 2025). This positioning is central to its role in the paper: it is not presented as a sensor-captured benchmark, but as a generated spatial dataset designed to reduce the data bottleneck for large-scale 3D learning.

The designation “pseudo-3D” is technically significant. It distinguishes Objects365-v2-3D from RGB-D and LiDAR corpora by clarifying that the reconstructed geometry is algorithmically inferred from monocular imagery. A plausible implication is that the dataset inherits the appearance realism and long-tail diversity of web-scale image collections while simultaneously inheriting the uncertainty and failure modes of monocular depth and camera calibration.

2. Lifting pipeline and geometric reconstruction

The paper applies the same pipeline used for COCO-3D to Objects365-v2 images. The described workflow consists of relative depth estimation, metric depth estimation, scale calibration and fusion of the two depth maps, camera parameter prediction and 3D projection, and lifting 2D annotations to 3D, followed by automatic filtering and spot manual quality assessment (Miao et al., 24 Jul 2025).

For relative depth, the pipeline uses MoGe as the relative depth predictor. MoGe first predicts a 3D point cloud from a single view and then derives a depth map optimized with a multi-scale local geometric loss under affine-invariant alignments. Its output is a relative depth map $d_r$ with strong local geometric accuracy but unknown affine scale. For metric depth, the pipeline uses Metric3D v2, which predicts metric depth and surface normals and is trained across diverse indoor and outdoor scenes. Its output is a metric depth map $d_m$ in approximate real-world meters, typically with good global scale but comparatively less detailed local geometry in some cases.

The fusion step aligns the mean of the relative depth map to the mean of the metric depth map over valid pixels. Let $\mathcal{I}$ denote invalid indices and $\mathcal{V}$ the set of valid pixels. The scalar scale factor is defined as

$$s = \frac{\frac{1}{|\mathcal{V}|}\sum_{i \in \mathcal{V}} d_{m,i}}{\frac{1}{|\mathcal{V}|}\sum_{i \in \mathcal{V}} d_{r,i}} \tag{1}$$

and the scale-calibrated depth is

$$d_{sc,i} = s \cdot d_{r,i}, \quad \forall i \in \mathcal{V}. \tag{2}$$

This preserves the local geometric structure of $d_r$ while aligning its overall scale to $d_m$, thereby approximating metric depth.

Camera metadata are not available in Objects365-v2, so the pipeline estimates them. Intrinsics $K$ are predicted with WildCamera, which provides focal length and principal point and is described as scale-aware and robust to cropping. Extrinsics $[R|T]$ are derived using PerspectiveFields, which predicts a per-pixel upward vector and latitude information. From these predictions, the pipeline constructs a rotation matrix $d_m$0 aligning the scene to a standard world frame with $d_m$1-axis up, consistent with ScanNet and Structured3D, and a translation vector $d_m$2 locating the camera in that frame (Miao et al., 24 Jul 2025).

Given $d_m$3, image coordinates $d_m$4, intrinsics $d_m$5, and extrinsics $d_m$6, the 3D projection is

$d_m$7

for the camera-frame point, and

$d_m$8

for the world-frame point. The collection $d_m$9 forms a metric 3D point cloud, and associating RGB values from the source image yields a pseudo-RGBD or colored point cloud.

3. Annotation transfer from 2D to 3D

Objects365-v2 provides 2D bounding boxes but not masks. To obtain object-level 3D structure, the pipeline first uses SAM, initialized by the 2D boxes, to generate 2D instance masks (Miao et al., 24 Jul 2025). These masks and boxes are then lifted into 3D using the calibrated depth and camera parameters.

For each mask pixel, the pipeline computes a 3D position using the projection equations above, producing 3D instance point sets. For each 2D bounding box, it computes the minimum and maximum depth values inside the box, and combines those depths with the 2D box extents to define a 3D bounding volume that is standardized as a 3D bounding box annotation. In this way, Objects365-v2-3D inherits the category labels and instance delineations of Objects365-v2 while re-expressing them in 3D world coordinates.

The paper also specifies post-processing. Invalid or noisy points are removed through automatic filtering, and a subset of scenes is visually inspected to confirm alignment between 2D and 3D annotations. This means that the 3D object annotations are neither manually labeled from scratch nor directly sensor-measured; rather, they are derived products of monocular reconstruction, segmentation, and geometric projection.

4. Modalities, coordinate conventions, and scale-aware representation

Per scene, Objects365-v2-3D contains the original RGB image, multiple depth representations used during construction, estimated camera parameters, a point cloud, and both 2D and 3D annotations (Miao et al., 24 Jul 2025). The paper explicitly defines the projection and coordinate conventions: the camera coordinate system is standard pinhole, depth is along the camera $\mathcal{I}$0 direction, and the world coordinate system is gravity-aligned with the $\mathcal{I}$1-axis pointing upward.

Component	Source in pipeline	Role
RGB image	Original Objects365-v2 frame	Appearance source
Relative depth $\mathcal{I}$2	MoGe	Fine geometry
Metric depth $\mathcal{I}$3	Metric3D v2	Metric-scale anchor
Scale-calibrated depth $\mathcal{I}$4	Fusion of $\mathcal{I}$5 and $\mathcal{I}$6	Main stored depth
Intrinsics $\mathcal{I}$7	WildCamera	Focal length and principal point
Extrinsics $\mathcal{I}$8	PerspectiveFields-derived calibration	Gravity-aligned camera pose
3D point cloud	Projection via Eqs. (3)–(4)	World-coordinate geometry
2D instance masks	SAM from bounding boxes	Instance delineation
3D instance masks / point sets	Lifted mask pixels	Object-level 3D support
3D bounding boxes	Min/max depth within 2D box plus box extents	Box-level 3D annotation

The paper characterizes the resulting representation as “authentic, scale-aware 3D data.” In its terminology, this means that reconstructions approximate real-world metric sizes rather than arbitrary shapes, and that scale is intended to be consistent across scenes. The mechanism for achieving this consists of four linked elements: Metric3D v2 as the metric depth anchor, MoGe as the detailed relative geometry estimator, scale calibration through Eqs. (1)–(2), and global orientation calibration through PerspectiveFields and the $\mathcal{I}$9-up world frame (Miao et al., 24 Jul 2025).

Because the reconstruction is generated from a single image, the representation is necessarily partial: only visible surfaces are lifted into the point cloud. The paper does not specify file formats for Objects365-v2-3D, but states that the dataset structure is analogous to COCO-3D, with per-scene 3D data and annotation files.

5. Relation to other 3D and pseudo-3D datasets

The appendix compares Objects365-v2-3D with both object-level and scene-level 3D datasets (Miao et al., 24 Jul 2025). Object-level datasets such as ShapeNet, ModelNet, 3D-Future, ABO, and CO3D focus on individual objects rather than full scenes. Scene-level datasets such as ScanNet, Matterport3D, S3DIS, Structured3D, ARKitScenes, and ScanNet++ provide scene geometry, but are described in the paper as typically indoor only, limited to approximately $\mathcal{V}$0 to $\mathcal{V}$1 scenes, and constrained in category diversity, often to roughly 13–25 semantic classes.

Against that backdrop, Objects365-v2-3D is defined by three properties emphasized in the paper: approximately 2M scenes, 365 categories, and indoor-plus-outdoor in-the-wild imagery. It is derived from a web-scale 2D dataset with rich object detection annotations, and therefore differs structurally from both sensor-captured indoor scene datasets and simulation-based or AI-generated 3D data.

The paper further distinguishes its pipeline from prior pseudo-3D resources such as SpatialRGPT and SpatialBot by noting that those datasets often focus on coarse spatial question answering and may lack camera intrinsics and extrinsics. By contrast, the Objects365-v2-3D pipeline explicitly calibrates metric scale, camera intrinsics, camera extrinsics, and gravity direction. The paper therefore argues that the resulting dataset is usable not only for high-level reasoning but also for low-level 3D vision tasks such as segmentation and detection (Miao et al., 24 Jul 2025).

6. Experimental role, practical use, and limitations

Within the paper’s empirical study, Objects365-v2-3D is not the primary training or evaluation dataset. The main body reports detailed experiments on COCO-3D and on sensor-captured datasets including ScanNet, S3DIS, Matterport3D, and Structured3D, whereas Objects365-v2-3D is used primarily to demonstrate scale and diversity advantages in comparison tables (Miao et al., 24 Jul 2025). The text explicitly states that there are no quantitative experiments isolating Objects365-v2-3D as a training or evaluation source. All reported 3D perception and 3D-LLM experiments use COCO-3D as the synthetic pretraining dataset, while Objects365-v2-3D is presented as an additional large-scale resource built by the same pipeline.

The project page provided for the dataset family is https://ZhangGongjie.github.io/TowardsSSI-page/, and the paper states that the 3D scene data and annotations will be made publicly available. Licensing and usage conditions are not detailed in the paper itself. For downstream use, the paper recommends standard point-cloud pipelines such as Pointcept, Uni3D, SpUNet, and PTv2, and emphasizes using one unified set of hyperparameters across datasets to avoid overfitting to a particular distribution. It also notes that scenes can vary substantially in physical extent, from approximately 1 m to hundreds of meters, especially for outdoor content (Miao et al., 24 Jul 2025).

The limitations are those of the underlying monocular lifting pipeline. Because depth and pose are inferred, errors in Metric3D v2, MoGe, WildCamera, or PerspectiveFields can produce incorrect absolute scale, local geometric artifacts, or misaligned camera orientation in ambiguous cases. In addition, because Objects365-v2 is organized around object detection, some scenes may be close-ups with limited contextual extent, which can affect 3D spatial reasoning. The paper describes mitigation through automatic filtering, manual spot checks, and statistical checks of physical sizes.

A common misunderstanding would be to treat Objects365-v2-3D as equivalent to a fully observed sensor-captured 3D benchmark. The paper does not support that interpretation. Its claims are instead about scalability, diversity, scale awareness, and automatic generation from existing 2D corpora. In that narrower but important sense, Objects365-v2-3D functions as a generated 3D counterpart to a large 2D detection dataset and, according to the paper, as a step toward scalable spatial intelligence based on web-scale image resources (Miao et al., 24 Jul 2025).