SOPE: 6D Pose, Size & Shape Dataset

• SOPE is a synthetic dataset designed for category-level 6D object pose, size, and dense shape estimation across 149 diverse object categories.
• It employs advanced simulation pipelines with physically accurate RGB-D renderings and realistic depth noise to mimic actual sensor behavior.
• Its unified annotation suite and extensive data support zero-shot, cross-domain generalization for robust applications in robotics and computer vision.

SOPE (Synthetic Object Pose Estimation) is a large-scale synthetic dataset purpose-built to advance category-level 6D object pose, size, and dense shape estimation. It provides extensive high-fidelity RGB-D renderings, unified geometric annotations, and diversified semantic coverage across a wide range of everyday object categories. SOPE has become foundational in evaluating and pretraining deep models for open-set 6D understanding, especially in robotics and computer vision contexts where generalization beyond closed sets of templates or CAD models is central (Zhang et al., 13 Oct 2025, Zhang et al., 2024).

1. Purpose and Design Scope

SOPE was created to overcome the limitations of prior corpora that were constrained to few categories, template-based annotation, or lacked unified dense shape supervision. It is designed to:

• Enable large-scale, simulation-only training for category-level joint 6D pose, metric size, and dense 3D shape estimation.
• Cover a broad set of 149 object categories (e.g., dishes, utensils, tools, bottles, toys, electronics), structured to enable category-agnostic generalization.
• Provide every sample with all target labels—6D pose, size, and point-based ground-truth dense shape.

A primary motivation was to provide sufficient coverage for downstream zero-shot and cross-domain generalization tasks, as well as to facilitate synthetic-to-real transfer using paired synthetic and real benchmarks (Zhang et al., 13 Oct 2025, Zhang et al., 2024).

2. Dataset Composition and Statistics

Property	Value/Description	Notes
Number of Categories	149	Broadest among category-level 6D datasets (Zhang et al., 13 Oct 2025)
Number of Instances	4,162 (Omni6DPose), 5,000	Instance count varies slightly by paper (Zhang et al., 2024, Zhang et al., 13 Oct 2025)
Number of Frames	475,000 (Omni6DPose SOPE)	Each frame: RGB-D, pose labels for all instances present
Number of Annotations	5,000,000+	One 6D annotation per object per frame
Per-category Coverage	≥25,000 annotations/category	Over-sampling for transparent and specular categories
Intended Split	Train only	Evaluation occurs on real (ROPE) or mixed datasets

Most variant counts and summary statistics are directly specified in the Omni6DPose paper (Zhang et al., 2024). The SOPE dataset as described in (Zhang et al., 13 Oct 2025) comprises 5,000 instances spanning all 149 categories, yielding an average of approximately 33 instances per category.

3. Rendering and Data Generation Pipeline

SOPE images are generated entirely in simulation, using physically plausible, randomized and context-rich pipelines:

• RGB Images (Context-Aware Mixed Reality): Scene generation places each 3D object into a random background derived from large-scale indoor panoramas (Matterport3D, ScanNet++, IKEA), with ray-traced lighting and material property randomization. This guarantees physically accurate shadows, occlusions, and material/illumination effects by true light transport simulation—not raster overlays.
• Depth Simulation: Depth is rendered using structured-light/active stereo simulation, closely modeling the RealSense D415 sensor characteristics. IR dot patterns are rendered, disparities are evaluated via stereo matching, and zero-mean Gaussian noise (with variance as a quadratic function of depth) is introduced to mimic real sensor behavior.
• Domain and Appearance Randomization: Object materials, colors, light source temperature/intensity, distractor occlusion, and camera poses are heavily randomized. Camera viewpoints are sampled on a hemisphere (elevation: [0°,60°], azimuth: [0°,360°], distance: [0.5,1.5] m). Between 1 and 3 distractor objects are placed in front of the target, ensuring robust occlusion statistics.
• Point Cloud Extraction: From each simulated depth image, points corresponding to the target object are back-projected via camera intrinsics and sampled (1,024 points per partial view; 2,048 for full shapes when available) (Zhang et al., 13 Oct 2025).

4. Annotation Schema and File Formats

SOPE supplies a unified multi-task annotation suite per sample:

• 6D Pose: Rigid transform $p = (R, t) \in SE(3)$, where $R$ is a 3×3 rotation and $t \in \mathbb{R}^3$ is the canonical→camera translation. Format supports rotation matrices or equivalent quaternions; all annotations are given in the camera frame (z forward, y up).
  • Symmetry-Aware Loss: During training, all object symmetries are enumerated in a set $\mathcal{G}_R$, and the final loss is the minimum smooth L1 error over this set (per (Zhang et al., 13 Oct 2025)).
  • NOCS Alignment: Object coordinates follow the NOCS canonicalization protocol.
  • Example Annotation (JSON-like):
  • $R$0
  • (Zhang et al., 2024)
• Object Size: $s \in \mathbb{R}^3$, indicating the object's full extent along the x, y, z axes (in camera frame).
• Dense Shape Supervision:
  • Complete Shape: Ground-truth point cloud $$P_{\text{dense}}^{gt} \in \mathbb{R}^{2048\times 3}$$.
  • Partial Shape: Input view-specific point cloud $P_{\text{partial}} \in \mathbb{R}^{1024\times 3}$, sampled from observed surface.
  • These facilitate joint learning of pose, size, and voxel/point-based 3D reconstruction (Zhang et al., 13 Oct 2025).
• Data Modalities: Each scene/frame provides an RGB image, a channel-aligned simulated depth map, and per-object mask and annotation files. The camera intrinsics for all samples replicate a RealSense D415: $f_x = f_y \approx 615\, \text{px}$, $c_x \approx 320\, \text{px}$, $c_y \approx 240\, \text{px}$ (Zhang et al., 2024).

5. Benchmarking, Metrics, and Comparative Analysis

SOPE's unified design enables benchmarking on single- and multi-task 6D understanding benchmarks. Key statistics (on "seen" categories with (Zhang et al., 13 Oct 2025)'s method):

• AUC (3D IoU at 25/50/75%): 56.4 / 39.8 / 12.7
• Mean Rotation Error: 16°
• Mean Translation Error: 0.99 cm

These metrics exceed those of previous category-level methods, with marked improvements attributed to dataset scale, instance diversity, and unified dense shape supervision. Evaluation protocols for cross-domain testing are defined by training solely on SOPE and assessing direct transfer to real-world datasets, such as ROPE (Zhang et al., 2024).

Compared to prior datasets:

• Category Coverage: 149 in SOPE vs. 12 (PhoCaL), 30 (T-LESS), ∼100 (HouseCat6D).
• Instance Scale: 4,162–5,000 models vs. a few hundred in earlier corpora.
• Material Diversity: ∼20% transparent, ∼10% specular, remainder diffuse.
• Occlusion: ∼30% of frames have >25% occlusion by distractors.

Dataset	Categories	Instances	Dense Shape	Context/Lighting	Occlusion
SOPE	149	4,162–5k	Yes (2048)	Mixed-reality	Distractors, 30%+
PhoCaL	12	-	Unknown	Simple	Low
T-LESS	30	-	Unknown	Factory/close-up	Low
HouseCat6D	~100	-	Unknown	Simple	Low

6. Usage, Preprocessing, and Recommendations

SOPE is distributed for research under the CC BY-NC 4.0 license. Official release includes download links, intrinsics files, and code for scene export/import. Researchers should:

• Convert raw depth PNGs to metric point clouds using the provided camera matrix.
• Apply per-frame segmentation masks to isolate object pixels.
• Perform farthest point sampling (1,024 points) for partial clouds and resize RGB crops (e.g., to 224×224 for GenPose++ pipelines).
• Train models exclusively on SOPE synthetic data, with evaluation performed on cross-domain real benchmarks such as ROPE.

SOPE’s scalability, alignment with real-world object categories, and multi-modal annotation enable rigorous benchmarking for sim-to-real transfer, zero-shot recognition, and open-set 6D pose challenges (Zhang et al., 13 Oct 2025, Zhang et al., 2024).

7. Significance and Limitations

SOPE represents a milestone in dataset construction for category-level 6D understanding. Its primary innovations include scale, breadth of semantic coverage, unified annotation (pose, size, complete shape per instance), and physically plausible simulation pipelines. These properties facilitate the development and benchmarking of category-agnostic 6D pose estimators, especially those requiring dense shape signals for high-fidelity manipulation, grasp planning, and embodied AI.

Limitations include lack of native real sensor noise modeling in all variants (basic “simulated depth” only in (Zhang et al., 13 Oct 2025)), absence of a predefined held-out test split (as SOPE is “train-only” by construction in (Zhang et al., 2024)), and no explicit occlusion-severity annotations in every release. A plausible implication is that precise performance under extreme occlusion or rare material types should be cross-validated on real benchmarks (e.g., ROPE, HANDAL).

Overall, SOPE provides the scale, diversity, and annotation density needed to drive forward universal and category-agnostic 6D perception research, acting as the recommended pretraining corpus for next-generation pose, size, and shape estimation frameworks (Zhang et al., 13 Oct 2025, Zhang et al., 2024).