RoboTwin-OD Object Library

• RoboTwin-OD Object Library is an open-source repository of photorealistic 3D digital twin models with extensive physical and semantic annotations for robotic simulation.
• The library integrates automated generative pipelines and high-fidelity scan-based digitization to produce collision-accurate meshes and realistic PBR textures.
• It supports advanced simulation integration with standardized APIs and URDF outputs for both single-arm and dual-arm robotic systems.

The RoboTwin-OD Object Library is an open-source, large-scale repository of photorealistic and physically accurate 3D digital twin object models designed to support robotic simulation, manipulation policy learning, and sim-to-real transfer for both single-arm and dual-arm scenarios. Spanning hundreds of everyday object classes with detailed physical, semantic, and manipulation-relevant annotations, RoboTwin-OD is constructed through a combination of automated generative pipelines, human curation, and rigorous benchmarking within modern simulated and real-world robotic platforms (Chen et al., 22 Jun 2025, Mu et al., 17 Apr 2025, Qiu et al., 1 Aug 2025, Mu et al., 2024). Recent integration with the Omni-Scan pipeline introduces high-fidelity 3D Gaussian Splat representations and robust object digitization from bi-manual robot scan data (Qiu et al., 1 Aug 2025).

1. Library Scope, Structure, and Construction

RoboTwin-OD comprises 731 distinct object instances distributed across 147 semantic categories (Chen et al., 22 Jun 2025). Objects are sourced from three main pipelines: in-house 3D scanning reconstructions (534 instances), Objaverse mesh imports (153), and SAPIEN PartNet-Mobility articulated object models (44). In-house models are reconstructed via RGB-to-3D pipelines (notably, the Deemos Rodin platform for image-conditioned latent diffusion) and post-processed with convex decomposition to yield simulation-ready, collision-accurate physical meshes (Mu et al., 17 Apr 2025, Mu et al., 2024). Each asset includes geometry (triangulated mesh with 10⁴–10⁵ faces), UV- or per-vertex mapped PBR textures (albedo, normal, optionally roughness/metallic), and physics metadata (mass, inertia, friction).

The taxonomic composition, as captured in Table 1, reveals diverse representation across containers (bottles, mugs, plates), tools, apparel, and other manipulanda:

Statistic	Mean	Std. Dev.
Physical height (m)	0.152	0.081
Mass (kg)	0.327	0.154
Number of grasp axes (Gₒ)	6.4	2.1
Number of placement points (Pₒ)	3.8	1.2

Objects are versioned, reproducibly stored by category and instance (e.g., /objects/mug/0), and packaged with all required simulation and rendering assets (Chen et al., 22 Jun 2025).

2. Representation Format, Annotation, and Physical Parameterization

Each object o in the set O is annotated via a labeling function φ: O → S × M, where S is the set of semantic labels (category, instance ID, material class, color palette, texture descriptor, 15 natural-language descriptions) and M is the set of manipulation-relevant labels (physical dimensions, mass, friction coefficients, collision mesh, and manipulation affordances) (Chen et al., 22 Jun 2025, Mu et al., 17 Apr 2025).

Manipulation affordances are represented as:

• Grasping points: $\{g_i\} \subset \mathbb{R}^3, i=1\ldots G_o$
• Grasp axes: $\{a_i\} \subset S^2$
• Placement points: $\{p_j\} \subset \mathbb{R}^3, j=1\ldots P_o$
• Functional points (e.g., handle pivots, lid hinges)

Principal axes attached to keypoints specify function, approach, and lateral directions:

$\{a_i\} \subset S^2$8

Physical parameter ranges for mass, friction, and restitution are category- and material-inferred, supporting domain randomization protocols. Jittered material properties (mass, roughness), random assignment of texture slots from a library of 12,000 samples, and minor scale/noise augment realism and diversity (Chen et al., 22 Jun 2025).

3. Automated Generation and High-Fidelity 3D Digitization

RoboTwin-OD supports two major pipelines for object generation: (a) a 2D-to-3D generative diffusion model ("Rodin") and (b) the Omni-Scan bi-manual digitalization approach.

a) Generative Model Pipeline

A single RGB image is captioned and segmented (via GPT-4V), expanded into 20–30 appearance variants using SDXL (Stable Diffusion-XLTurbo), and reconstructed into 3D with latent diffusion in a VAE-style architecture ($D=256$ latent), followed by mesh extraction and texture lookup (Mu et al., 17 Apr 2025, Mu et al., 2024). The training loss combines latent diffusion reconstruction and photometric/view consistency, with Chamfer distance used for mesh quality.

b) Omni-Scan Pipeline and 3D Gaussian Splatting (Qiu et al., 1 Aug 2025)

Physical scanning leverages a bi-manual ABB YuMi robot: the right and left grippers alternate to expose occluded surfaces. DepthAnything V2 produces per-pixel depth, Segment Anything (SAM) generates masks, and RAFT/RAFT-Stereo isolate foreground and reject gripper/background pixels via a combined scoring criterion:

$$S_{NR}(M) = \frac{1}{|M|}\sum_{p\in M} |D_{curr}(p) - D_{empty}(p)|$$

$S_{NG}(M) = 1 - \frac{|M \cap G|}{|M|}$

Accepted object masks must satisfy $S_{NR}\geq 140$ and $S_{NG}\geq 0.89$.

Each object is reconstructed as a 3D Gaussian Splat (3DGS) model, where each splat is parameterized by mean $\mu_i \in \mathbb{R}^3$, covariance $\Sigma_i$, and weight $\{a_i\} \subset S^2$0, with pixel-wise photometric, smoothness, and opacity losses used for training. Left- and right-gripper point clouds are merged via ICP-refined handover transforms.

Quantitative results include detection of visual (e.g., scratch, tape) and geometric defects with $\{a_i\} \subset S^2$1 accuracy on 12 scanned objects. Reconstructed models achieve PSNR $\{a_i\} \subset S^2$2–$\{a_i\} \subset S^2$3dB, SSIM $\{a_i\} \subset S^2$4–$\{a_i\} \subset S^2$5, and LPIPS $\{a_i\} \subset S^2$6–$\{a_i\} \subset S^2$7.

4. Data Formats, Access Patterns, and Simulation Integration

Each object bundle includes:

• Mesh (.obj + .mtl), PBR textures
• Uniformly sampled point cloud (.ply)
• URDF fragment: includes geometry, inertial, and collision tags
• Physical/semantic metadata in JSON
• For Omni-Scan objects: high-res RGB images, depth, masks, camera/extrinsics, and serialized 3DGS parameters

Standardized API design exposes object loading, ray/mask/depth queries per view, 3DGS rendering, and simulation object spawning. Sample interfaces include: $\{a_i\} \subset S^2$9 Export to simulation is achieved via URDF linking, marching cubes mesh extraction, and spawn scripts compatible with PyBullet, Isaac Gym, and ROS2 [2508