---
title: RoboTwin-OD Object Library
url: https://www.emergentmind.com/topics/robotwin-od-object-library
type: topic
---

# RoboTwin-OD Object Library

The RoboTwin-OD Object Library is an open-source, large-scale repository of photorealistic and physically accurate 3D digital twin object models designed to support robotic simulation, manipulation policy learning, and sim-to-real transfer for both single-arm and dual-arm scenarios. Spanning hundreds of everyday object classes with detailed physical, semantic, and manipulation-relevant annotations, RoboTwin-OD is constructed through a combination of automated generative pipelines, human curation, and rigorous benchmarking within modern simulated and real-world robotic platforms [2506.18088, 2504.13059, 2508.00354, 2409.02920]. Recent integration with the Omni-Scan pipeline introduces high-fidelity 3D Gaussian Splat representations and robust object digitization from bi-manual robot scan data [2508.00354].

## 1. Library Scope, Structure, and Construction

RoboTwin-OD comprises 731 distinct object instances distributed across 147 semantic categories [2506.18088]. Objects are sourced from three main pipelines: in-house 3D scanning reconstructions (534 instances), Objaverse mesh imports (153), and SAPIEN PartNet-Mobility articulated object models (44). In-house models are reconstructed via RGB-to-3D pipelines (notably, the Deemos Rodin platform for image-conditioned latent diffusion) and post-processed with convex decomposition to yield simulation-ready, collision-accurate physical meshes [2504.13059, 2409.02920]. Each asset includes geometry (triangulated mesh with 10⁴–10⁵ faces), UV- or per-vertex mapped PBR textures (albedo, normal, optionally roughness/metallic), and physics metadata (mass, inertia, friction).

The taxonomic composition, as captured in Table 1, reveals diverse representation across containers (bottles, mugs, plates), tools, apparel, and other manipulanda:

| Statistic          | Mean    | Std. Dev. |
|--------------------|---------|-----------|
| Physical height (m)| 0.152   | 0.081     |
| Mass (kg)          | 0.327   | 0.154     |
| Number of grasp axes (Gₒ)| 6.4 | 2.1     |
| Number of placement points (Pₒ)| 3.8 | 1.2|

Objects are versioned, reproducibly stored by category and instance (e.g., `/objects/mug/0`), and packaged with all required simulation and rendering assets [2506.18088].

## 2. Representation Format, Annotation, and Physical Parameterization

Each object o in the set O is annotated via a labeling function φ: O → S × M, where S is the set of semantic labels (category, instance ID, material class, color palette, texture descriptor, 15 natural-language descriptions) and M is the set of manipulation-relevant labels (physical dimensions, mass, friction coefficients, collision mesh, and manipulation affordances) [2506.18088, 2504.13059].

**Manipulation affordances** are represented as:
- Grasping points: $\{g_i\} \subset \mathbb{R}^3, i=1\ldots G_o$
- Grasp axes: $\{a_i\} \subset S^2$
- Placement points: $\{p_j\} \subset \mathbb{R}^3, j=1\ldots P_o$
- Functional points (e.g., handle pivots, lid hinges)

Principal axes attached to keypoints specify function, approach, and lateral directions:

```json
"axes": {
  "function":[0,0,1],
  "approach":[1,0,0],
  "lateral":[0,1,0]
}
```

Physical parameter ranges for mass, friction, and restitution are category- and material-inferred, supporting domain randomization protocols. Jittered material properties (mass, roughness), random assignment of texture slots from a library of 12,000 samples, and minor scale/noise augment realism and diversity [2506.18088].

## 3. Automated Generation and High-Fidelity 3D Digitization

RoboTwin-OD supports two major pipelines for object generation: (a) a 2D-to-3D generative diffusion model ("Rodin") and (b) the Omni-Scan bi-manual digitalization approach.

### a) Generative Model Pipeline
A single RGB image is captioned and segmented (via GPT-4V), expanded into 20–30 appearance variants using SDXL (Stable Diffusion-XLTurbo), and reconstructed into 3D with latent diffusion in a VAE-style architecture ($D=256$ latent), followed by mesh extraction and texture lookup [2504.13059, 2409.02920]. The training loss combines latent diffusion reconstruction and photometric/view consistency, with Chamfer distance used for mesh quality.

### b) Omni-Scan Pipeline and 3D Gaussian Splatting [2508.00354]
Physical scanning leverages a bi-manual ABB YuMi robot: the right and left grippers alternate to expose occluded surfaces. DepthAnything V2 produces per-pixel depth, Segment Anything (SAM) generates masks, and RAFT/RAFT-Stereo isolate foreground and reject gripper/background pixels via a combined scoring criterion:
$$
S_{NR}(M) = \frac{1}{|M|}\sum_{p\in M} |D_{curr}(p) - D_{empty}(p)|
$$
$$
S_{NG}(M) = 1 - \frac{|M \cap G|}{|M|}
$$
Accepted object masks must satisfy $S_{NR}\geq 140$ and $S_{NG}\geq 0.89$.

Each object is reconstructed as a 3D Gaussian Splat (3DGS) model, where each splat is parameterized by mean $\mu_i \in \mathbb{R}^3$, covariance $\Sigma_i$, and weight $w_i$, with pixel-wise photometric, smoothness, and opacity losses used for training. Left- and right-gripper point clouds are merged via ICP-refined handover transforms.

Quantitative results include detection of visual (e.g., scratch, tape) and geometric defects with $91.7\%$ accuracy on 12 scanned objects. Reconstructed models achieve PSNR $\approx 28$–$31$dB, SSIM $\approx 0.98$–$0.99$, and LPIPS $\approx 0.01$–$0.03$.

## 4. Data Formats, Access Patterns, and Simulation Integration

Each object bundle includes:
- Mesh (.obj + .mtl), PBR textures
- Uniformly sampled point cloud (.ply)
- URDF fragment: includes geometry, inertial, and collision tags
- Physical/semantic metadata in JSON
- For Omni-Scan objects: high-res RGB images, depth, masks, camera/extrinsics, and serialized 3DGS parameters

Standardized API design exposes object loading, ray/mask/depth queries per view, 3DGS rendering, and simulation object spawning. Sample interfaces include:
```python
Loader.get_rays(image_id)
3DGSModel.render_view(pose)
DefectInspector.compare_models(model_A, model_B)
```
Export to simulation is achieved via URDF linking, marching cubes mesh extraction, and spawn scripts compatible with PyBullet, Isaac Gym, and ROS2 [2508

Source: https://www.emergentmind.com/topics/robotwin-od-object-library