PhysX-Omni: Unified Simulation-Ready 3D Generation

• PhysX-Omni is a unified framework that generates simulation-ready 3D assets for rigid, deformable, and articulated objects from single images.
• It uses a coarse-to-fine Vision-Language Model pipeline with template-based RLE and TRELLIS decoding to convert images into detailed 3D structures with physical attributes.
• The framework is validated on PhysXVerse and PhysX-Bench, demonstrating robust performance in geometry, scale, material, and kinematics for direct simulation use.

Searching arXiv for the primary paper and closely related work on simulation-ready physical 3D generation. PhysX-Omni is a unified framework for simulation-ready physical 3D generation that targets rigid, deformable, and articulated objects within a single Vision-LLM pipeline. It is presented as a system that takes a single complete or partially occluded image and predicts high-resolution 3D geometry, absolute scale, material, affordance, kinematics, and function description, with outputs intended to be directly usable in physics engines such as PhysX and MuJoCo (Cao et al., 20 May 2026). The framework is accompanied by PhysXVerse, described as the first general simulation-ready 3D dataset, and PhysX-Bench, a ground-truth-free benchmark that evaluates geometry, absolute scale, material, affordance, kinematics, and function description (Cao et al., 20 May 2026).

1. Definition and problem setting

PhysX-Omni addresses the problem of generating simulation-ready physical 3D assets rather than appearance-only 3D reconstructions. In the formulation used by the work, a simulation-ready asset is not just a mesh; it is a bundle consisting of geometry, scale, materials, kinematics or deformation, affordances, and a description. The paper writes this as

$$\text{Asset} = (\text{Geometry}, \text{Scale}, \text{Materials}, \text{Kinematics/Deformation}, \text{Affordances}, \text{Description}).$$

This definition is central to the framework’s scope because it positions the problem as one of physical asset generation for downstream simulation rather than static 3D content generation (Cao et al., 20 May 2026).

The motivation is that most prior 3D generation methods focus on appearance and geometry while omitting physical attributes such as absolute scale, mass-related properties, material parameters, joint structures, and affordances. The paper further states that many prior systems are limited to a single asset category, such as rigid, articulated, or deformable objects, and that their outputs are often not immediately usable in simulation without substantial manual cleanup (Cao et al., 20 May 2026).

PhysX-Omni therefore proposes a unified solution in which a single model can ingest an image of an object of any of these categories and output a simulation-ready asset. The stated downstream domains are robotics, embodied AI, and simulation or gaming pipelines. This suggests a broader shift from appearance-centric 3D generation toward physical world modeling in a simulator-compatible form (Cao et al., 20 May 2026).

2. Unified framework and coarse-to-fine generation

At a high level, PhysX-Omni is described as a VLM-based, coarse-to-fine 3D generator. The paper specifies a pipeline with four stages: input, global reasoning, local part-level generation, and decoding (Cao et al., 20 May 2026).

The input is a single RGB image, and the object may be partially occluded. At the coarse level, the system uses a Vision-LLM with a Qwen2.5-VL-7B backbone to infer the object category, semantic label, absolute scale, tree-structured part hierarchy, and global physical properties and coarse kinematics. At the fine level, the system generates geometry tokens for each part using a new RLE-based 3D representation and predicts per-part physical attributes including material, affordance, joint type and limits, and deformability. The generated geometry tokens are then converted into voxel grids, and TRELLIS is used as a voxel-based 3D decoder to reconstruct meshes and textures; these are combined with physical parameters to form a simulation-ready asset in a standard simulation format (Cao et al., 20 May 2026).

A central design choice is that rigid, deformable, and articulated objects share the same hierarchical part tree representation, the same text or VLM token space, and the same geometry encoding. According to the paper, differences among asset types are represented only through type flags and the presence and type of joints. The authors argue that this unification simplifies training and encourages cross-type knowledge transfer, such as reasoning over analogous parts across different categories (Cao et al., 20 May 2026).

This unified treatment distinguishes PhysX-Omni from category-specific physical asset generators. A plausible implication is that the framework is intended not merely as a geometry generator with auxiliary tags, but as a model of object structure whose output space is aligned with simulation pipelines.

3. Geometry representation and model architecture

The technical centerpiece of PhysX-Omni is a geometry representation intended to be both VLM-friendly and high fidelity. For each part, geometry is voxelized into a 3D binary grid

$V(x,y,z) \in \{0,1\},$

and then sliced along the $z$-axis into 2D binary masks

$S_z(x,y) = V(x,y,z), \quad z=1,\dots,N_z.$

Each slice is encoded using 2D run-length encoding, producing a sequence of run-length tuples $(v_i,\ell_i)$, where $v_i \in \{0,1\}$ and $\ell_i \in \mathbb{N}$. These are serialized into ordinary numeric or string tokens; the paper explicitly states that no new special tokens are added to the VLM vocabulary (Cao et al., 20 May 2026).

To further reduce sequence length, PhysX-Omni introduces template layers for cross-slice redundancy. Some slices are stored as template layers with full RLE, and other slices are represented as variations relative to a template. The paper describes this as template-based RLE and attributes three benefits to it: strong sequence-length reduction, preservation of full-resolution occupancy without lossy VQ-encoding, and explicit slice-by-slice geometric structure (Cao et al., 20 May 2026).

The backbone is Qwen2.5-VL-7B-Instruct, with a maximum sequence length of 16,384 tokens. Training is reported as 5 epochs on 64 × A100 GPUs for approximately 14 days, with peak learning rate $2 \times 10^{-5}$, cosine decay, 0.03 warmup, and effective batch size 128 (Cao et al., 20 May 2026). During training, the model receives multi-view images, specifically 25 views per object, together with text tokens representing global attributes, part-level geometry, and physical attributes. At inference time, the model performs multi-turn generation, conditioning part generation on the image, the global description, and already generated parts (Cao et al., 20 May 2026).

Geometry decoding uses TRELLIS to convert voxel outputs into high-quality meshes with appearance. The paper emphasizes that no extra refinement or segmentation stage is required, contrasting this with PhysX-Anything, which relied on a separate mesh segmentation module (Cao et al., 20 May 2026). This suggests that the representation is not only a geometry tokenization scheme but also a means of eliminating a major error source in previous systems.

4. Physical attributes and simulation-ready asset modeling

PhysX-Omni predicts the physical attributes needed for simulation across rigid, deformable, and articulated objects. For rigid objects, the representation includes part-level voxel grids or meshes, collision shapes derived from voxels or simplified primitives, material parameters such as friction coefficient $\mu$ and restitution $e$, and mass inferred from volume $V(x,y,z) \in \{0,1\},$0 and density $V(x,y,z) \in \{0,1\},$1 according to

$V(x,y,z) \in \{0,1\},$2

The paper states that the inertia tensor is then computed by the simulator from the mesh or volume and mass using standard rigid body formulas, though it does not detail those formulas (Cao et al., 20 May 2026).

For deformable objects, the framework predicts geometry together with deformable physical settings in the simulator. The paper does not provide explicit FEM or mass–spring parameterizations, but it states that standard deformable settings in the simulator are used, with stiffness and damping inferred from material attributes such as softness. The work includes free-fall deformation demonstrations showing squashing and bouncing, but explicitly notes that the paper does not dwell on the underlying matrix formulations (Cao et al., 20 May 2026). Any more specific description of deformable simulation internals would therefore exceed the stated evidence.

For articulated objects, articulation is represented as a kinematic chain of parts and joints. For each joint $V(x,y,z) \in \{0,1\},$3, the system predicts the type, axis $V(x,y,z) \in \{0,1\},$4, anchor position $V(x,y,z) \in \{0,1\},$5, and limits $V(x,y,z) \in \{0,1\},$6 for revolute joints or analogous linear limits for prismatic joints. Forward kinematics is written as

$V(x,y,z) \in \{0,1\},$7

The framework predicts connectivity, joint axes, anchors, limits, and overall articulation parameters, and these are evaluated via MSE against ground-truth articulation parameters (Cao et al., 20 May 2026).

The model also predicts absolute scale and uses an error term written as

$V(x,y,z) \in \{0,1\},$8

Material labels or properties and affordances are represented as discrete or continuous attributes and evaluated using PSNR on heatmaps. This combination of geometric and physical prediction is the core of the paper’s notion of “simulation-ready” generation (Cao et al., 20 May 2026).

5. PhysXVerse and PhysX-Bench

PhysXVerse is introduced as the first general simulation-ready physical 3D dataset. The paper reports that it contains 8.7K+ assets across 2.9K+ categories, spanning indoor and outdoor categories such as furniture, appliances, toys, helicopters, tanks, racing cars, skyscrapers, robots, UAVs, and vehicles. Asset complexity ranges from 1 to 65 parts, with a long-tailed distribution from simple rigid items to highly articulated machines (Cao et al., 20 May 2026).

The annotation pipeline begins from PartVerse meshes and human-verified segmentation, followed by preprocessing to filter invalid meshes and merge tiny or noisy parts. Physical annotation is human-in-the-loop: a GPT-like VLM proposes scale, materials, affordances, function descriptions, and kinematic information from rendered views, and human annotators verify and correct these proposals. The dataset provides geometry, part segmentation, scale, material and density, friction-related parameters, joint information when applicable, affordance annotations, and functional descriptions (Cao et al., 20 May 2026).

PhysX-Bench is presented as a ground-truth-free benchmark for evaluating simulation-ready 3D generation in the wild. It uses Qwen3.5-122B-A10B together with physical simulation to score generated assets along six attributes: geometry, absolute scale, material, affordance, kinematics, and function description (Cao et al., 20 May 2026). Geometry is broken into CLIP Score, 3D Consistency, and Visual Quality; absolute scale is evaluated through a symmetric percentage error

$V(x,y,z) \in \{0,1\},$9

material is evaluated by observing simulator behavior in free-fall and water-drop scenarios; affordance is evaluated through VLM judgments of plausibility and misranking; kinematics is scored by prior-part motion consistency, revealed-entity plausibility, and global articulation coherence via

$z$0

Function description is assessed by comparing part-level descriptions against rendered part masks and full-object views (Cao et al., 20 May 2026).

The paper reports strong correlation between PhysX-Bench and human judgment: absolute scale, affordance, material, and description achieve Spearman $z$1; kinematics achieves $z$2 and Pearson $z$3; geometry achieves $z$4 and $z$5 (Cao et al., 20 May 2026). This is a central claim because it underwrites the benchmark’s role as an automatic proxy for human evaluation.

6. Training formulation, empirical results, and downstream applications

The training procedure is described as autoregressive next-token prediction over geometry and physical attribute tokens conditioned on images and partial outputs:

$z$6

equivalently minimizing

$z$7

The paper then gives a conceptual decomposition into geometry reconstruction, scale, attribute, and kinematics terms, while noting that in practice most of these are realized through token-level cross-entropy because outputs are textified (Cao et al., 20 May 2026).

On conventional metrics, the work compares PhysX-Omni with Articulate-Anything, MonoArt, PhysXGen, and PhysX-Anything. On PhysXVerse, reported results include PSNR 21.52, Chamfer Distance $z$8, and F-score $z$9, all identified as the best among the compared methods. On physical attributes for PhysXVerse, the reported values are: absolute scale error 2.79, material 27.23, affordance 21.47, kinematics 0.9185, and description 31.05 (Cao et al., 20 May 2026). The paper states that similar improvements hold on PhysX-Mobility.

On PhysX-Bench, the work reports geometry metrics of CLIP 0.767, 3D consistency 64.48, and visual quality 90.0. For physical attributes, the reported scores are absolute scale 64.26, material 59.89, affordance 70.57, kinematics 80.72, and description 39.02, with the paper emphasizing strongest overall performance, especially on physics-related dimensions (Cao et al., 20 May 2026).

The paper also reports an ablation against a text-based voxel index baseline similar to PhysX-Anything. According to the description, template-based RLE improves PSNR, Chamfer Distance, and F-score, and “dramatically boosts” kinematic and scale predictions. Qualitative comparisons are said to show cleaner and more coherent structures, particularly for complex articulated objects such as strollers and tractors (Cao et al., 20 May 2026).

Two downstream applications are highlighted. The first is simulation-ready scene generation, in which a real scene image is processed using Depth Anything v2 for depth estimation and SAM2 for 2D segmentation, after which PhysX-Omni generates per-object simulation-ready assets that are inserted into the reconstructed 3D layout. The second is robotic policy learning, where assets generated by PhysX-Omni are deployed in a MuJoCo-based simulation framework, robopal, for contact-rich tasks such as grasping and pushing. The paper states that the generated assets behave in a physically plausible way under interaction, with articulated objects moving as expected and deformables deforming realistically in contact (Cao et al., 20 May 2026). This suggests that the framework is positioned not merely as an evaluator of physical semantics, but as an asset generator intended for direct simulator use.

7. Relation to adjacent “PhysX-Omni” directions and limitations

The term “PhysX-Omni” sits within a broader arXiv landscape of PhysX- or Omniverse-aligned systems, but those systems target different technical problems. OmniDrones is a drone reinforcement learning platform built on Omniverse Isaac Sim with PhysX as the physics core and a GPU-parallel RL workflow; it emphasizes control modes, sensor modalities, benchmark tasks, and vectorized RL environments rather than physical 3D asset generation (Xu et al., 2023). OmniPhysGS, by contrast, is a differentiable Material Point Method and 3D Gaussian Splatting framework for text-guided, multi-material dynamic scene synthesis; it uses constitutive Gaussians and expert constitutive models for elastic, plastic, and fluid materials rather than VLM-based simulator-ready object generation (Lin et al., 31 Jan 2025). These systems are related in their concern with physically grounded simulation, but their output objects and simulator interfaces differ substantially from those of PhysX-Omni.

Within its own scope, PhysX-Omni explicitly acknowledges several limitations. It can underperform specialized appearance-centric models on purely visual metrics for highly detailed or complex structures. Extremely intricate mechanical devices and very complex deformable materials remain challenging. Long sequences of up to 16k tokens and voxel decoding incur non-trivial computational cost. The dataset, although large and diverse, still has category-distribution biases and limited coverage of some object types and rare materials (Cao et al., 20 May 2026).

The paper identifies several future directions: combining the model with larger-scale geometry pre-training, expanding PhysXVerse with more complex deformables and richer annotations such as compliance and actuation models, better integrating appearance-oriented supervision while maintaining physical correctness, and extending from object-level generation to scene-level generative modeling with dynamic, interactive environments (Cao et al., 20 May 2026).

Taken together, these limitations clarify a potential misconception. PhysX-Omni is not presented as a complete physics simulator or as a replacement for engines such as PhysX; it is a generation framework that predicts simulator-ready assets and physical annotations intended to be consumed by such engines. Its contribution lies in unifying rigid, deformable, and articulated physical asset generation under a single VLM-compatible representation and evaluation regime, thereby creating a bridge between image-conditioned 3D generation and downstream physics-based simulation (Cao et al., 20 May 2026).