SimWeaver-Asset: Unified 3D Simulation

• SimWeaver-Asset is a unified asset generation and simulation module that creates high-fidelity 3D assets from raw inputs for diverse simulation needs.
• It integrates state-of-the-art generative modeling, object-centric reconstruction, and physics-based parameter estimation to enable zero-shot sim-to-real transfer.
• Its modular architecture and automated pipelines support scalable deployment in robotics, autonomous vehicles, and embodied AI research.

SimWeaver-Asset is the asset generation and simulation module underpinning the SimWeaver ecosystem for robotics, autonomous vehicle simulation, and embodied AI research. It provides a unified framework for creating, augmenting, and deploying high-fidelity 3D assets—from deformable textiles and traffic actors to articulated machinery—supporting fully-automated pipelines from raw inputs (RGB, LiDAR, or CAD/B-reps) through to physics-parameterized, simulator-ready objects. SimWeaver-Asset integrates recent advances in generative modeling, object-centric reconstruction, physical parameter estimation, and hierarchical articulation, with system-level interfaces designed for zero-shot sim-to-real transfer in diverse closed-loop environments (Hu et al., 13 Jun 2026).

1. System Architecture and Asset Representations

SimWeaver-Asset encompasses a modular architecture supporting diverse asset classes. For deformables, all assets are represented as manifold triangle meshes $M=(V,F)$ with associated UV parameterizations and PBR texture maps (albedo, roughness, normal). Each asset carries a physical property vector $p=(\rho, \lambda_s, \kappa_b, \mu_s, \mu_d)$, sampled from empirical distributions over fabric categories (e.g., silk or nylon) (Hu et al., 13 Jun 2026). For rigid and articulated agents (vehicles, tools, assembly parts), representations extend to Gaussian splat fields, convex hull decompositions, or B-rep structures, with corresponding mass, inertia, and joint dynamics parameters.

Architectural stages are specialized by asset domain:

• Deformables: Image-to-mesh pipelines leveraging segmentation, single-image geometry networks, UV mapping, and texture synthesis, coupled with physical parameter classifiers and samplers grounded in measurement libraries (e.g., RGBench).
• Autonomous Vehicle Assets: A multi-stage flow incorporating log ingestion, camera/LiDAR fusion, sparse-view multi-view generation (SparseViewDiT), 3D Gaussian lifting (TokenGS), and packaging for simulation via USD/glTF (Cao et al., 20 Apr 2026).
• Articulated Objects: Unified transformer-based decompositions (e.g., SIMART), employing sparse 3D VQ-VAE for efficient part-level encoding, with joint parameter prediction and URDF/SDF export (Zhang et al., 24 Mar 2026).
• Assembly Pairs: Automated interpenetration removal, diffusion-based mating-part generation, and clearance modulation, terminating in URDF/SDF assets with verifiable non-penetration constraints (Wang et al., 7 Mar 2025).
• Real-World Scanning: Full automation from RGB-D data and robot-in-the-loop physical system identification, producing triangulated meshes, convex approximations, and calibrated inertial parameters (Pfaff et al., 1 Mar 2025).

2. Mathematical and Algorithmic Foundations

Mathematical formalism is tailored to the asset type and pipeline stage:

• Physics-Driven Deformables: Mesh geometry $M=(V,F)$, UV chart $u:V\to\mathbb{R}^2$, and sampled $p_j\sim D_{c,j}$ for each property, enabling direct variation and domain randomization (Hu et al., 13 Jun 2026).
• Sparse-View Generative Modeling: Multiview generation is formalized by a flow-matching loss:

$$\mathcal L_{FM} = \mathbb{E}_{t,x_0,x_1,c}\|\ v_\theta(x_t,t,c) - (x_1 - x_0)\|^2$$

and geometry-constrained regularization:

$$\mathcal L_\mathrm{geo} = \lambda_{geo}\sum_{i=1}^{v_{out}}\|\,\Pi_{in\to out}(\hat X_i) - x_{in\to out}\|^2$$

(Cao et al., 20 Apr 2026).

• 3D Gaussian Splatting: Asset fields parameterized as $g_k = (\mu_k, \Sigma_k, \alpha_k, c_k)$ with photometric and regularization losses minimized over differentiable splatting renders.
• Latent Diffusion in 3D Asset Space: High-quality actors are generated by diffusion in a learned latent code space:

$$q(c^{(t)} \mid c^{(t-1)}) = \mathcal{N}(\sqrt{1-\beta^{(t)}}c^{(t-1)}, \beta^{(t)}I)$$

with denoising U-Net $f_\mathrm{diff}$ and decoding via triplane-based neural rendering (Yang et al., 24 Apr 2026).

• Articulation & Kinematic Prediction: MLLM architectures output per-part joint JSON with origin, axis, and limits; joints are reconstructed as

$p=(\rho, \lambda_s, \kappa_b, \mu_s, \mu_d)$0

(Zhang et al., 24 Mar 2026).

• Physical Parameter Identification: System identification via torque trajectory regression, subject to pseudo-inertia positivity:

$p=(\rho, \lambda_s, \kappa_b, \mu_s, \mu_d)$1

solved via convex SDP (Pfaff et al., 1 Mar 2025).

3. Data Curation, Augmentation, and Pipeline Engineering

Data preparation strategies are asset-domain specific:

• Deformables: Large-scale mesh imports (e.g., CLOTH3D), synthetic and real asset generation from single images, empirical sampling of physics parameters, and normalization of UV/pixel scales. Validation includes landmark graph extraction for manipulation primitives.
• Traffic Participants/General AV: Large-scale curation ($p=(\rho, \lambda_s, \kappa_b, \mu_s, \mu_d)$2278k objects), rigorous occlusion and mis-segmentation filtering (Mask2Former + VLM), farthest-point view sampling, and self-distillation with low-vs-high quality benchmarks (Cao et al., 20 Apr 2026). Photometric augmentation and ISP noise injection are used to align with real sensor artifacts.
• Articulation and Assembly: Automated part segmentation and kinematic extraction (SIMART), grid-based occupancy and B-rep graph construction (MatchMaker) (Wang et al., 7 Mar 2025, Zhang et al., 24 Mar 2026). Sample diversity and assembly complexity metrics ensure broad coverage.

Augmentation schemes include occluder injection, geometric perturbation, synthetic domain mixing, brightness/contrast jitter, and cross-domain blending for robustness in real-to-sim scenarios.

4. Training Methodologies, Inference, and API Design

Distinct model training workflows are used for different asset classes:

• Deformables: EmbodiedGen-based image-to-asset pipeline: segmentation, geometry, UV, PBR texture, fabric classification, and parameter sampling; all steps are differentiable and support batch parallelism (Hu et al., 13 Jun 2026).
• SparseViewDiT+TokenGS: Three-stage schedule: pretrain on synthetic, adapt to real, and self-distill; optimizer AdamW with staged learning rates, token batch sizes, and explicit cross-attention modules (Cao et al., 20 Apr 2026).
• 3D Diffusion/Latent-Generative Models: Dual-phase regime, reconstruct-then-generate; actor codebank construction via occlusion-aware neural rendering, then diffusion-based generative sampling and mesh decoding (Yang et al., 24 Apr 2026).
• Articulated/Assembly Assets: MLLM fine-tuning (Qwen3-VL), sparse 3D VQ-VAE quantization, joint JSON generation, and graph-based part/axis mapping (Zhang et al., 24 Mar 2026). Assembly: interpenetration removal by SDF+gradient descent, diffusion completion of mating parts, morphological clearance specification (Wang et al., 7 Mar 2025).

Inference APIs expose standardized generation endpoints: asset-in, asset-out; mesh+texture+physics, with variants for unconditional (sample) and conditional (few-shot, image-to-3D) synthesis.

5. Simulator and Pipeline Integration

SimWeaver-Asset assets are packaged for seamless interoperability in large-scale simulation pipelines:

• Deformables: Direct import of mesh, UV, textures, and physics into SimWeaver-Sim for gravity, contact, and manipulation; landmark graphs handed to SimWeaver-Syn for grasp synthesis and trajectory planning (Hu et al., 13 Jun 2026).
• Rigid/Articulated: URDF/SDF output with full joint, inertial, and collision specifications; immediate compatibility with MuJoCo, Isaac Gym, Gazebo, and SimWeaver-Sim. Simulator-in-the-loop refinement ensures non-penetration and physical realism for articulated assets (Pham, 18 May 2026).
• Automated Pipelines: APIs accept batches of multi-modal assets, returning scene subgraphs (.usd/.usda, .png/.exr maps, metadata JSON); Python plugins support asset spawn, pose insert, and scaling, managing collision proxies and PBR materials (Cao et al., 20 Apr 2026).

Integration guidelines include multi-GPU inference batching, simulator-native coordinate conventions (e.g., FLU), and hierarchical asset registration/configuration to maximize downstream reproducibility and extendability.

6. Quantitative Metrics, Evaluation Protocols, and Experimental Validation

SimWeaver-Asset’s effectiveness is established via rigorous evaluation protocols:

• Rendering Quality: Held-out view metrics: PSNR, SSIM, LPIPS; DINOv3 rigid/part embeddings (ED-R/ED-P), completeness/coverage measures based on silhouette-IoU over spherical directions (Cao et al., 20 Apr 2026).
• Simulation Robustness: Physical robustness metrics—penetration depth $p=(\rho, \lambda_s, \kappa_b, \mu_s, \mu_d)$3, positional/orientation drift $p=(\rho, \lambda_s, \kappa_b, \mu_s, \mu_d)$4, joint oscillation flags, and closed-loop stability under manipulation—evaluated across hundreds of objects and tasks (Pham, 18 May 2026).
• Assembly and Articulation: Shape diversity via EGAD similarity, shape complexity, grasp difficulty, and assembly/success rates for RL insertion tasks (typically 88%+ in sim, 82% real) (Wang et al., 7 Mar 2025).
• Latent Generative Models: Asset class FID-C, full-frame FID, and NVS PSNR/SSIM; ablation of Gaussian-based rendering, shadow channeling, and cross-attention demonstrate performance margins (Dobre et al., 27 Dec 2025).
• Automation and Scale: Processing throughput (e.g., $p=(\rho, \lambda_s, \kappa_b, \mu_s, \mu_d)$56s per asset on H100 for AV objects), batchable inference, and fully-automated real2sim loops for object digitization at scale (Pfaff et al., 1 Mar 2025).

7. Limitations and Future Prospects

SimWeaver-Asset’s current boundaries include:

• Data and Geometry Sensitivity: Noisy or non-watertight meshes may cause simulation failures or incorrect inertia/joint limits (Pham, 18 May 2026). Asset quality is also contingent on sparse-view completeness and segmentation accuracy.
• Articulation Coverage: Current MLLM paradigms are limited to primary joint types (fixed/revolute/prismatic); extending to compound or gear joints requires schema and prompt extension (Zhang et al., 24 Mar 2026).
• Scalability: While batchable, large-scale deployments require substantial compute (32GB+ GPU, multi-node clusters for MLLM fine-tuning). Some pipelines are sensitive to prompt-template brittleness and require strict formatting.
• Simulator Feedback Dependency: Real-time physics refinement (for articulated assets) necessitates headless physics backends and is bottlenecked by penetration refinement loops.

A plausible implication is that simulation fidelity and sim-to-real generalization will continue to benefit from integration of measurement-driven property estimation, active loop simulator feedback, and expansion of generative backbones to new topologies and material domains. SimWeaver-Asset offers an extensible, measurement-grounded platform for asset-centric embodied intelligence, supporting both academic research and scalable simulation deployment (Hu et al., 13 Jun 2026, Cao et al., 20 Apr 2026, Pham, 18 May 2026, Zhang et al., 24 Mar 2026, Wang et al., 7 Mar 2025, Pfaff et al., 1 Mar 2025, Dobre et al., 27 Dec 2025, Yang et al., 24 Apr 2026).