Automatically Improving Simulation Physics for Articulated Objects

Abstract: Simulation is a central tool for scalable robot learning, but its effectiveness depends on the quality of object assets. While modern 3D datasets provide rich geometric and kinematic representations, they typically lack the physical properties required for stable and realistic interaction, requiring significant manual effort to construct simulation-ready articulated objects. In this thesis, we introduce interaction-readiness, which characterizes whether an object can be reliably simulated under manipulation. We propose a quantitative evaluation framework that decomposes interaction-readiness into measurable components, enabling systematic analysis of object quality and revealing failure modes not captured by conventional evaluation. We further present a multi-modal, simulator-in-the-loop approach for generating interaction-ready articulated objects from incomplete 3D assets. The method integrates geometric, visual, and semantic information to infer physical properties and refines them through iterative simulator feedback to improve physical consistency. Experiments across diverse articulated objects and manipulation tasks show that object quality directly impacts simulation stability, interaction behavior, and policy performance. Objects refined by our method exhibit more stable and realistic dynamics, enabling more reliable downstream learning and evaluation. Overall, this thesis demonstrates the importance of physical realism for articulated objects in simulation and introduces a practical multi-modal refinement approach, guided by simulator feedback, for constructing such objects at scale.

• The paper introduces Asset2Sim, a pipeline that uses multi-modal feature extraction and simulator feedback to autonomously refine physical parameters in 3D models.
• It demonstrates significant improvements in physical robustness, semantic fidelity, and interaction realism over traditional VLM-only approaches.
• Empirical evaluations show near-perfect task success rates and enhanced sim-to-real policy transfer, making it highly valuable for robot learning.

Automated Physically Realistic Simulation for Articulated Objects: An Expert Summary of Asset2Sim

Motivation and Context

Simulated environments have become fundamental substrates for data collection, training, and benchmarking in robot learning, especially for manipulation tasks. However, there remains a pronounced gap between the scale and diversity of geometric-kinematic 3D object repositories and the availability of physically realistic, simulation-ready articulated assets. State-of-the-art datasets like ShapeNet and PartNet-Mobility provide ample geometric and kinematic information but systematically omit crucial physical parameters—such as mass, inertia, friction, and damping—necessary for stable, realistic interaction. Automated pipelines for converting static models to simulation-ready format predominantly focus on geometry and articulation structure, leaving the critical task of specifying and validating physical parameters a largely manual, trial-and-error process.

This paper addresses the interaction-readiness gap by introducing a rigorous evaluation framework for articulated objects in simulation and proposing Asset2Sim, a multi-modal, simulator-in-the-loop method for generating physically plausible, interaction-ready articulated assets from incomplete 3D models. The workflow leverages geometric, visual, and semantic cues with foundation-model inference and iteratively refines object parameters using direct feedback from a physics engine.

Problem Formulation: Interaction-Readiness and Evaluation

The authors define interaction-readiness as a composite metric of an asset’s suitability for physically plausible manipulation. Traditional evaluations—task success rates or qualitative inspection—fail to disentangle object fidelity from policy optimization and downstream task entanglements. Instead, the proposed framework decomposes object quality into six complementary axes:

• Physical Robustness: Quantified via penetration depth, static positional/orientational drift post-initialization, and oscillatory joint behavior.
• Semantic Accuracy: Alignment with expert reference scale/configuration and with user natural language specifications.
• Behavioral Fidelity: Zero-shot success and sim-to-real correlation of policies pretrained on large-scale real-world data when deployed in simulation.
• Interaction Realism: Human expert teleoperation judgment of dynamic plausibility.
• Visual Realism: Assessment by foundation VLMs acting as discriminators of Newtonian and causal consistency.
• Learning Feasibility: The amenability of the object dynamics for reinforcement learning policy convergence.

This multidimensional protocol effectively separates simulation artifacts from learning algorithmic limitations and exposes previously obscured failure modes.

Asset2Sim: Simulator-in-the-Loop Asset Refinement

Pipeline Overview

The Asset2Sim approach comprises three principal stages:

1. Structured Multi-Modal Feature Extraction: The input—typically an incomplete URDF and associated meshes—is analyzed to extract geometric, semantic, visual (multi-view renders), and analytic features, thereby forming a comprehensive context for property inference.
2. Vision-LLM (VLM)-Guided Physical Property Synthesis: A VLM is prompted with the assembled feature context and optional user guidance to generate a structured overlay specifying:
   • Global scaling and contact properties.
   • Per-link mass and inertia (using geometric measurements, VLM-inferred densities, and hollow factors).
   • Per-joint damping, friction, and stiffness, critically enforcing the presence of all terms for each joint.
   • Optionally, initial joint positions for a desired articulation state.
3. Simulator-Guided State Refinement: The candidate asset undergoes iterative simulation to evaluate joint configuration—a contact penetration measure is minimized via targeted updates. Proposed changes (both magnitude and joint focus) are steered by collision analysis and semantics, with acceptance gated by improvement in numerical feasibility.

   Figure 1: The Asset2Sim pipeline: multi-modal feature extraction, VLM property synthesis, and closed-loop simulator-based refinement yield interaction-ready articulated assets.

The key innovation is the closed-loop integration of model-driven physical reasoning with simulator-enforced constraints, as opposed to previous open-loop static prediction.

Empirical Evaluation: Numerical and Qualitative Metrics

Physical Robustness and Semantic Fidelity

Asset2Sim achieves 100% pass rates on static stability and joint oscillation when tested on diverse assets, outperforming VLM-only approaches which often yield penetration, drift, or oscillatory artifacts. The multi-modal refinement procedure attains the lowest deviation from expert reference scale and initial joint configuration. Prompt alignment with natural language queries approaches 97%, indicating robust semantic understanding and preservation despite later physical refinements.

Behavioral Fidelity and Policy Predictivity

Zero-shot deployment of strong real-world-trained vision-language-action policies ($\pi_{0}$, $\pi_{0.5}$, GR00T N1.6) yields simulation success rates for Asset2Sim that are 2–4x higher than baselines and reveal sim-to-real success rate correlation (SRCC) up to 0.68—the highest among tested methods, directly quantifying enhanced physical fidelity in learned dynamics.

Learning Feasibility

For policy learning via PPO, Asset2Sim supports near-perfect final task success rates (mean 97.5%) across all articulation tasks, while baselines either fail almost completely (Direct VLM, Human GenSim2), or succeed only on a subset (VLM-IVW).

Motion and Interaction Realism

Human experts, employing teleoperation, assign realism scores of 7.04/10 to Asset2Sim assets versus ≤4 for all other methods. Similarly, state-of-the-art VLM judges consistently prefer Asset2Sim outputs in terms of Newtonian and causal motion consistency.

Large-Scale Generalization

Across 93 assets from 10 object categories, Asset2Sim maintains a 96% success rate on stability checks, vastly exceeding VLM baselines, which are particularly brittle on high-DOF or tightly-coupled jointed objects.

Figure 2: Distribution of stability outcomes over 93 articulated assets. Asset2Sim achieves markedly higher rates of valid, stable object creation.

Failure Modes: Diagnostics of Predictive Baselines

Qualitative analysis of VLA-manipulated simulation trajectories reveals that methods relying on static VLM estimation (Direct VLM, VLM-IVW) yield pervasive pathologies: joint misconfiguration, nonphysical oscillation, excessive compliance/stiffness, and internal collisions, particularly notable in stapler, suitcase, and cabinet tasks.

Figure 3: Stapler task. Only Asset2Sim produces feasible, interactable configurations; baselines exhibit floating bodies and unactuatable joints.

Figure 4: Suitcase task. Baselines demonstrate over-stiffness or abrupt dynamics; Asset2Sim enables physically consistent closures.

Figure 5: Cabinet task. Only Asset2Sim produces stable prismatic motion; all baselines have oscillatory or immovable drawers.

The underlying cause is the lack of closed-loop correction: VLM-predicted states are only weakly constrained by physical or inter-part dependencies, compounding errors, and violating interaction-readiness.

Computational Efficiency

Despite leveraging simulator feedback, Asset2Sim maintains practical efficiency: on average, only 2.26 VLM queries per asset are needed, compared to 25 for multi-view aggregation, and orders of magnitude lower human hours than manual curation.

Implications and Future Directions

Practical Impact: The pipeline enables scalable creation of physically plausible articulated objects suitable for robot learning research and benchmarking. It dramatically reduces or eliminates manual parameter tuning bottlenecks, bridging the gap between geometric-kinematic datasets and high-fidelity manipulation simulation.

Theoretical Impact: Asset2Sim demonstrates that grounding multi-modal foundation model reasoning with iterative simulator constraints enables robust, semantically aligned, and stable asset creation. This challenges the prevailing paradigm of one-shot prediction for physical property estimation, instead advocating for interactive, feedback-driven physical reasoning.

Potential Extensions: Future work could explore further joint inference of kinematics and physical properties (especially for assets with incomplete or noisy structure), end-to-end sim-to-real policy transfer leveraging the improved forward simulation fidelity, and expanding the principles to full-scene synthesis for task curricula and environment variation in reinforcement learning.

Conclusion

Asset2Sim presents a robust, criteria-aligned, scalable pipeline for generating physically realistic articulated objects from raw 3D assets, leveraging foundation models and a simulator-in-the-loop paradigm. It closes a longstanding gap between ease of 3D asset creation and the demands of simulation-based robot learning, offering both a general evaluation protocol and a practical asset generation methodology. This work will influence large-scale robotic simulation, benchmarking, and sim-to-real tranfser for manipulation-centric research.