Real2Render2Real: Scaling Robot Data Without Dynamics Simulation or Robot Hardware (2505.09601v1)

Abstract: Scaling robot learning requires vast and diverse datasets. Yet the prevailing data collection paradigm-human teleoperation-remains costly and constrained by manual effort and physical robot access. We introduce Real2Render2Real (R2R2R), a novel approach for generating robot training data without relying on object dynamics simulation or teleoperation of robot hardware. The input is a smartphone-captured scan of one or more objects and a single video of a human demonstration. R2R2R renders thousands of high visual fidelity robot-agnostic demonstrations by reconstructing detailed 3D object geometry and appearance, and tracking 6-DoF object motion. R2R2R uses 3D Gaussian Splatting (3DGS) to enable flexible asset generation and trajectory synthesis for both rigid and articulated objects, converting these representations to meshes to maintain compatibility with scalable rendering engines like IsaacLab but with collision modeling off. Robot demonstration data generated by R2R2R integrates directly with models that operate on robot proprioceptive states and image observations, such as vision-language-action models (VLA) and imitation learning policies. Physical experiments suggest that models trained on R2R2R data from a single human demonstration can match the performance of models trained on 150 human teleoperation demonstrations. Project page: https://real2render2real.com

Real2Render2Real: Scaling Robot Data Generation Without Dynamics Simulation or Robot Hardware

The paper presents Real2Render2Real (R2R2R), an innovative approach to generate extensive and diverse datasets for robot training, circumventing the dependency on traditional object dynamics simulation or teleoperation of robot hardware. This method leverages general-purpose computational techniques to scale up data collection for robot learning. R2R2R aims to address the apparent limitations associated with prevailing robot data collection methods, primarily characterized by their labor-intensive and hardware-dependent nature.

R2R2R takes a novel approach by utilizing a smartphone-captured multi-view scan of objects alongside a single human demonstration video. From these inputs, R2R2R synthesizes high-fidelity, robot-agnostic demonstrations. The core components involved are the reconstruction of detailed 3D object geometry and appearance, followed by the tracking of 6-DoF object motion. The system manages trajectory synthesis through the use of 3D Gaussian Splatting (3DGS) and converts these representations to meshes ensuring compatibility with scalable rendering engines such as IsaacLab.

Key Methodological Aspects

1. Simulation-Free Data Generation: R2R2R directly sets object and robot poses per frame using photorealistic rendering engines by defining all objects as kinematic bodies, avoiding the complexities of dynamics simulation.
2. Scalable Rendering: Employing the NVIDIA RTX 4090 GPU, R2R2R can generate trajectories at approximately 27 times the speed of human teleoperation autonomously, facilitating the generation of vast datasets without necessitating any robot hardware.
3. Policy Learning: The system integrates well with vision-language-action models and imitation learning policies, adapting effectively for models that have been traditionally reliant on teleoperated data.
4. Task and Environment Diversity: R2R2R demonstrates its effectiveness across diverse tasks—single-object handling, multi-object interaction, manipulating articulated objects, and even handling bimanual coordination—while incorporating varied environmental settings for improved robustness and generalization.

Experimental Findings and Implications

Results from 1,050 physical robot experiments reveal that policies trained on R2R2R-generated data match or even surpass the efficacy achieved with extensive human teleoperated demonstrations. This signifies R2R2R’s capacity for producing scalable, high-throughput data generations which potentially diminish the requirement for physical robot interactions in the initial data collection phase. The experiments highlight that R2R2R-derived models achieve superior success in manipulation tasks as they scale, further suggesting a cost-effective alternative of obtaining abundant training data for imitation learning.

Future Directions

The implications of this framework are vast in artificial intelligence and robotics research, offering a feasible path to alleviating data scarcity without traditional physical constraints. The work opens multiple avenues for further exploration, such as integrating physics simulations to enhance fidelity in dynamic object manipulation, expanding its applicability to more diverse manipulation strategies beyond prehensile object handling, and integrating finer grasping mechanics for varied robot hands.

In summary, R2R2R signifies a transformative strategy within robot learning pipelines by advocating for a scalable, internet-enabled approach to data collection. Its methodology promises extensive applications and improvement in deploying generalist robot policies across complex, dynamic, and visually rich environments.