SplatSim: Zero-Shot Sim2Real Transfer of RGB Manipulation Policies Using Gaussian Splatting (2409.10161v3)

Abstract: Sim2Real transfer, particularly for manipulation policies relying on RGB images, remains a critical challenge in robotics due to the significant domain shift between synthetic and real-world visual data. In this paper, we propose SplatSim, a novel framework that leverages Gaussian Splatting as the primary rendering primitive to reduce the Sim2Real gap for RGB-based manipulation policies. By replacing traditional mesh representations with Gaussian Splats in simulators, SplatSim produces highly photorealistic synthetic data while maintaining the scalability and cost-efficiency of simulation. We demonstrate the effectiveness of our framework by training manipulation policies within SplatSim and deploying them in the real world in a zero-shot manner, achieving an average success rate of 86.25%, compared to 97.5% for policies trained on real-world data. Videos can be found on our project page: https://splatsim.github.io

• The paper introduces a novel framework that uses Gaussian Splatting to create highly realistic synthetic data, effectively bridging the gap between simulated and real-world RGB manipulation tasks.
• The experimental validation across four manipulation tasks, with success rates ranging from 70% to 95% (average 86.25%), demonstrates the scalable and cost-effective potential of the approach.
• The study highlights future opportunities for integrating high-fidelity synthetic data with reinforcement learning to enhance robotic performance in diverse and challenging real-world scenarios.

SplatSim: Zero-Shot Sim2Real Transfer of RGB Manipulation Policies Using Gaussian Splatting

SplatSim, presented by Qureshi et al. from Carnegie Mellon University, addresses the persistent challenge in robotics: transitioning manipulation policies trained in simulation to real-world environments (Sim2Real), particularly when using RGB images. The conventional gap between synthetic and real-world visual data significantly hampers the efficacy of such policies. This paper introduces a novel framework leveraging Gaussian Splatting to bridge this gap, offering a promising approach for zero-shot Sim2Real transfer.

Framework and Methodology

Gaussian Splatting, the cornerstone of SplatSim, replaces traditional mesh representations in simulators to achieve photorealistic synthetic data, which is crucial for effective Sim2Real transfer. The primary challenge in using RGB images for robot learning lies in the stark visual differences between simulated and real-world environments. Gaussian Splatting addresses this by generating highly detailed and photorealistic renderings, thereby reducing this visual discrepancy.

Key contributions of the framework include:

• Scalable Data Generation: SplatSim generates photorealistic environments using Gaussian Splats, supplanting traditional mesh-based rendering.
• Zero-World Data Dependency: The framework eliminates the need for extensive real-world data collection by relying solely on initial video captures and existing simulation data.
• Zero-Shot Transfer: Policies trained entirely on synthetic data produced by SplatSim demonstrate robust performance in real-world settings without necessitating further adaptation.

Experimental Validation

The effectiveness of SplatSim is validated through rigorous experiments across four distinct manipulation tasks: T-Push, Pick-Up-Apple, Orange on Plate, and Assembly tasks. Notably, these tasks are designed to cover a broad spectrum of manipulation complexities and interactions.

• T-Push Task: This task involves non-prehensile manipulation, a critical aspect of real-world robotic applications. The SplatSim-trained policy achieved a 90% success rate, indicating substantial zero-shot transfer efficacy.
• Pick-Up-Apple Task: The framework exhibited a 95% success rate in this object grasping and manipulation task, underscoring the high fidelity of its generated synthetic data.
• Orange on Plate Task: Achieving a 90% success rate, this task further illustrates the ability of SplatSim to handle precise object placements.
• Assembly Task: Although more complex, with an average success rate of 70%, the task showcases the framework’s ability to deal with intricate interactions.

In these experiments, the overall average success rate for SplatSim-trained policies was 86.25%, compared to 97.5% for policies trained on real-world data. This performance highlights the potential of SplatSim as a viable alternative to labor-intensive real-world training.

Implications and Future Directions

The practical implications of SplatSim are profound. It offers a scalable and cost-effective solution to the Sim2Real challenge, enabling the development and deployment of robust robotic manipulation policies with significantly reduced dependence on real-world data. This capability is particularly advantageous for tasks where real-world data acquisition is challenging, hazardous, or economically infeasible, such as in agriculture or underwater exploration.

Theoretically, this research contributes to the broader understanding of how high-fidelity synthetic data can enhance learning and generalization in robotics. It demonstrates the substantial impact of visual realism in bridging the domain gap between simulated and real environments.

Future research directions could explore the integration of SplatSim with reinforcement learning frameworks to enhance the adaptability and skill generalization of robotic policies in diverse real-world scenarios. Enhancements to handle complex, non-rigid objects such as clothes or biological materials could represent another significant advance, broadening the applicability of the framework.

In summary, SplatSim offers a robust, scalable solution to the Sim2Real problem for RGB-based manipulation tasks. By leveraging Gaussian Splatting, this framework represents a significant step forward in deploying effective robotic policies in real-world environments, facilitating advancements in various application domains.