Visual sim-to-real transfer for pixel-based locomotion policies

Establish robust and generalizable methods for visual sim-to-real transfer of locomotion and navigation policies trained from RGB pixel observations in simulation, ensuring that behaviors learned in photorealistic simulators reliably and consistently transfer to real robotic platforms without additional fine-tuning.

Background

The paper introduces GaussGym, a photorealistic, high-throughput real-to-sim framework that uses 3D Gaussian Splatting integrated with vectorized physics simulation to train locomotion and navigation policies directly from RGB pixels. While the system demonstrates initial zero-shot transfer to real-world stair climbing, the authors emphasize that closing the visual sim-to-real gap remains challenging.

The Limitations section explicitly acknowledges that visual sim-to-real transfer is a difficult and largely unsolved problem. Although GaussGym provides a scalable platform and shows promising perceptive behaviors in simulation, the authors note that more extensive experiments are needed to assess generalization and reliable real-world performance across diverse tasks and environments.