Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World (1703.06907v1)

Abstract: Bridging the 'reality gap' that separates simulated robotics from experiments on hardware could accelerate robotic research through improved data availability. This paper explores domain randomization, a simple technique for training models on simulated images that transfer to real images by randomizing rendering in the simulator. With enough variability in the simulator, the real world may appear to the model as just another variation. We focus on the task of object localization, which is a stepping stone to general robotic manipulation skills. We find that it is possible to train a real-world object detector that is accurate to $1.5$cm and robust to distractors and partial occlusions using only data from a simulator with non-realistic random textures. To demonstrate the capabilities of our detectors, we show they can be used to perform grasping in a cluttered environment. To our knowledge, this is the first successful transfer of a deep neural network trained only on simulated RGB images (without pre-training on real images) to the real world for the purpose of robotic control.

• The paper introduces domain randomization to train deep neural networks in simulation, effectively bridging the reality gap for robotic object detection.
• It employs a modified VGG-16 architecture with randomized textures, distractors, and camera positions, achieving a localization error of just 1.5 cm.
• The approach is validated with a Fetch robot in cluttered environments, highlighting its practical advantages for sim-to-real transfer in robotics.

Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World

The paper under review presents a method known as "domain randomization" to address the challenge of transferring deep neural networks for robotic applications from simulated environments to real-world scenarios. The core motivation behind this research is to bridge the 'reality gap' which exists due to discrepancies between simulated and real environments, particularly in robotics. By leveraging domain randomization, the authors aim to provide a simple yet effective approach to make the real world appear as just another variation of the simulated environment to the trained model.

Overview

The paper specifically focuses on training object detectors using domain randomization. Object localization is chosen as the primary task, which is a fundamental aspect of robotic manipulation. The authors demonstrate that a deep neural network trained solely on simulated images, without any real-world pre-training, can accurately detect and localize objects to within \SI{1.5}{\centi\meter} in real-world images. The method showcases robustness to both distractors and partial occlusions.

Methodology

Domain Randomization

The approach involves randomizing various aspects of the simulation environment during training to expose the network to a broad range of conditions. This includes:

• Number and shape of distractor objects.
• Position and texture of all objects.
• Camera position, orientation, and lighting conditions.
• Type and amount of random noise added to images.

Random textures are generated using simple algorithmic methods, avoiding the need for realistic textures. Training on these diversified simulated images allows the trained network to generalize to real-world images without further training.

Model Architecture

The object detector is based on a modified VGG-16 convolutional neural network. The adaptation involves using smaller fully connected layers and the exclusion of dropout. Training is conducted using the Adam optimizer with a low learning rate of $1 \mathrm{e}{-4}$ to promote better convergence and avoid local optima.

Experiments

Evaluation of Localization Accuracy

The object detectors were tested across various geometric shapes under different conditions such as the presence of distractors and partial occlusions. Results indicated an average localization error of \SI{1.5}{\centi\meter}, demonstrating the efficacy of the trained models in real-world scenarios. The authors conducted extensive experiments to determine the influence of pre-training, amount of simulated data, and diversity of textures on the performance.

Ablation Study

Sensitivity to factors such as the number of textures, camera randomization, and the inclusion of distractors during training was examined. It was found that:

• A large variety of textures improves real-world performance significantly.
• Randomizing the camera position provided modest benefits but was not critical.
• Training with distractors is essential to achieve robustness in real-world situations.

Robotic Manipulation

To illustrate the practical applications, the trained object detectors were integrated with a Fetch robot for grasping tasks. The system was able to successfully detect and grasp objects in cluttered environments in 95% of the tested cases.

Implications and Future Directions

The implications of this research are manifold. The proposed domain randomization technique simplifies the process of transferring learned behaviors from simulation to the real world, reducing the need for extensive real-world data collection and system identification. This can accelerate the implementation of deep learning models in real-world robotic applications.

Future research could explore several enhancements:

• Incorporation of higher resolution and depth information for improved precision.
• Optimization of network architectures tailored for specific robotic tasks.
• Combining domain randomization with domain adaptation techniques to further enhance performance across diverse real-world environments.

In conclusion, domain randomization presents a promising approach for training deep neural networks in simulated environments and effectively transferring them to real-world applications. This work lays a foundation for future advancements in seamless sim-to-real transfer, particularly in robotic manipulation tasks.