Training Deep Networks with Synthetic Data: Bridging the Reality Gap by Domain Randomization (1804.06516v3)

Abstract: We present a system for training deep neural networks for object detection using synthetic images. To handle the variability in real-world data, the system relies upon the technique of domain randomization, in which the parameters of the simulator$-$such as lighting, pose, object textures, etc.$-$are randomized in non-realistic ways to force the neural network to learn the essential features of the object of interest. We explore the importance of these parameters, showing that it is possible to produce a network with compelling performance using only non-artistically-generated synthetic data. With additional fine-tuning on real data, the network yields better performance than using real data alone. This result opens up the possibility of using inexpensive synthetic data for training neural networks while avoiding the need to collect large amounts of hand-annotated real-world data or to generate high-fidelity synthetic worlds$-$both of which remain bottlenecks for many applications. The approach is evaluated on bounding box detection of cars on the KITTI dataset.

Training Deep Networks with Synthetic Data: Bridging the Reality Gap by Domain Randomization

The paper assesses the potential of synthetic data enhanced through domain randomization (DR) for training deep neural networks (DNNs), particularly in object detection tasks. The work is motivated by the high cost and resource intensity associated with collecting and annotating large datasets of real images.

Domain Randomization Technique

The core contribution of the paper lies in the realization of domain randomization. This technique generates a diverse set of synthetic images by randomizing parameters such as lighting, pose, and object textures in non-photorealistic ways. The idea is that a DNN, when exposed to a wide variety of synthetic images, will learn to generalize better to real-world data.

In their experimental setup, the authors used 100,000 synthetic images generated with DR to train an object detection DNN. This dataset's breadth and variability forced the network to focus on the essential features of the objects rather than specific, real-world imaging details. Consequently, DR-synthesized images become efficient training data that effectively bridge the reality gap.

Empirical Results

The paper employs three state-of-the-art object detection networks—Faster R-CNN, R-FCN, and SSD—to evaluate the effectiveness of DR. These networks were trained on both the DR dataset and the Virtual KITTI (VKITTI) dataset, a high-fidelity synthetic dataset.

Key Results:

• Faster R-CNN trained on VKITTI achieved an average precision (AP) of 79.7%, while the same model trained on DR data achieved 78.1%.
• R-FCN showed a significant improvement when trained on DR data (71.5%) compared to VKITTI (64.6%).
• SSD also benefited more from DR, showing an AP of 46.3% vs. 36.1% with VKITTI.

Fine-Tuning on Real Data

The research found that fine-tuning the networks on real data after initial training on synthetic data yielded better results overall. Importance was placed on demonstrating that, with just a moderate amount of real-world data fine-tuning, models trained on DR-synthetic data could surpass those trained solely on real or high-fidelity synthetic data.

• DR, when fine-tuned on all available real images, achieved an AP of 98.5%, surpassing that of VKITTI fine-tuned data (96.9%) and the model trained exclusively on real data (96.4%).

Ablation Studies

The ablation studies explored the impact of various DR components:

• Excluding random textures or reducing texture variants resulted in a significant drop in performance.
• The presence of flying distractors (additional non-target objects in the training images) improved the network's ability to generalize.
• Freezing early-layer weights during training was detrimental. In fact, allowing the full network to train end-to-end demonstrated significant performance improvements.

Additionally, the effect of dataset size on performance showed that pretraining with ImageNet weights followed by DR-trained networks saturated performance metrics well, even with just 10,000 images.

Practical and Theoretical Implications

This research has several important implications:

1. Cost Reduction: DR makes it feasible to train effective DNNs without the need for extensive, high-cost, real-world data collection and annotation.
2. Generalization: Training with DR-enhanced data can yield models that generalize more robustly to new, unseen data compared to those trained with more photorealistic data.
3. Synthetic Data Use: The paper expands the potential use cases of synthetic data, advocating for broader usage in situations where real data acquisition is impractical or expensive.

Future Developments in AI

The work opens avenues for further research:

• Extending DR techniques to a broader array of objects and environments.
• Investigating the integration of DR with domain adaptation techniques to further improve generalization.
• Exploring the use of DR in different visual tasks beyond object detection, such as image segmentation and texture recognition.

Overall, this paper provides a compelling argument for the efficacy of domain randomization in training robust deep networks, promoting a shift towards more cost-effective and scalable data generation methodologies in the field of AI.