Deep Learning for Spacecraft Pose Estimation from Photorealistic Rendering (1907.04298v2)

Abstract: On-orbit proximity operations in space rendezvous, docking and debris removal require precise and robust 6D pose estimation under a wide range of lighting conditions and against highly textured background, i.e., the Earth. This paper investigates leveraging deep learning and photorealistic rendering for monocular pose estimation of known uncooperative spacecrafts. We first present a simulator built on Unreal Engine 4, named URSO, to generate labeled images of spacecrafts orbiting the Earth, which can be used to train and evaluate neural networks. Secondly, we propose a deep learning framework for pose estimation based on orientation soft classification, which allows modelling orientation ambiguity as a mixture of Gaussians. This framework was evaluated both on URSO datasets and the ESA pose estimation challenge. In this competition, our best model achieved 3rd place on the synthetic test set and 2nd place on the real test set. Moreover, our results show the impact of several architectural and training aspects, and we demonstrate qualitatively how models learned on URSO datasets can perform on real images from space.

• The paper introduces a novel deep learning framework using a high-fidelity simulator and Gaussian-based soft classification to enhance spacecraft pose estimation.
• It leverages advanced data augmentation and sim-to-real transfer techniques to overcome issues from harsh lighting and complex backgrounds.
• By employing ResNet variants and strategic hyperparameter tuning, the approach achieves competitive rankings in ESA’s synthetic and real test challenges.

Deep Learning for Spacecraft Pose Estimation from Photorealistic Rendering

The paper by Pedro F. Proença and Yang Gao presents an in-depth exploration of leveraging deep learning (DL) models and photorealistic rendering for improving spacecraft pose estimation during on-orbit proximity operations. Traditional approaches in such scenarios face substantial challenges due to harsh lighting conditions and complex backgrounds like Earth's highly textured surface. The authors propose an advanced methodological framework tackling pose estimation tasks utilizing a novel simulator (URSO) and a robust DL architecture that comprehensively addresses orientation ambiguities.

Simulator and Framework

The paper introduces URSO, a simulator developed using Unreal Engine 4, which generates labeled photorealistic datasets of spacecraft orbiting Earth. The tool enables the augmentation and testing of DL models designed for 6D spacecraft pose estimation in both synthetic and potentially real environments. The simulator's core advantage lies in its ability to provide a high-fidelity representation of space conditions, thus facilitating the training of models in a controlled setting where data is traditionally sparse and costly.

The proposed DL framework segments into two main branches for pose estimation: one addressing the 3D location and the other focused on orientation. The orientation branch is particularly innovative, using orientation soft classification instead of direct regression techniques. By adopting a Gaussian-based soft assignment classification approach, the model can effectively encapsulate uncertainties and ambiguities, such as those arising from symmetrical spacecraft designs or complex lighting conditions.

In evaluations, the framework achieved competitive placements in the ESA's satellite pose estimation challenge — third in the synthetic test set and second in the real test set. Additionally, it demonstrated successful generalization when trained on URSO-generated data and subsequently evaluated using realistic space images.

Key Findings and Implications

Several critical insights emerged from the empirical studies reported in the paper:

1. Data Augmentation: Applying random camera orientation perturbations helps combat overfitting, significantly enhancing model robustness by diversifying training scenarios. This approach is pertinent in simulating realistic operational variances experienced in space.
2. Orientation Estimation: The orientation soft classification strategy proved superior to direct regression methods. It not only improves accuracy but also provides a probabilistically interpretable output useful for post-processing and decision-making.
3. Network Architecture and Training: The choice of architecture (ResNet variants) and strategic bottleneck layer configurations are instrumental in optimizing performance while managing computational efficiency. Here, the authors suggest specific hyperparameter fine-tuning insights that can be adapted for future DL frameworks in space applications.
4. Sim-to-Real Transfer: Utilizing effective sim-to-real augmentation strategies enables the application of models from synthetic environments to actual space scenarios. Techniques such as grayscale transformation, noise addition, and dropout help bridge the domain gap.

Future Developments

The research opens avenues for multiple potential developments. The integration of temporal elements into the DL framework, leveraging recurrent architectures, may address dynamic tracking tasks. Furthermore, expansions of the simulator to incorporate more varied spacecraft models and mission scenarios would enhance diversity in dataset generation, supporting broader research purviews.

In conclusion, the research provides a pivotal contribution to the field of space robotics, establishing a novel approach to DL-based spacecraft pose estimation. It highlights key methodological advancements and offers detailed evaluations that inform both theoretical and practical undertakings in aerospace engineering.