RoboPEPP: Vision-Based Robot Pose and Joint Angle Estimation through Embedding Predictive Pre-Training (2411.17662v2)

Abstract: Vision-based pose estimation of articulated robots with unknown joint angles has applications in collaborative robotics and human-robot interaction tasks. Current frameworks use neural network encoders to extract image features and downstream layers to predict joint angles and robot pose. While images of robots inherently contain rich information about the robot's physical structures, existing methods often fail to leverage it fully; therefore, limiting performance under occlusions and truncations. To address this, we introduce RoboPEPP, a method that fuses information about the robot's physical model into the encoder using a masking-based self-supervised embedding-predictive architecture. Specifically, we mask the robot's joints and pre-train an encoder-predictor model to infer the joints' embeddings from surrounding unmasked regions, enhancing the encoder's understanding of the robot's physical model. The pre-trained encoder-predictor pair, along with joint angle and keypoint prediction networks, is then fine-tuned for pose and joint angle estimation. Random masking of input during fine-tuning and keypoint filtering during evaluation further improves robustness. Our method, evaluated on several datasets, achieves the best results in robot pose and joint angle estimation while being the least sensitive to occlusions and requiring the lowest execution time.

• The paper presents RoboPEPP, a vision-based method for estimating robot pose and joint angles by integrating physical model knowledge into a self-supervised embedding predictive pre-training framework.
• RoboPEPP demonstrates superior performance and reduced execution time compared to previous methods across various robot datasets, proving robust against occlusions and truncations.
• This approach sets a new standard for robust robot perception, enabling more seamless integration of robots in collaborative settings and opening avenues for applications in dynamic prediction and imitation learning.

An Overview of RoboPEPP: Vision-Based Robot Pose and Joint Angle Estimation

RoboPEPP presents an innovative approach to vision-based robot pose and joint angle estimation, addressing challenges in scenarios where robots exhibit unknown joint angles. This method is particularly relevant for applications in collaborative robotics and human-robot interactions. The paper critiques current frameworks that utilize neural network encoders for extracting image features to predict joint angles and robot pose, highlighting their limitations during occlusions and truncations.

Novel Contributions

The central contribution of RoboPEPP lies in its integration of the robot's physical model knowledge into the encoder using a self-supervised, masking-based embedding-predictive architecture. By masking the robot's joints and predicting embeddings from unmasked regions, the encoder's understanding of the robot's physical structure is significantly enhanced. This pre-trained encoder-predictor pair is further fine-tuned with joint angle prediction networks, ensuring robustness against occlusions through random input masking and keypoint filtering during evaluation. RoboPEPP distinguishes itself from prior work by demonstrating superior performance on several datasets and offering reduced execution time compared to existing methodologies.

Experimental Evaluation

The robustness of RoboPEPP is thoroughly validated across diverse datasets, emphasizing scenarios involving real-world and synthetic visual data of robots such as the Franka Emika Panda, Kuka iiwa7, and Rethink Robotics Baxter. The method outperforms traditional approaches like DREAM variants, RoboPose, and real-time holistic frameworks that assume known joint angles. This analysis is supported by quantitative outcomes using the ADD metric and qualitative comparisons that illustrate excellent alignment between predicted and actual robot structures, even in occluded images.

Implications and Future Directions

RoboPEPP sets a benchmark for robust robot perception, emphasizing the significance of embedding robot physicality in predictive tasks. The paper leaves the door open for future exploration into more dynamic settings where real-time adaptations to environmental changes are required. Furthermore, the paper hints at avenues for extending this methodology into broader robotics applications, such as dynamic prediction and imitation learning, contingent on advancements in embedding predictive architectures.

Conclusion

RoboPEPP's embedding predictive pre-training framework represents a substantial stride in enhancing robot pose estimation under occlusion-heavy scenarios. By fostering a deeper understanding of physical models within the encoder, this methodology underscores an improvement in both robustness and computational efficiency. This portends broader implications in AI-driven robotics, paving paths for seamless integration of robots in collaborative environments, thus aligning with the forward trajectory of autonomous systems development.