PIRATR: Parametric Object Inference for Robotic Applications with Transformers in 3D Point Clouds

Abstract: We present PIRATR, an end-to-end 3D object detection framework for robotic use cases in point clouds. Extending PI3DETR, our method streamlines parametric 3D object detection by jointly estimating multi-class 6-DoF poses and class-specific parametric attributes directly from occlusion-affected point cloud data. This formulation enables not only geometric localization but also the estimation of task-relevant properties for parametric objects, such as a gripper's opening, where the 3D model is adjusted according to simple, predefined rules. The architecture employs modular, class-specific heads, making it straightforward to extend to novel object types without re-designing the pipeline. We validate PIRATR on an automated forklift platform, focusing on three structurally and functionally diverse categories: crane grippers, loading platforms, and pallets. Trained entirely in a synthetic environment, PIRATR generalizes effectively to real outdoor LiDAR scans, achieving a detection mAP of 0.919 without additional fine-tuning. PIRATR establishes a new paradigm of pose-aware, parameterized perception. This bridges the gap between low-level geometric reasoning and actionable world models, paving the way for scalable, simulation-trained perception systems that can be deployed in dynamic robotic environments. Code available at https://github.com/swingaxe/piratr.

• The paper presents PIRATR, an extension of PI3DETR, enabling robust parametric object detection and multi-class 6-DoF pose estimation in 3D point clouds.
• It employs geometry-aware Transformer networks and synthetic data generation to overcome occlusion challenges in real-world robotic deployments.
• Experimental results show a mAP of 0.919, validating PIRATR’s synthetic-to-real transferability and performance in dynamic outdoor environments.

PIRATR: Parametric Object Inference for Robotic Applications with Transformers in 3D Point Clouds

Introduction

The paper "PIRATR: Parametric Object Inference for Robotic Applications with Transformers in 3D Point Clouds" (2602.05557) introduces an innovative method for 3D object detection in robotic applications using LiDAR-derived point clouds. The proposed system, PIRATR, builds upon the existing PI3DETR framework, enhancing its capabilities to manage occlusion-affected data and multi-class 6-DoF pose estimation. This approach leverages the strengths of 3DETR, employing modular class-specific heads that predict parametric objects like grippers, loading platforms, and pallets. PIRATR's capability to detect these objects directly from synthetic training data without additional real-world fine-tuning dramatically reduces the complexity and time associated with preparing autonomous systems for real-world applications.

PIRATR is designed to operate efficiently in environments characterized by skilled labor shortages, such as construction sites where automated forklifts and cranes are becoming increasingly vital. With the use of a LiDAR sensor, the framework avoids common issues tied to vision-based systems, such as dependency on lighting conditions, making PIRATR a robust solution for dynamic outdoor environments.

Figure 1: Image of the autonomous forklift operating in an outdoor environment, where our parametric object detection method is tested and deployed. The main detection targets such as the gripper, loading platform, and pallets are visible to provide understanding of the perception task and context.

Methodology

Synthetic Data Generation

A significant aspect of PIRATR's development was the creation of synthetic training data, leveraging Blender to simulate diverse environmental scenarios without manual intervention. This process included the generation of approximately 5,000 samples characterized by realistic simulation of occlusions and sensor-specific artifacts (Figure 2). Ground-truth data was established using carefully parameterized CAD models for accuracy in training.

The synthetic dataset was pivotal for training the model, simulating a variety of interactions and components crucial for accurate detection in real-world conditions. The use of non-repetitive LiDAR scanning mechanisms aimed to replicate real-world complexity, ensuring the data collected supported robust model training.

Figure 2: Left: reference image from the forklift-mounted camera. Center: input point cloud captured with a Livox Mid70 LiDAR. Right: 3D annotations of the supervision targets: gripper, loading platform, and pallets.

PIRATR Framework

PIRATR extends the PI3DETR framework by incorporating geometry-aware matching and prediction feed-forward networks, which allows for handling incomplete point clouds resulting from occlusions. This design facilitates the detection and classification of parametric objects, offering flexibility and easy integration of new object categories by simply updating the class-specific heads without re-designing the pipeline.

The network architecture utilizes Transformers for processing input point clouds, transforming sampled queries into output embeddings which are then interpreted for class-specific configurations. This operation is crucial for not just geometric positioning but also for estimating task-relevant properties, such as gripper opening angles.

Real-World Testing and Deployment

The framework's efficacy was demonstrated on an autonomous forklift equipped with a Livox Mid-70 LiDAR, achieving impressive results in detection precision. PIRATR maintained strong performance in outdoor scenarios without further tuning, validating synthetic-to-real transferability. The robustness tests further showcased the model's reliability under various occlusion situations, emphasizing its readiness for real-world deployments.

Figure 3: Qualitative synthetic-to-real prediction of PIRATR, which is trained solely on synthetic data and evaluated on real scans. Predicted classes: gripper (yellow), loading platforms (cyan), and pallets (magenta).

Experimental Results

Experiments conducted revealed that PIRATR achieves a mean average precision (mAP) of 0.919 on real-world datasets, with gripper, loading platform, and pallet classes showing significant performance consistency between synthetic and real data evaluations (Figure 4).

The framework's performance in occlusion scenarios, where target objects were partially obscured, illustrates PIRATR's robust detection capabilities and adaptability, laying groundwork for broader applications in complex environments where visibility is compromised.

Figure 4: Boxplots of synthetic-to-real evaluation error distributions for angles and distances.

Conclusion

PIRATR introduces an effective and scalable approach for parametric object detection and pose estimation in 3D point clouds, suitable for dynamic robotic environments. With its ability to train solely on synthetic data and generalize to real-world scenarios without fine-tuning, PIRATR significantly reduces operational constraints and setup time for deploying autonomous systems.

Future research directions may explore expanding PIRATR's object type support, enhancing synthetic data realism, and integrating temporal dynamics for further robustness. As the demand for automation in various sectors grows, PIRATR sets a benchmark for efficient, real-world deployable perception systems.

Figure 5: Synthetic-to-real robustness evaluation on gripper predictions.

Figure 6: Synthetic-to-real predictions under occlusion scenarios caused by a human standing in front of the objects, shown from left to right: gripper, loading platform, and pallet.

Figure 7: Synthetic-to-real robustness evaluation on loading platform predictions.

Figure 8: Synthetic-to-real robustness evaluation on pallet predictions.