PIRATR: Parametric 3D Perception for Robotics

• PIRATR is a 3D detection framework that infers CAD-consistent object models and 6-DoF poses directly from a single LiDAR point cloud.
• Leveraging transformer-based feature extraction with PointNet++ sampling, it predicts class-specific parameters for grippers, platforms, and pallets.
• Trained on synthetic data with extensive domain randomization, the system achieves robust real-world performance even amid noise and occlusion.

Searching arXiv for PIRATR and closely related antecedent models to ground the article in current literature. arXiv_search(query="PIRATR Parametric Object Inference for Robotic Applications with Transformers in 3D Point Clouds", max_results=5, sort_by="relevance") arXiv_search(query="PI3DETR 3DETR point clouds transformer", max_results=10, sort_by="relevance") arXiv_search(query="3DETR A Simple End-to-End 3D Detection Transformer", max_results=5, sort_by="relevance") arXiv_search(query="PointNet++ Deep Hierarchical Feature Learning on Point Sets in a Metric Space", max_results=5, sort_by="relevance") arXiv_search(query="PI3DETR parametric 3D curves edges transformer", max_results=10, sort_by="relevance") PIRATR, short for Parametric Object Inference for Robotic Applications with Transformers in 3D Point Clouds, is an end-to-end 3D perception framework for robotic use cases in point clouds that takes a single LiDAR point cloud as input, detects multiple object classes, and directly predicts for each object its 6-DoF pose together with class-specific parameters (Schwingshackl et al., 5 Feb 2026). In the reported implementation, the target classes are crane grippers, loading platforms, and pallets, and the predicted parameters are applied to predefined CAD meshes to obtain fully specified 3D model instances usable for robot control rather than merely coarse 3D bounding boxes (Schwingshackl et al., 5 Feb 2026). PIRATR is presented as an extension of PI3DETR that repurposes a transformer-based parametric detection pipeline from curve-level inference to object-level pose and state estimation under sparse, noisy, and occlusion-affected outdoor LiDAR sensing, with training performed entirely in simulation and deployment on real scans from an automated forklift platform (Schwingshackl et al., 5 Feb 2026).

1. Problem setting and conceptual scope

PIRATR is formulated for heavy-duty outdoor robotics, specifically an automated forklift scenario with a crane and pallets, where the sensing modality is a Livox Mid-70 LiDAR producing noisy, sparse, and heavily occluded point clouds (Schwingshackl et al., 5 Feb 2026). The relevant downstream tasks include aligning the forklift to a pallet, docking to a loading platform, and interacting with a crane gripper, all of which require accurate 6-DoF pose estimation and, in the case of articulated tools, additional state variables such as the gripper opening (Schwingshackl et al., 5 Feb 2026).

The framework is explicitly positioned against conventional 3D object detectors that output axis-aligned or oriented 3D bounding boxes. In PIRATR, the detection target is instead a parametric object: each class is associated with a known CAD mesh and a parameter vector that specifies its world configuration (Schwingshackl et al., 5 Feb 2026). This shifts the representation from generic geometric localization to a pose-aware, CAD-consistent object model that is closer to what a planner or controller requires, including task-relevant semantics such as pallet alignment geometry or gripper articulation (Schwingshackl et al., 5 Feb 2026).

This suggests that PIRATR occupies an intermediate layer between low-level geometric perception and robot action generation. The paper describes this as “pose-aware, parameterized perception,” emphasizing that the output is intended to feed directly into world models and motion planners without extra geometry fitting or ICP-style alignment (Schwingshackl et al., 5 Feb 2026).

2. Parametric object model and task formulation

The input is a point cloud $P$ from LiDAR, after subsampling, containing $M$ object instances of interest. Each instance $i \in \{1,\dots,M\}$ has a class label $c_i \in C=\{1,2,3\}$, where $c=1$ denotes a crane gripper, $c=2$ a loading platform, and $c=3$ a pallet (Schwingshackl et al., 5 Feb 2026). The ground-truth target is a class-conditional parameter vector

$$T_i = \begin{cases} [\mathbf{p}_i,\mathbf{q}_i,\alpha_i], & c_i=1,\ [\mathbf{p}_i,\mathbf{q}_i], & c_i\in\{2,3\}, \end{cases}$$

where $\mathbf{p}_i \in \mathbb{R}^3$ is position, $\mathbf{q}_i \in \mathbb{R}^4$ is a unit quaternion, and $M$0 is the gripper opening angle (Schwingshackl et al., 5 Feb 2026).

The class-dependent semantics of $M$1 are part of the formulation rather than a post hoc convention. For grippers, $M$2 is the center of mass; for loading platforms, it is the front-right corner facing the driver cabin; for pallets, it is the bottom center point reflecting ground support (Schwingshackl et al., 5 Feb 2026). The model therefore predicts not just a rigid transform but a manipulation-relevant reference frame.

Given a class mesh $M$3 and parameters $M$4, PIRATR instantiates the object through a mapping

$M$5

which applies the pose and state configuration to the mesh and produces a compact 64-point surface representation used for geometric losses (Schwingshackl et al., 5 Feb 2026). For grippers, the opening angle is realized through a simple mechanical rule that rotates the gripper “fingers” relative to the base according to $M$6 (Schwingshackl et al., 5 Feb 2026).

The treatment of symmetry is integral to both supervision and evaluation. Grippers and pallets are treated as invariant under a 180° rotation about the vertical $M$7-axis, in addition to the usual quaternion sign symmetry, whereas loading platforms retain only quaternion sign symmetry because front/back orientation is semantically important (Schwingshackl et al., 5 Feb 2026). All positions and meshes are normalized to $M$8, and the gripper opening angle is scaled to $M$9 for training (Schwingshackl et al., 5 Feb 2026).

Class	Reference point $i \in \{1,\dots,M\}$0	Extra parameter / symmetry
Crane gripper	Center of mass	Opening angle $i \in \{1,\dots,M\}$1; 180° yaw symmetry
Loading platform	Front-right corner	No extra continuous parameter; no 180° yaw symmetry
Pallet	Bottom center point	No extra continuous parameter; 180° yaw symmetry

3. Architecture and extensibility

PIRATR follows the 3DETR/PI3DETR pattern: it downsamples and encodes the point cloud via a PointNet++-style set abstraction, applies a transformer encoder-decoder with query embeddings anchored at sampled 3D locations, and uses a shared classifier together with class-specific heads to produce object hypotheses (Schwingshackl et al., 5 Feb 2026). This places it in the family of end-to-end transformer-based 3D detection architectures derived from 3DETR (Ghosh et al., 2021) and built on PointNet++-style feature extraction (Qi et al., 2017).

The input point cloud is preprocessed to at most 32k points. A SAModule performs farthest point sampling to pick a subset of points, exemplified by 2048 points, computes local permutation-invariant features, and outputs per-point features of dimension $i \in \{1,\dots,M\}$2 (Schwingshackl et al., 5 Feb 2026). Query positions $i \in \{1,\dots,M\}$3 are sampled via farthest point sampling with $i \in \{1,\dots,M\}$4, and each query position is encoded using a sinusoidal positional embedding (Schwingshackl et al., 5 Feb 2026).

The transformer backbone is copied from PI3DETR and consists of 3 encoder layers and 9 decoder layers with token dimension 768 (Schwingshackl et al., 5 Feb 2026). The decoder consumes positional and learned query features, attends to the encoded point features, and produces output embeddings $i \in \{1,\dots,M\}$5, each of which is interpreted by the output heads (Schwingshackl et al., 5 Feb 2026).

For each output embedding $i \in \{1,\dots,M\}$6, PIRATR applies a shared classification head $i \in \{1,\dots,M\}$7 that predicts probabilities over four classes,

$i \in \{1,\dots,M\}$8

corresponding to no-object, gripper, loading platform, and pallet (Schwingshackl et al., 5 Feb 2026). In parallel, three class-specific regression heads map the same token to parameter vectors. For grippers,

$i \in \{1,\dots,M\}$9

where the position is predicted as an offset from the query point,

$c_i \in C=\{1,2,3\}$0

the quaternion is normalized, and $c_i \in C=\{1,2,3\}$1 is the normalized opening angle (Schwingshackl et al., 5 Feb 2026). Loading-platform and pallet heads predict $c_i \in C=\{1,2,3\}$2 with class-dependent semantics (Schwingshackl et al., 5 Feb 2026).

A notable architectural feature is modular extensibility. To add a new object class, the procedure is to provide a CAD mesh $c_i \in C=\{1,2,3\}$3 and a parametrization $c_i \in C=\{1,2,3\}$4, add a new FFN head $c_i \in C=\{1,2,3\}$5, extend the classifier, and introduce symmetry handling and a corresponding $c_i \in C=\{1,2,3\}$6 mapping for the Chamfer loss, without changing the transformer core or point processing pipeline (Schwingshackl et al., 5 Feb 2026). This suggests that PIRATR is designed as a general parametric-object detector rather than a fixed three-class model.

4. Training pipeline, synthetic data, and supervision

All training is performed purely on synthetic data generated in Blender (Schwingshackl et al., 5 Feb 2026). The synthetic environment contains geometrically accurate CAD models of the forklift, truck with loading crane and gripper, and pallets, while omitting small details such as cables (Schwingshackl et al., 5 Feb 2026). The crane is fixed in world coordinates; the supervision targets are the gripper, loading platform, and pallets (Schwingshackl et al., 5 Feb 2026).

Scene generation uses extensive domain randomization. The forklift pose is sampled on a circle of radius 5–16 m around the crane, with orientation toward one of 8 predefined points of interest on the truck or away from it (Schwingshackl et al., 5 Feb 2026). The LiDAR sensor is mounted on the forklift mast, so mast motion naturally varies sensor height and tilt (Schwingshackl et al., 5 Feb 2026). The gripper pose is sampled on a circle of radius 3.5–8 m around the back of the truck with random height from 0.5–4.5 m, random orientation, and random opening angle $c_i \in C=\{1,2,3\}$7 (Schwingshackl et al., 5 Feb 2026). Pallets are placed using Poisson-disk sampling modulated by fractal Perlin noise, may contain boxes or other pallets, and can be stacked up to two high, while vegetation and random wall meshes are added to simulate outdoor clutter (Schwingshackl et al., 5 Feb 2026).

The LiDAR simulation reproduces the Livox Mid-70 non-repetitive scanning pattern by using official Livox-SDK sample data consisting of timestamp, azimuth, and elevation sequences for more than 400k rays, converting spherical angles to direction vectors, and raycasting in Blender Geometry Nodes up to a maximum distance of 25 m (Schwingshackl et al., 5 Feb 2026). The resulting raw simulated point cloud is subsampled to 32k points via farthest point sampling, and objects with fewer than a class-specific count of hits are dropped from the ground truth because they are too occluded (Schwingshackl et al., 5 Feb 2026).

The supervision combines classification, parameter regression, and mesh-level geometric consistency. Quaternion ambiguity is handled through a symmetry-aware loss

$c_i \in C=\{1,2,3\}$8

where the symmetry set $c_i \in C=\{1,2,3\}$9 depends on the class (Schwingshackl et al., 5 Feb 2026). The geometry-aware parameter loss adds position error, symmetry-aware quaternion error, and opening-angle error for grippers (Schwingshackl et al., 5 Feb 2026). In addition, a Chamfer distance is computed between the 64-point surface samples of predicted and ground-truth posed meshes,

$c=1$0

so that training explicitly penalizes whole-object misalignment rather than only reference-point discrepancies (Schwingshackl et al., 5 Feb 2026).

Prediction-to-target assignment is performed via a geometry-aware Hungarian matching with cost

$c=1$1

yielding a bipartite matching between $c=1$2 ground-truth targets and $c=1$3 predictions; unmatched predictions are treated as no-object (Schwingshackl et al., 5 Feb 2026). The total training loss is

$c=1$4

with weighted cross-entropy for class imbalance and geometric terms for matched objects (Schwingshackl et al., 5 Feb 2026).

Training is implemented in PyTorch and initialized from a PI3DETR checkpoint (Schwingshackl et al., 5 Feb 2026). The schedule consists of 330 epochs with AdamW at learning rate $c=1$5, followed by 100 additional epochs at $c=1$6, with effective batch size 132 via gradient accumulation and gradient clipping with L2-norm 0.2 (Schwingshackl et al., 5 Feb 2026). Data augmentation includes random rotations up to $c=1$7 around $c=1$8 and $c=1$9, and with probability $c=2$0, Gaussian point noise with standard deviation in $c=2$1 in normalized coordinates (Schwingshackl et al., 5 Feb 2026).

5. Inference, deployment, and empirical results

At inference time on the real robotic platform, the Livox Mid-70 provides 10 “packages” of 10k points per second through a ROS2 interface (Schwingshackl et al., 5 Feb 2026). The preprocessing pipeline removes noisy points based on intensity and spatial stability, discards points beyond 25 m, subsamples point clouds in the 50k–400k range down to 32k via farthest point sampling, rotates points by 180° around the $c=2$2-axis to match Blender’s training convention, runs the network, and then rotates predicted poses back by $c=2$3 around $c=2$4 to the ROS2/world convention (Schwingshackl et al., 5 Feb 2026). The output for each detected object is a class label, position, orientation, and, for grippers, opening angle (Schwingshackl et al., 5 Feb 2026).

Because these outputs correspond to parametric CAD instantiations, they can be consumed directly by robot-control logic. For pallets, the bottom-center pose and orientation can be used to align forks and plan entry into pallet slots; for loading platforms, the front-right corner and pose indicate where the forklift must park or where the loading surface lies; for grippers, the framework yields both global pose and opening state, supporting reasoning about grasp configuration and collision prediction with the semi-articulated mesh (Schwingshackl et al., 5 Feb 2026).

Evaluation uses a synthetic dataset of 5000 scenes and a real dataset of 73 scenes collected with the real Livox Mid-70 on the forklift, manually annotated for 34 grippers, 32 loading platforms, and 70 pallets (Schwingshackl et al., 5 Feb 2026). Geometric quality is measured by position $c=2$5 error, geodesic orientation error, yaw error, and gripper opening-angle error, while detection accuracy is measured by per-class AP and mAP, with a prediction counted as correct when the class matches and the Chamfer distance between posed meshes is below $c=2$6 (Schwingshackl et al., 5 Feb 2026).

On the synthetic test set, PIRATR reports mean AP $c=2$7, with AP values of 0.997 for grippers, 0.990 for loading platforms, and 0.958 for pallets (Schwingshackl et al., 5 Feb 2026). Mean position errors are 0.10 m for grippers, 0.12 m for loading platforms, and 0.12 m for pallets; mean geodesic orientation errors are 4.90°, 0.92°, and 12.29°, respectively; and mean gripper opening error is 6.59° (Schwingshackl et al., 5 Feb 2026). The paper notes that pallets are the hardest category, plausibly because of greater variation, clutter, and stacking (Schwingshackl et al., 5 Feb 2026).

On real outdoor LiDAR scans, without additional fine-tuning, the reported detection performance is mAP $c=2$8, with AP values of 0.962 for grippers, 0.988 for loading platforms, and 0.805 for pallets (Schwingshackl et al., 5 Feb 2026). Mean position errors are 0.13 m for grippers, 0.24 m for loading platforms, and 0.14 m for pallets; mean geodesic orientation errors are 6.37°, 1.57°, and 15.18°; mean yaw errors are 2.46°, 1.19°, and 12.32°; and mean gripper opening error is 6.69° (Schwingshackl et al., 5 Feb 2026). Boxplots reported in the paper indicate that more than 50% of angle errors are below 10° and more than 50% of position errors are below 20 cm for all classes (Schwingshackl et al., 5 Feb 2026).

Qualitative results show predicted gripper, platform, and pallet meshes aligned with real scenes even under occlusions, clutter, and vegetation (Schwingshackl et al., 5 Feb 2026). Robustness tests further indicate accurate predictions across a range of distances, heights, rotations, and gripper openings, with a documented failure mode for an extreme close-up loading platform at 80° rotation (Schwingshackl et al., 5 Feb 2026). Additional experiments show imperfect stack detection for pallets carrying unseen cylindrical objects and retained detection capability under human occlusion despite the absence of such cases in training (Schwingshackl et al., 5 Feb 2026).

Runtime, including the full pipeline from raw input to final predictions on an RTX 4090, is approximately 218 ms at 50k raw points, 371 ms at 200k raw points, and 598 ms at 400k raw points (Schwingshackl et al., 5 Feb 2026). The paper characterizes this as operation in the hundreds-of-milliseconds range, suitable for near-real-time deployment in slow-moving heavy machinery (Schwingshackl et al., 5 Feb 2026).

6. Relation to prior work, limitations, and prospective extensions

PIRATR inherits its architectural template from transformer-based 3D point-cloud detection, especially 3DETR (Ghosh et al., 2021), and more directly from PI3DETR, whose core architecture and geometry-aware matching ideas are repurposed from parametric edge detection to object-level pose and state estimation (Schwingshackl et al., 5 Feb 2026). Relative to mainstream LiDAR detectors that rely on voxel-based, pillar-based, BEV, or point-based pipelines and typically output 3D boxes, PIRATR operates directly on raw point clouds, avoids intermediate voxel/BEV representations, and predicts parametric CAD-consistent object shapes rather than only boxes (Schwingshackl et al., 5 Feb 2026).

The framework also differs from primitive-fitting and CAD-reconstruction systems based on RANSAC or learned primitive detection. Those methods produce compact geometric primitives such as planes, cylinders, or edges, but are described in the paper as usually not being multi-class object detectors operating on semantic objects in cluttered outdoor scenes (Schwingshackl et al., 5 Feb 2026). PIRATR instead binds geometry to semantics, pose, and task-relevant state variables in a single detection architecture (Schwingshackl et al., 5 Feb 2026).

The paper highlights five main contributions: a generic, fully differentiable, end-to-end network for 3D multi-class parametric multi-object detection and 6-DoF pose estimation; a synthetic LiDAR data-generation pipeline with realistic Livox scanning and domain randomization; explicit integration of object state estimation through class-specific FFNs; quantitative synthetic and real evaluation with robustness tests; and deployment on an outdoor autonomous forklift performing loading tasks (Schwingshackl et al., 5 Feb 2026). The associated code is available at the project repository listed in the paper (Schwingshackl et al., 5 Feb 2026).

Several limitations are stated or implied. PIRATR depends on accurate CAD meshes and clear parametrizations for each class, which constrains it to object categories with known geometry and low-dimensional configuration spaces (Schwingshackl et al., 5 Feb 2026). The experimental scope covers only three classes, all of relatively fixed geometry apart from pose and gripper opening (Schwingshackl et al., 5 Feb 2026). Although robustness to occlusion is strong, failures remain under extreme clutter or unseen load configurations, particularly for stacked pallets with novel objects on top (Schwingshackl et al., 5 Feb 2026). The training setup is sensor-specific to the Livox Mid-70 and would require re-simulating scan pattern and noise for transfer to different LiDARs (Schwingshackl et al., 5 Feb 2026). Runtime is acceptable for slow heavy machinery but would likely require optimization for faster platforms or lower-end hardware (Schwingshackl et al., 5 Feb 2026).

Future work proposed in the paper includes extending the supported object classes, improving synthetic realism to further strengthen sim-to-real generalization, reducing training time through more efficient architectures or strategies, and incorporating temporal information from multiple frames for higher robustness and accuracy (Schwingshackl et al., 5 Feb 2026). A plausible implication is that PIRATR is best understood not as a finished perception stack, but as a concrete blueprint for simulation-trained, transformer-based parametric object perception in robotics, with CAD-grounded outputs tailored to manipulation and docking tasks rather than generic scene understanding (Schwingshackl et al., 5 Feb 2026).