---
title: 'PIRATR: Parametric 3D Perception for Robotics'
url: https://www.emergentmind.com/topics/piratr
type: topic
---

# PIRATR: Parametric 3D Perception for Robotics

Searching arXiv for PIRATR and closely related antecedent models to ground the article in current literature.
arXiv_search(query="PIRATR Parametric Object Inference for Robotic Applications with Transformers in 3D Point Clouds", max_results=5, sort_by="relevance")
arXiv_search(query="PI3DETR 3DETR point clouds transformer", max_results=10, sort_by="relevance")
arXiv_search(query="3DETR A Simple End-to-End 3D Detection Transformer", max_results=5, sort_by="relevance")
arXiv_search(query="PointNet++ Deep Hierarchical Feature Learning on Point Sets in a Metric Space", max_results=5, sort_by="relevance")
arXiv_search(query="PI3DETR parametric 3D curves edges transformer", max_results=10, sort_by="relevance")
PIRATR, short for *Parametric Object Inference for Robotic Applications with Transformers in 3D Point Clouds*, is an end-to-end 3D perception framework for robotic use cases in point clouds that takes a single LiDAR point cloud as input, detects multiple object classes, and directly predicts for each object its 6-DoF pose together with class-specific parameters [2602.05557]. In the reported implementation, the target classes are crane grippers, loading platforms, and pallets, and the predicted parameters are applied to predefined CAD meshes to obtain fully specified 3D model instances usable for robot control rather than merely coarse 3D bounding boxes [2602.05557]. PIRATR is presented as an extension of PI3DETR that repurposes a transformer-based parametric detection pipeline from curve-level inference to object-level pose and state estimation under sparse, noisy, and occlusion-affected outdoor LiDAR sensing, with training performed entirely in simulation and deployment on real scans from an automated forklift platform [2602.05557].

## 1. Problem setting and conceptual scope

PIRATR is formulated for heavy-duty outdoor robotics, specifically an automated forklift scenario with a crane and pallets, where the sensing modality is a Livox Mid-70 LiDAR producing noisy, sparse, and heavily occluded point clouds [2602.05557]. The relevant downstream tasks include aligning the forklift to a pallet, docking to a loading platform, and interacting with a crane gripper, all of which require accurate 6-DoF pose estimation and, in the case of articulated tools, additional state variables such as the gripper opening [2602.05557].

The framework is explicitly positioned against conventional 3D object detectors that output axis-aligned or oriented 3D bounding boxes. In PIRATR, the detection target is instead a *parametric object*: each class is associated with a known CAD mesh and a parameter vector that specifies its world configuration [2602.05557]. This shifts the representation from generic geometric localization to a pose-aware, CAD-consistent object model that is closer to what a planner or controller requires, including task-relevant semantics such as pallet alignment geometry or gripper articulation [2602.05557].

This suggests that PIRATR occupies an intermediate layer between low-level geometric perception and robot action generation. The paper describes this as “pose-aware, parameterized perception,” emphasizing that the output is intended to feed directly into world models and motion planners without extra geometry fitting or ICP-style alignment [2602.05557].

## 2. Parametric object model and task formulation

The input is a point cloud \(P\) from LiDAR, after subsampling, containing \(M\) object instances of interest. Each instance \(i \in \{1,\dots,M\}\) has a class label \(c_i \in C=\{1,2,3\}\), where \(c=1\) denotes a crane gripper, \(c=2\) a loading platform, and \(c=3\) a pallet [2602.05557]. The ground-truth target is a class-conditional parameter vector
\[
T_i =
\begin{cases}
[\mathbf{p}_i,\mathbf{q}_i,\alpha_i], & c_i=1,\\
[\mathbf{p}_i,\mathbf{q}_i], & c_i\in\{2,3\},
\end{cases}
\]
where \(\mathbf{p}_i \in \mathbb{R}^3\) is position, \(\mathbf{q}_i \in \mathbb{R}^4\) is a unit quaternion, and \(\alpha_i \in \mathbb{R}^+\) is the gripper opening angle [2602.05557].

The class-dependent semantics of \(\mathbf{p}_i\) are part of the formulation rather than a post hoc convention. For grippers, \(\mathbf{p}_i\) is the center of mass; for loading platforms, it is the front-right corner facing the driver cabin; for pallets, it is the bottom center point reflecting ground support [2602.05557]. The model therefore predicts not just a rigid transform but a manipulation-relevant reference frame.

Given a class mesh \(\mathcal{M}^{c_i}\) and parameters \(T_i\), PIRATR instantiates the object through a mapping
\[
\Phi : (\mathcal{M}^{c_i}, T_i) \mapsto \mathbb{R}^{64\times 3},
\]
which applies the pose and state configuration to the mesh and produces a compact 64-point surface representation used for geometric losses [2602.05557]. For grippers, the opening angle is realized through a simple mechanical rule that rotates the gripper “fingers” relative to the base according to \(\alpha\) [2602.05557].

The treatment of symmetry is integral to both supervision and evaluation. Grippers and pallets are treated as invariant under a 180° rotation about the vertical \(z\)-axis, in addition to the usual quaternion sign symmetry, whereas loading platforms retain only quaternion sign symmetry because front/back orientation is semantically important [2602.05557]. All positions and meshes are normalized to \([-1,1]\), and the gripper opening angle is scaled to \([-1,1]\) for training [2602.05557].

| Class | Reference point \(\mathbf{p}\) | Extra parameter / symmetry |
|---|---|---|
| Crane gripper | Center of mass | Opening angle \(\alpha\); 180° yaw symmetry |
| Loading platform | Front-right corner | No extra continuous parameter; no 180° yaw symmetry |
| Pallet | Bottom center point | No extra continuous parameter; 180° yaw symmetry |

## 3. Architecture and extensibility

PIRATR follows the 3DETR/PI3DETR pattern: it downsamples and encodes the point cloud via a PointNet++-style set abstraction, applies a transformer encoder-decoder with query embeddings anchored at sampled 3D locations, and uses a shared classifier together with class-specific heads to produce object hypotheses [2602.05557]. This places it in the family of end-to-end transformer-based 3D detection architectures derived from 3DETR [2109.08159] and built on PointNet++-style feature extraction [1706.02413].

The input point cloud is preprocessed to at most 32k points. A SAModule performs farthest point sampling to pick a subset of points, exemplified by 2048 points, computes local permutation-invariant features, and outputs per-point features of dimension \(d=768\) [2602.05557]. Query positions \(\{q_j\}_{j=1}^K\) are sampled via farthest point sampling with \(K=128\), and each query position is encoded using a sinusoidal positional embedding [2602.05557].

The transformer backbone is copied from PI3DETR and consists of 3 encoder layers and 9 decoder layers with token dimension 768 [2602.05557]. The decoder consumes positional and learned query features, attends to the encoded point features, and produces output embeddings \(\{o_j\}_{j=1}^K\), each of which is interpreted by the output heads [2602.05557].

For each output embedding \(o_j\), PIRATR applies a shared classification head \(\theta^{\text{cls}}\) that predicts probabilities over four classes,
\[
\hat{\pi}_j \in [0,1]^4,
\]
corresponding to no-object, gripper, loading platform, and pallet [2602.05557]. In parallel, three class-specific regression heads map the same token to parameter vectors. For grippers,
\[
\theta^{\text{G}}(o_j)=\hat{T}_j=[\hat{\mathbf{p}}_j,\hat{\mathbf{q}}_j,\hat{\alpha}_j],
\]
where the position is predicted as an offset from the query point,
\[
\hat{\mathbf{p}}_j=\hat{\Delta}_j+\mathbf{q}_j,
\]
the quaternion is normalized, and \(\hat{\alpha}_j\) is the normalized opening angle [2602.05557]. Loading-platform and pallet heads predict \([\hat{\mathbf{p}}_j,\hat{\mathbf{q}}_j]\) with class-dependent semantics [2602.05557].

A notable architectural feature is modular extensibility. To add a new object class, the procedure is to provide a CAD mesh \(\mathcal{M}^{c_{\text{new}}}\) and a parametrization \(T\), add a new FFN head \(\theta^{c_{\text{new}}}\), extend the classifier, and introduce symmetry handling and a corresponding \(\Phi\) mapping for the Chamfer loss, without changing the transformer core or point processing pipeline [2602.05557]. This suggests that PIRATR is designed as a general parametric-object detector rather than a fixed three-class model.

## 4. Training pipeline, synthetic data, and supervision

All training is performed purely on synthetic data generated in Blender [2602.05557]. The synthetic environment contains geometrically accurate CAD models of the forklift, truck with loading crane and gripper, and pallets, while omitting small details such as cables [2602.05557]. The crane is fixed in world coordinates; the supervision targets are the gripper, loading platform, and pallets [2602.05557].

Scene generation uses extensive domain randomization. The forklift pose is sampled on a circle of radius 5–16 m around the crane, with orientation toward one of 8 predefined points of interest on the truck or away from it [2602.05557]. The LiDAR sensor is mounted on the forklift mast, so mast motion naturally varies sensor height and tilt [2602.05557]. The gripper pose is sampled on a circle of radius 3.5–8 m around the back of the truck with random height from 0.5–4.5 m, random orientation, and random opening angle \(\alpha\) [2602.05557]. Pallets are placed using Poisson-disk sampling modulated by fractal Perlin noise, may contain boxes or other pallets, and can be stacked up to two high, while vegetation and random wall meshes are added to simulate outdoor clutter [2602.05557].

The LiDAR simulation reproduces the Livox Mid-70 non-repetitive scanning pattern by using official Livox-SDK sample data consisting of timestamp, azimuth, and elevation sequences for more than 400k rays, converting spherical angles to direction vectors, and raycasting in Blender Geometry Nodes up to a maximum distance of 25 m [2602.05557]. The resulting raw simulated point cloud is subsampled to 32k points via farthest point sampling, and objects with fewer than a class-specific count of hits are dropped from the ground truth because they are too occluded [2602.05557].

The supervision combines classification, parameter regression, and mesh-level geometric consistency. Quaternion ambiguity is handled through a symmetry-aware loss
\[
\mathcal{L}_\text{quat}^{\mathcal{S}}(\hat{\mathbf{q}},\mathbf{q})
= \min_{\mathbf{q}' \in \mathcal{E}_\mathcal{S}(\hat{\mathbf{q}})} \ell_1(\mathbf{q}',\mathbf{q}),
\]
where the symmetry set \(\mathcal{S}\) depends on the class [2602.05557]. The geometry-aware parameter loss adds position error, symmetry-aware quaternion error, and opening-angle error for grippers [2602.05557]. In addition, a Chamfer distance is computed between the 64-point surface samples of predicted and ground-truth posed meshes,
\[
\mathcal{L}_\text{CD}\!\left(\Phi(\mathcal{M}^{c_i},\hat{T}_j), \Phi(\mathcal{M}^{c_i},T_i)\right),
\]
so that training explicitly penalizes whole-object misalignment rather than only reference-point discrepancies [2602.05557].

Prediction-to-target assignment is performed via a geometry-aware Hungarian matching with cost
\[
\mathcal{C}_\text{match}(j,i) = -\hat{p}_j(c_i) + \mathcal{L}_\text{param}(j,i),
\]
yielding a bipartite matching between \(M\) ground-truth targets and \(K\) predictions; unmatched predictions are treated as no-object [2602.05557]. The total training loss is
\[
\mathcal{L}_\text{total}(j,i)
= -w_{c_i}\log \hat{\pi}_j
+ \mathds{1}_{\{c_i\neq 0\}}\,\mathcal{L}_\text{param}(j,i)
+ \mathds{1}_{\{c_i\neq 0\}}\,
\mathcal{L}_\text{CD}\!\left(\Phi(\mathcal{M}^{c_i},\hat{T}_j), \Phi(\mathcal{M}^{c_i},T_i)\right),
\]
with weighted cross-entropy for class imbalance and geometric terms for matched objects [2602.05557].

Training is implemented in PyTorch and initialized from a PI3DETR checkpoint [2602.05557]. The schedule consists of 330 epochs with AdamW at learning rate \(10^{-4}\), followed by 100 additional epochs at \(10^{-5}\), with effective batch size 132 via gradient accumulation and gradient clipping with L2-norm 0.2 [2602.05557]. Data augmentation includes random rotations up to \(5^\circ\) around \(x\) and \(y\), and with probability \(1/3\), Gaussian point noise with standard deviation in \((0.0,0.04]\) in normalized coordinates [2602.05557].

## 5. Inference, deployment, and empirical results

At inference time on the real robotic platform, the Livox Mid-70 provides 10 “packages” of 10k points per second through a ROS2 interface [2602.05557]. The preprocessing pipeline removes noisy points based on intensity and spatial stability, discards points beyond 25 m, subsamples point clouds in the 50k–400k range down to 32k via farthest point sampling, rotates points by 180° around the \(z\)-axis to match Blender’s training convention, runs the network, and then rotates predicted poses back by \(-180^\circ\) around \(z\) to the ROS2/world convention [2602.05557]. The output for each detected object is a class label, position, orientation, and, for grippers, opening angle [2602.05557].

Because these outputs correspond to parametric CAD instantiations, they can be consumed directly by robot-control logic. For pallets, the bottom-center pose and orientation can be used to align forks and plan entry into pallet slots; for loading platforms, the front-right corner and pose indicate where the forklift must park or where the loading surface lies; for grippers, the framework yields both global pose and opening state, supporting reasoning about grasp configuration and collision prediction with the semi-articulated mesh [2602.05557].

Evaluation uses a synthetic dataset of 5000 scenes and a real dataset of 73 scenes collected with the real Livox Mid-70 on the forklift, manually annotated for 34 grippers, 32 loading platforms, and 70 pallets [2602.05557]. Geometric quality is measured by position \(\ell_2\) error, geodesic orientation error, yaw error, and gripper opening-angle error, while detection accuracy is measured by per-class AP and mAP, with a prediction counted as correct when the class matches and the Chamfer distance between posed meshes is below \(0.00125\) [2602.05557].

On the synthetic test set, PIRATR reports mean AP \(=0.982\), with AP values of 0.997 for grippers, 0.990 for loading platforms, and 0.958 for pallets [2602.05557]. Mean position errors are 0.10 m for grippers, 0.12 m for loading platforms, and 0.12 m for pallets; mean geodesic orientation errors are 4.90°, 0.92°, and 12.29°, respectively; and mean gripper opening error is 6.59° [2602.05557]. The paper notes that pallets are the hardest category, plausibly because of greater variation, clutter, and stacking [2602.05557].

On real outdoor LiDAR scans, without additional fine-tuning, the reported detection performance is mAP \(=0.919\), with AP values of 0.962 for grippers, 0.988 for loading platforms, and 0.805 for pallets [2602.05557]. Mean position errors are 0.13 m for grippers, 0.24 m for loading platforms, and 0.14 m for pallets; mean geodesic orientation errors are 6.37°, 1.57°, and 15.18°; mean yaw errors are 2.46°, 1.19°, and 12.32°; and mean gripper opening error is 6.69° [2602.05557]. Boxplots reported in the paper indicate that more than 50% of angle errors are below 10° and more than 50% of position errors are below 20 cm for all classes [2602.05557].

Qualitative results show predicted gripper, platform, and pallet meshes aligned with real scenes even under occlusions, clutter, and vegetation [2602.05557]. Robustness tests further indicate accurate predictions across a range of distances, heights, rotations, and gripper openings, with a documented failure mode for an extreme close-up loading platform at 80° rotation [2602.05557]. Additional experiments show imperfect stack detection for pallets carrying unseen cylindrical objects and retained detection capability under human occlusion despite the absence of such cases in training [2602.05557].

Runtime, including the full pipeline from raw input to final predictions on an RTX 4090, is approximately 218 ms at 50k raw points, 371 ms at 200k raw points, and 598 ms at 400k raw points [2602.05557]. The paper characterizes this as operation in the hundreds-of-milliseconds range, suitable for near-real-time deployment in slow-moving heavy machinery [2602.05557].

## 6. Relation to prior work, limitations, and prospective extensions

PIRATR inherits its architectural template from transformer-based 3D point-cloud detection, especially 3DETR [2109.08159], and more directly from PI3DETR, whose core architecture and geometry-aware matching ideas are repurposed from parametric edge detection to object-level pose and state estimation [2602.05557]. Relative to mainstream LiDAR detectors that rely on voxel-based, pillar-based, BEV, or point-based pipelines and typically output 3D boxes, PIRATR operates directly on raw point clouds, avoids intermediate voxel/BEV representations, and predicts parametric CAD-consistent object shapes rather than only boxes [2602.05557].

The framework also differs from primitive-fitting and CAD-reconstruction systems based on RANSAC or learned primitive detection. Those methods produce compact geometric primitives such as planes, cylinders, or edges, but are described in the paper as usually not being multi-class object detectors operating on semantic objects in cluttered outdoor scenes [2602.05557]. PIRATR instead binds geometry to semantics, pose, and task-relevant state variables in a single detection architecture [2602.05557].

The paper highlights five main contributions: a generic, fully differentiable, end-to-end network for 3D multi-class parametric multi-object detection and 6-DoF pose estimation; a synthetic LiDAR data-generation pipeline with realistic Livox scanning and domain randomization; explicit integration of object state estimation through class-specific FFNs; quantitative synthetic and real evaluation with robustness tests; and deployment on an outdoor autonomous forklift performing loading tasks [2602.05557]. The associated code is available at the project repository listed in the paper [2602.05557].

Several limitations are stated or implied. PIRATR depends on accurate CAD meshes and clear parametrizations for each class, which constrains it to object categories with known geometry and low-dimensional configuration spaces [2602.05557]. The experimental scope covers only three classes, all of relatively fixed geometry apart from pose and gripper opening [2602.05557]. Although robustness to occlusion is strong, failures remain under extreme clutter or unseen load configurations, particularly for stacked pallets with novel objects on top [2602.05557]. The training setup is sensor-specific to the Livox Mid-70 and would require re-simulating scan pattern and noise for transfer to different LiDARs [2602.05557]. Runtime is acceptable for slow heavy machinery but would likely require optimization for faster platforms or lower-end hardware [2602.05557].

Future work proposed in the paper includes extending the supported object classes, improving synthetic realism to further strengthen sim-to-real generalization, reducing training time through more efficient architectures or strategies, and incorporating temporal information from multiple frames for higher robustness and accuracy [2602.05557]. A plausible implication is that PIRATR is best understood not as a finished perception stack, but as a concrete blueprint for simulation-trained, transformer-based parametric object perception in robotics, with CAD-grounded outputs tailored to manipulation and docking tasks rather than generic scene understanding [2602.05557].

Source: https://www.emergentmind.com/topics/piratr