PTv3-Extreme (PTv3-EX) Segmentation

• The paper introduces PTv3-EX, a semantic segmentation framework that enhances PTv3 with multi-frame training to improve spatial context handling.
• It preserves the original PTv3 architecture while integrating plug-and-play multi-frame fusion and a no-clipping-point policy to incorporate distant returns.
• By applying a simple model ensemble, PTv3-EX significantly boosts test mIoU on the Waymo Open Dataset, demonstrating minimal architectural changes with major performance gains.

Point Transformer V3 Extreme (PTv3-EX) is a semantic segmentation framework for large-scale LiDAR point clouds, developed to compete in the 2024 Waymo Open Dataset Challenge. PTv3-EX enhances the core Point Transformer V3 (PTv3) architecture by introducing multi-frame training, a no-clipping-point policy, and simple model ensembling, achieving state-of-the-art performance without any modification to the underlying network structure (Wu et al., 2024).

1. Core Architecture and Preservation in PTv3-EX

PTv3-EX inherits the entire PTv3 backbone unchanged. The PTv3 architecture is composed of:

• Patch grouping with space-filling curves: Given a point cloud $P = \{p_i\in\mathbb{R}^3\}_{i=1}^N$, points are serialized via Morton or Hilbert ordering into a 1D sequence. Neighboring points are borrowed across patch boundaries to ensure divisibility by the patch size $S$.
• Stacked Transformer blocks: Each of the $L$ blocks consists of:
  • Local self-attention (FlashAttention) on contiguous sub-sequences of length $S$, with shifts or shuffles in ordering every other layer.
  • Depthwise sparse convolution ("xCPE") for conditional positional encoding, implemented as a 1×1 sparse convolution and residual connection.
• Per-point prediction head: A point-wise MLP produces logits for 23 semantic classes.

Formally, in block $l$ with tokens $X^{(l)} \in \mathbb{R}^{M \times d}$, attention computation is: $Q, K, V = X^{(l)}W_Q, X^{(l)}W_K, X^{(l)}W_V$ with head-wise: $$\mathrm{head}_h = \mathrm{softmax}\left(\frac{Q_h K_h^\top}{\sqrt{d_h}} + B_h\right) V_h$$ and the depthwise sparse convolution step: $$Y^{(l)} = X^{(l)} + \mathrm{SparseConv1\times1}(X^{(l)})$$ These outputs are fused via PreNorm residual connections. No changes are introduced to attention mechanisms, random orderings, or bottleneck configurations: all advances are exclusively "plug-and-play" at the system level (Wu et al., 2024).

2. Multi-Frame Input Fusion and Training Protocol

To address the sparse long-range context in single-frame LiDAR data, PTv3-EX concatenates three temporally adjacent frames—current ($t$), previous ($t-1$), and second previous ($t-2$)—into a unified point set. Rigid alignment using known ego-motion transformations aligns past frames into the current frame's coordinate system: $$\widehat P^{(t-k)} = T_{t\leftarrow t-k}(P^{(t-k)}), \quad k=1,2$$ The model input is: $$P^{\mathrm{fuse}} = P^{(t)} \cup \widehat P^{(t-1)} \cup \widehat P^{(t-2)}$$ A timestamp channel or learnable frame embedding distinguishes points by origin frame.

Supervision is limited to the current frame’s points $P^{(t)}$, while the network processes features from all concatenated frames. The composite segmentation loss is: $$\mathcal{L} = \frac{1}{|P^{(t)}|} \sum_{p_i \in P^{(t)}} \left[\mathrm{CE}(y_i, \hat y_i) + \lambda\,\mathrm{Lovasz}(y_i, \hat y_i)\right], \quad \lambda=1$$ This strategy maximizes spatial coverage during both training and inference while constraining the label targets to unambiguous current-frame locations.

3. No-Clipping-Point Policy

Deviation from standard practice is realized by removing axis-aligned spatial clipping on input points. While previous methods—including the original PTv3—clipped to $x, y \in [-75.2, +75.2]$ and $z \in [-4.0, +2.0]$, PTv3-EX processes all LiDAR returns:

$$\text{Clipping rule dropped:} \quad P \gets \{p=(x,y,z)\mid |x|\le75.2,\,|y|\le75.2,\, -4\le z\le2\}$$

This enables training on isolated, distant points ("distant returns") not typically present in the input. Empirical measurements confirm substantial impact:

• Multi-frame input with clipping: mIoU ≈ 72.3% (Δ +0.2% over baseline)
• Multi-frame input without clipping: mIoU = 74.8% (Δ +2.7%)

This policy is identified as the dominant factor in enabling multi-frame fusion to benefit semantic segmentation at long range.

4. Model Ensembling Protocol

For leaderboard submission, PTv3-EX combines the predictions of three independently trained models ($\mathcal{M}_1, \mathcal{M}_2, \mathcal{M}_3$), differing only by random seed. Each model produces a per-point logit vector $\ell_i^{(m)} \in \mathbb{R}^{23}$, and fusion is performed by unweighted mean:

$$\bar\ell_i = \frac13 \sum_{m=1}^3 \ell_i^{(m)}, \qquad \hat y_i = \arg\max_{c} \bar\ell_{i,c}$$

This simple ensemble elevates test mIoU from 0.7276 (single model) to 0.7397 (ensemble of three). No other voting or weighting strategies are applied.

5. Experimental Performance and Ablation Results

Table 1: Representative results for PTv3 and PTv3-EX on the Waymo Open Dataset (WOD) semantic segmentation leaderboard:

Class	PTv3-val	PTv3-test	PTv3-EX-val	PTv3-EX-test (single)	PTv3-EX-test (×3)
Car	0.9447	0.9571	0.9463	0.9662	0.9662
Truck	0.6207	0.6793	0.6283	0.7397	0.7397
Bus	0.8665	0.7482	0.8920	0.7792	0.7792
...	...	...	...	...	...
mIoU	0.7213	0.7068	0.7480	0.7276	0.7397

Further system statistics:

• Parameters: 46.2 million (single model)
• Training latency: 482 ms/iteration (4 × A100 GPUs)
• Inference time: 253 ms/frame (RTX 4090, batch size 1)

Ablation summary:

• Baseline PTv3, single-frame, with clipping: 72.13% val mIoU
• • Multi-frame (with clipping): ~72.3%
• • Multi-frame no-clip: 74.8%

6. Analysis and Lessons for Future Work

• The combination of no-clipping and multi-frame input drives the major performance leap (Δ ≈ +2.7% mIoU) by allowing the model to leverage distant, semantically informative points.
• A simple averaging ensemble grants a significant but distinctly secondary improvement (+1.2% mIoU), appropriate primarily for test submission and not routine validation.
• The original PTv3 design, involving 1D serialization, structured grouping, and FlashAttention, demonstrates natural scalability to multi-frame input scenarios; no further architectural innovation is mandated.
• Recommended extensions include learned temporal weights/attention for frames, dynamic (scene-aware) clipping, and single-model knowledge distillation from ensembles.

PTv3-Extreme empirically validates that leading performance in large-scale point cloud segmentation can be achieved through system-level enhancements—multi-frame fusion, relaxed data curation, and ensembling—applied on top of robust yet concise architectures without introducing additional network complexity (Wu et al., 2024).