Superpoint Transformer (SPT) in 3D Segmentation

• SPT is a family of transformer architectures that tokenizes point clouds into superpoints, reducing computational complexity while preserving structural details.
• It employs a hierarchical partitioning strategy to aggregate locally homogeneous regions, enabling efficient sparse attention over superpoint graphs.
• The framework underpins diverse applications—including semantic, instance, and panoptic segmentation—offering significant speed-ups and parameter efficiency.

Superpoint Transformer (SPT) denotes a family of superpoint-centric transformer architectures for 3D point-cloud understanding in which a point cloud is first partitioned into connected, homogeneous regions and subsequent attention and prediction operate primarily on those regions rather than on raw points or dense voxels. In the literature, the term refers most directly to a hierarchical transformer backbone for large-scale 3D semantic segmentation, but it also underlies the query-based instance-segmentation formulation instantiated as SPFormer and later serves as a backbone or design template for scalable panoptic segmentation, GPU-only superpoint pipelines, and multimodal 3D reasoning systems (Robert et al., 2023, Sun et al., 2022, Robert et al., 2024, Geist et al., 29 Nov 2025, Deng et al., 2 Jan 2025).

1. Conceptual basis and scope

SPT is motivated by the observation that raw point clouds are dense, spatially redundant, and poorly matched to uniform compute allocation. Superpoints group locally homogeneous points according to geometry and radiometry, yielding adaptive regions that tend to respect object boundaries while reducing the number of processing units by one to two orders of magnitude in the semantic-segmentation formulation, and by typical superpoint-to-point ratios of about $20\text{–}30\times$ in later panoptic work (Robert et al., 2023, Robert et al., 2024).

Within this paradigm, each superpoint becomes a token. The transformer then models relations among tokens that correspond to coherent scene parts rather than individual points. This differs from point-wise and voxel-wise transformers, whose memory and compute scale with raw sampling density, and from earlier superpoint systems such as SuperPoint Graph, which rely on local graph convolutions and more expensive preprocessing (Robert et al., 2023).

Two design lineages are especially prominent. The first is the hierarchical SPT backbone for semantic segmentation, which constructs a multilevel partition and applies sparse self-attention on superpoint graphs. The second is the superpoint-based set-prediction formulation of SPFormer, which uses a sparse 3D U-Net to extract point features, pools them to superpoints, and predicts instances directly as masks over superpoints using learnable queries rather than 3D boxes, semantic branches, or post-hoc aggregation (Robert et al., 2023, Sun et al., 2022).

A recurrent misconception is that SPT is merely a transformer applied after arbitrary clustering. The cited works instead treat the partition as structurally central: superpoint purity, graph connectivity, hierarchical aggregation, and mask or clustering objectives are all defined with respect to the superpoint representation itself (Robert et al., 2023, Sun et al., 2022, Robert et al., 2024).

2. Superpoint construction and hierarchical representation

In the original semantic-segmentation formulation, SPT builds a hierarchical partition $P=[P_0,P_1,\dots,P_I]$, where $P_0$ is the set of points and each $P_{i+1}$ partitions the elements of $P_i$. A point-level graph $G=(C,E,w)$ is constructed with $k$-NN connectivity, using $k=10$ and $w=1$, and superpoints arise from a piecewise-constant approximation problem:

$$\mathcal{J}(e; f, \mathcal{G}, \lambda) = \|e - f\|^2 + \lambda \sum_{(u,v)\in \mathcal{E}} w_{u,v}\,[e_u \neq e_v].$$

The features used for partitioning include radiometry—RGB/HSV for S3DIS and KITTI-360 or intensity for DALES—together with PCA-based geometric descriptors such as linearity, planarity, scattering, verticality, and elevation (Robert et al., 2023).

This hierarchy is not only a preprocessing device. Fine levels are intended to be near-semantically pure, while coarser levels capture larger structures such as walls or facades. For each level, SPT constructs a superpoint adjacency graph by connecting superpoints whose closest points are within a gap $P=[P_0,P_1,\dots,P_I]$0, using an approximate superpoint-gap heuristic rather than global triangulation. That design is part of why the preprocessing is reported as $P=[P_0,P_1,\dots,P_I]$1 faster than earlier superpoint-based approaches (Robert et al., 2023).

SPFormer uses a different but related superpoint interface. The input cloud is $P=[P_0,P_1,\dots,P_I]$2, with each point storing $P=[P_0,P_1,\dots,P_I]$3. A sparse 3D U-Net produces point-wise features $P=[P_0,P_1,\dots,P_I]$4, and precomputed superpoints $P=[P_0,P_1,\dots,P_I]$5 are obtained offline using the superpoint graph approach of Landrieu and Simonovsky 2018. Superpoint features are then formed by average pooling,

$P=[P_0,P_1,\dots,P_I]$6

producing a compact set of superpoint tokens for the decoder (Sun et al., 2022).

Later work identifies the partition stage as a systems bottleneck and modifies it directly. EZ-SP keeps SPT’s superpoint-level classifier but replaces the CPU-bound partition step with a fully GPU combinatorial partitioner that optimizes the same contour-regularized objective class:

$P=[P_0,P_1,\dots,P_I]$7

Its learnable point encoder shapes embeddings so that the greedy solver yields semantically coherent superpoints, while the full pipeline remains GPU-only (Geist et al., 29 Nov 2025).

3. Backbone architecture and attention mechanisms

The canonical SPT architecture is a U-Net-like encoder–decoder defined over the hierarchical partition rather than over a voxel pyramid or raw-point sequence. Point features are embedded only at $P=[P_0,P_1,\dots,P_I]$8; there is no attention between raw points. Features are pooled from children to parents, fused with relative position, and refined by sparse attention on each superpoint graph. The encoder update is written as

$P=[P_0,P_1,\dots,P_I]$9

and the decoder propagates information from coarse to fine levels through parent–child links (Robert et al., 2023).

Attention in SPT is graph-sparse rather than dense. Standard scaled dot-product attention remains the reference mechanism, but the implementation augments queries, keys, and values with learned per-edge adjacency terms derived from handcrafted relational features. Message passing therefore follows the superpoint graph rather than all-pairs interactions, and the typical layer complexity scales with the number of superpoint edges rather than with the square of the number of raw points (Robert et al., 2023, Robert et al., 2024).

Typical SPT configurations are deliberately compact. Reported defaults use $P_0$0, $P_0$1 for S3DIS and DALES, $P_0$2 for KITTI-360, 16 attention heads, three encoder blocks per level, one decoder block per level, GraphNorm, and LeakyReLU MLPs. The small key space and sparse graph attention are explicit efficiency choices rather than incidental implementation details (Robert et al., 2023).

SPFormer replaces the hierarchical self-attention backbone with a query decoder operating over superpoint features. After superpoint pooling and linear projection to $P_0$3, $P_0$4 learnable queries are refined through $P_0$5 decoder layers. Positional encoding is removed, and the layer order is modified so that cross-attention precedes self-attention. The core masked cross-attention update is

$P_0$6

where $P_0$7 is derived from the previous layer’s predicted masks by thresholding at $P_0$8, thereby restricting the next layer’s cross-attention to foreground superpoints. Masks are predicted by

$P_0$9

with $P_{i+1}$0 produced by a separate mask branch (Sun et al., 2022).

A distinct architectural extension appears in 3D-LLaVA’s Omni Superpoint Transformer. There, superpoints are obtained by bottom-up clustering and pooled from a Sparse 3D U-Net, but OST is “primarily composed without cross-attention layers” and instead uses distance-adaptive self-attention,

$P_{i+1}$1

where $P_{i+1}$2 is the Euclidean distance between superpoint centroids. The same module serves as visual feature selector, prompt encoder, and referring mask decoder for a 3D large multimodal model (Deng et al., 2 Jan 2025).

4. Prediction paradigms built on SPT

For semantic segmentation, SPT predicts labels at the superpoint level and broadcasts them back to points through the partition. Training uses hierarchical supervision: a fine-scale cross-entropy term at $P_{i+1}$3 and additional coarse-scale losses on higher levels. This arrangement is intended to exploit the hierarchy both as an architectural scaffold and as a regularizer (Robert et al., 2023).

For instance segmentation, SPFormer recasts prediction as direct mask generation over superpoints. Each query outputs category logits $P_{i+1}$4, an IoU-aware score $P_{i+1}$5, and a superpoint mask $P_{i+1}$6. Ground-truth masks are defined by a majority-vote rule at the superpoint level: a superpoint is assigned to an instance if at least 50% of its points belong to that instance. Training uses Hungarian bipartite matching with a cost that combines classification and mask terms, and mask supervision uses binary cross-entropy plus Dice loss, with the paper reporting that Dice is indispensable and that Dice plus BCE gives the best performance relative to BCE or focal alone. Inference uses only the last decoder head, requires no NMS, and ranks predictions by $P_{i+1}$7 (Sun et al., 2022).

For panoptic segmentation, SuperCluster uses SPT in a different role. The SPT backbone generates superpoint embeddings $P_{i+1}$8, semantic distributions $P_{i+1}$9, and object-agreement scores $P_i$0 on adjacent superpoint pairs. Panoptic segmentation is then formulated as graph clustering with a Potts regularizer rather than as mask-set prediction. The clustering objective combines semantic agreement and positional consistency, while edge penalties are obtained from

$P_i$1

A notable property of this formulation is that all supervision is local to nodes and edges; there is no Hungarian matching or DETR-style instance assignment during training (Robert et al., 2024).

The same superpoint-centric logic also extends beyond conventional segmentation. In 3D-LLaVA, OST produces language-aligned visual tokens $P_i$2 from superpoints via an alignment head and projection $P_i$3, encodes clicks, boxes, or masks as prompt tokens $P_i$4, and decodes referring masks when the LLM emits a special $P_i$5 token. Pretraining combines classification, mask, and 2D-to-3D distillation losses, and instruction tuning updates only the visual projector, the SEG projector, and LoRA parameters on the LLM while freezing the Sparse 3D U-Net, OST, and main LLM body (Deng et al., 2 Jan 2025).

5. Efficiency, scaling behavior, and reported results

A central claim across the SPT literature is that computation is better aligned with scene complexity when the tokenization unit is a superpoint rather than a point or voxel. Because attention and message passing operate on a graph whose size is tied to superpoint count, SPT-based systems can process full floors, building-scale scans, or multi-million-point scenes in a single pass where point-level systems often require sliding windows, stitching, or large mask sets (Robert et al., 2023, Robert et al., 2024).

System	Task	Representative reported results
SPT (Robert et al., 2023)	Semantic segmentation	76.0% mIoU on S3DIS 6-fold, 63.5% on KITTI-360 Val, 79.6% on DALES; ~212k parameters; ~2 s inference on S3DIS Area 5
SPFormer (Sun et al., 2022)	3D instance segmentation	54.9% mAP on the ScanNetv2 hidden test set; 56.3 mAP, 73.9 AP50, 82.9 AP25 on ScanNetv2 validation; 247 ms per ScanNetv2 scene on RTX 3090
SuperCluster with SPT backbone (Robert et al., 2024)	3D panoptic segmentation	50.1 PQ on S3DIS Area 5 and 58.7 PQ on ScanNetV2; ~209k parameters for S3DIS/DALES; 3.3 s on a 9.2M-point S3DIS Area 5 scene
EZ-SP + SPT classifier (Geist et al., 29 Nov 2025)	GPU-only superpoint semantic segmentation	76.1 mIoU on S3DIS 6-fold versus 76.0 for SPT; partition $P_i$6 faster than CPU methods; $P_i$7 faster end-to-end inference than PointTransformer-v3

The semantic-segmentation SPT emphasizes parameter efficiency as much as absolute accuracy. It reports up to $P_i$8 fewer parameters than other state-of-the-art models while remaining competitive on S3DIS, KITTI-360, and DALES, and training in about 3 hours per S3DIS fold on a single GPU—$P_i$9 to $G=(C,E,w)$0 fewer GPU-hours than several popular point and voxel transformers or CNNs (Robert et al., 2023).

SPFormer emphasizes end-to-end instance prediction without boxes, semantic branches, aggregation, NMS, or refinement modules. On ScanNetv2, its inference-time breakdown is reported as 179 ms for superpoint extraction on CPU, 29 ms for the backbone, 18 ms for superpoint pooling, and 21 ms for the query decoder on GPU, totaling 247 ms per scene. Its ablations further report that superpoints are essential, that mask-based matching improves mAP by $G=(C,E,w)$1 over box-based matching, that the IoU-aware score head adds $G=(C,E,w)$2 on mAP/AP50/AP25, and that the preferred decoder setting is $G=(C,E,w)$3 layers with $G=(C,E,w)$4 queries (Sun et al., 2022).

SuperCluster demonstrates a different scalability regime. It processes S3DIS Area 5 with 9.2M points and 1,863 “thing” objects in 3.3 s on a V100-32GB GPU and reports one-pass inference examples such as 21.3M S3DIS points in 7.4 s, 10.9M ScanNet points in 6.8 s, 18.0M DALES points in 10.1 s, and 11.0M KITTI-360 points in 6.6 s on an A40. This suggests that, once prediction is reformulated as local node/edge estimation plus graph clustering, the superpoint backbone scales to scenes and object counts that are awkward for mask-per-instance pipelines (Robert et al., 2024).

EZ-SP sharpens the efficiency argument by isolating the partition stage. On S3DIS, the original SPT pipeline is reported as 808 GPU-s total, with 418 s spent in partitioning, whereas EZ-SP reduces partition time to 3.4 s and total time to 153 GPU-s while keeping inference time for the classifier itself at 14 s. Accuracy on S3DIS slightly improves, from 76.0 to 76.1 mIoU in 6-fold evaluation and from 68.9 to 69.6 mIoU in Area 5, indicating that the original SPT bottleneck was primarily systems-level rather than representational (Geist et al., 29 Nov 2025).

6. Limitations, nomenclature, and current directions

The most persistent limitation is dependence on superpoint quality. In SPFormer, oversegmentation errors can fragment objects or merge distinct instances, and very fine-grained or thin structures may not be captured well by superpoints. In the original SPT, partitions may cross semantic boundaries when geometry or radiometry contrast is weak, with examples such as curb versus road or whiteboard versus wall. SuperCluster likewise notes sensitivity to very sparse sampling, especially for thin structures near the sampling limit, while EZ-SP highlights failure cases under low geometric or radiometric contrast and domain shifts in intensity-only or textureless settings (Sun et al., 2022, Robert et al., 2023, Robert et al., 2024, Geist et al., 29 Nov 2025).

A second limitation is that partitioning is generally not learned end-to-end in the original formulations. SPT relies on hard hierarchical partitioning, SPFormer uses offline superpoints, SuperCluster solves a noncontinuous, nondifferentiable clustering objective, and EZ-SP—although learnable in its embedding generator—still relies on a greedy combinatorial solver for the final partition. This suggests a stable research frontier around differentiable or relaxed partitioning, learned edge weights, adaptive neighborhood construction, and end-to-end superpoint generation (Robert et al., 2023, Robert et al., 2024, Geist et al., 29 Nov 2025).

There are also task-specific constraints. SPFormer’s 50% rule for assigning superpoints to ground-truth masks can introduce label noise around instance boundaries; scenes with many small objects may require more queries and create matching conflicts; and performance drops when too few or too many decoder layers are used. The semantic SPT warns that larger model scales can overfit because superpoint graphs compress the effective training set, and it does not produce point-level predictions directly. SuperCluster observes that the lightweight SPT backbone trades semantic peak accuracy for scalability, and local panoptic supervision did not improve semantic segmentation over the backbone in its experiments (Sun et al., 2022, Robert et al., 2023, Robert et al., 2024).

Current extensions follow several directions already visible in the cited works: learnable or GPU-native partitioning, adaptive query allocation, multi-scale superpoint features, explicit geometric priors in mask branches, improved positional priors tailored to 3D scenes, graph-clustering heads for panoptic segmentation, and multimodal superpoint transformers that connect 3D geometry to LLMs (Sun et al., 2022, Geist et al., 29 Nov 2025, Robert et al., 2024, Deng et al., 2 Jan 2025).

Finally, the acronym itself requires caution. In point-cloud literature, SPT conventionally denotes Superpoint Transformer, whereas the 2026 image-classification paper titled “Is an Image Also Worth 16x16=256 Superpixels? A Framework for Attentional Image Classification” explicitly states that its SPT stands for Superpixel Transformers, not Superpoint Transformer. The two lines of work are conceptually related through irregular tokenization, but their reported definitions, algorithms, and experiments are distinct (Avelar et al., 26 May 2026).