SemRaFiner: Radar Panoptic Segmentation

• SemRaFiner is a two-stage, point-based transformer framework that performs panoptic segmentation by combining a moving-instance backbone with a density-aware refinement network.
• It uses a novel radius transformer layer for adaptive neighborhood processing and targeted training augmentation to correct common radar detection errors.
• Achieving a Panoptic Quality of 81.4% and a mIoU of 70.4% on RadarScenes, it outperforms LiDAR-based approaches while operating in real time under adverse conditions.

SemRaFiner is a two-stage, point-based transformer framework for panoptic segmentation in sparse and noisy automotive radar point clouds, introduced in “SemRaFiner: Panoptic Segmentation in Sparse and Noisy Radar Point Clouds” (Zeller et al., 9 Jul 2025). It addresses a radar-specific variant of scene understanding in which every radar point must receive both a semantic class and an instance identifier, while contending with extreme sparsity, noise, strongly varying density, ego-motion, multipath, class imbalance, and real-time constraints. The framework combines a moving-instance backbone with a density-aware semantic refinement network, and uses targeted training-time augmentation to correct typical backbone failure modes such as wrongly labeled moving points and spurious small instances (Zeller et al., 9 Jul 2025).

1. Problem formulation and radar-specific challenges

SemRaFiner targets panoptic segmentation in automotive radar point clouds. In this setting, the output consists of semantic labels

$$\mathcal{S}^{\mathrm{sem}} = \{s^{\mathrm{sem}}_1,\dots,s^{\mathrm{sem}}_N\}, \quad s^{\mathrm{sem}}_i \in \{1,\dots,C\},$$

and instance IDs

$$\mathcal{I}^r = \{I^r_1,\dots,I^r_N\}, \quad I^r_i \in \mathbb{N}.$$

The semantic label specifies classes such as car, truck, pedestrian, or static, while the instance ID distinguishes individual objects of the same class (Zeller et al., 9 Jul 2025).

The difficulty of the task is tied to the properties of automotive FMCW radar. A single scan contains only a few hundred points, approximately $500\text{–}600$ on RadarScenes, and includes false returns and multipath reflections. Point density varies strongly with range: nearby objects may yield several points, whereas a car at $70\text{–}80\,\mathrm{m}$ may be represented by only $1\text{–}2$ points. Ego-motion and multipath introduce spurious detections, and small or rare classes such as pedestrians, bikes, or far vehicles are represented by very few points. The system is also required to operate faster than the radar frame rate, which is approximately $17\,\mathrm{Hz}$ in RadarScenes (Zeller et al., 9 Jul 2025).

Radar remains attractive because it is complementary to cameras and LiDAR. It works in adverse weather, directly measures Doppler velocity, and provides radar cross section (RCS) as an additional cue about material and size. SemRaFiner is designed to exploit these strengths, especially Doppler-based moving instance segmentation, and then refine the output into full panoptic predictions (Zeller et al., 9 Jul 2025).

Evaluation follows panoptic metrics from the SemanticKITTI and panoptic LiDAR literature. The principal metrics are Panoptic Quality ($\mathrm{PQ}$), mean IoU ($\mathrm{mIoU}$), and per-class PQ and IoU. This choice is important because the method is not evaluated merely as semantic segmentation; it is explicitly assessed on the joint quality of semantic labeling and instance separation (Zeller et al., 9 Jul 2025).

2. Two-stage system design

SemRaFiner is organized as a two-stage, point-based transformer framework (Zeller et al., 9 Jul 2025). The first stage is a Moving Instance Segmentation Backbone, specifically the Radar Instance Transformer (RIT). The second stage is the SemRaFiner refinement network, which operates only on points predicted as moving.

RIT is treated as a black-box backbone. It receives the current scan $\mathcal{P}^t$ together with $T$ previous scans $$\mathcal{I}^r = \{I^r_1,\dots,I^r_N\}, \quad I^r_i \in \mathbb{N}.$$0, uses attention-based temporal aggregation and graph clustering internally, and outputs binary moving/static labels $$\mathcal{I}^r = \{I^r_1,\dots,I^r_N\}, \quad I^r_i \in \mathbb{N}.$$1 and moving instance labels $$\mathcal{I}^r = \{I^r_1,\dots,I^r_N\}, \quad I^r_i \in \mathbb{N}.$$2 (Zeller et al., 9 Jul 2025). During training, SemRaFiner uses ground-truth moving labels for filtering; during inference, it uses the backbone predictions.

The refinement network receives only the moving points of the current scan, together with their coordinates and radar features. Its role is twofold: to assign semantic labels to moving points and to refine the instance assignments produced by RIT. Static points remain from the backbone output, and the merged result yields the full panoptic prediction (Zeller et al., 9 Jul 2025).

The high-level workflow is therefore asymmetric. RIT is responsible for moving-object segmentation and initial instance grouping, while SemRaFiner performs density-aware feature extraction, semantic classification, and post hoc instance correction. This design suggests a deliberate decomposition of the radar panoptic problem: motion-based grouping is handled first, and semantic disambiguation is applied afterward in a radar-specific refinement stage (Zeller et al., 9 Jul 2025).

The full system remains real time. RIT plus SemRaFiner requires approximately $$\mathcal{I}^r = \{I^r_1,\dots,I^r_N\}, \quad I^r_i \in \mathbb{N}.$$3 per scan, corresponding to about $$\mathcal{I}^r = \{I^r_1,\dots,I^r_N\}, \quad I^r_i \in \mathbb{N}.$$4, which is faster than the RadarScenes data rate (Zeller et al., 9 Jul 2025).

3. Input representation and the radius transformer

Each radar point is encoded as a $$\mathcal{I}^r = \{I^r_1,\dots,I^r_N\}, \quad I^r_i \in \mathbb{N}.$$5-dimensional feature vector

$$\mathcal{I}^r = \{I^r_1,\dots,I^r_N\}, \quad I^r_i \in \mathbb{N}.$$6

where $$\mathcal{I}^r = \{I^r_1,\dots,I^r_N\}, \quad I^r_i \in \mathbb{N}.$$7 are 3D coordinates in a vehicle frame, $$\mathcal{I}^r = \{I^r_1,\dots,I^r_N\}, \quad I^r_i \in \mathbb{N}.$$8 is the radar cross section, and $$\mathcal{I}^r = \{I^r_1,\dots,I^r_N\}, \quad I^r_i \in \mathbb{N}.$$9 is the ego-motion-compensated Doppler velocity. For the BEV representation used by the refinement stage, $500\text{–}600$0, and the raw feature dimension is $500\text{–}600$1 (Zeller et al., 9 Jul 2025).

The central architectural novelty is the radius transformer layer, which is designed for density-aware point processing. For moving points in the current scan, the network uses BEV coordinates

$500\text{–}600$2

and input features

$500\text{–}600$3

The raw radar features are first projected to a higher dimension, for example $500\text{–}600$4, and then mapped to queries, keys, and values: $500\text{–}600$5 The key design choice is neighborhood construction through a ball query rather than $500\text{–}600$6-nearest neighbors. For each point,

$500\text{–}600$7

and the neighborhood is defined as

$500\text{–}600$8

This neighborhood is then capped at a maximum of $500\text{–}600$9 neighbors, with zero-padding if fewer are available (Zeller et al., 9 Jul 2025).

This radius-based construction addresses the strongly non-uniform density of radar point clouds. In crowded regions, the radius remains fixed and the neighbor cap prevents over-connectivity; in sparse regions, the ball query still provides a physically meaningful local neighborhood. The paper explicitly contrasts this with $70\text{–}80\,\mathrm{m}$0-NN attention, which may connect unrelated distant objects in sparse regions, and with grid-based attention, whose fixed cells may not align with object extents (Zeller et al., 9 Jul 2025).

Relative positional encoding is implemented with two linear layers, BatchNorm, and ReLU, producing

$70\text{–}80\,\mathrm{m}$1

Vector attention is then formed channel-wise: $70\text{–}80\,\mathrm{m}$2 followed by an MLP and softmax over neighbors, producing normalized weights $70\text{–}80\,\mathrm{m}$3. The output feature for point $70\text{–}80\,\mathrm{m}$4 is

$70\text{–}80\,\mathrm{m}$5

with $70\text{–}80\,\mathrm{m}$6 denoting element-wise multiplication (Zeller et al., 9 Jul 2025).

The layer is embedded in a residual radius transformer block. Two such blocks are used in SemRaFiner. After them, the network applies three MLPs in sequence: $70\text{–}80\,\mathrm{m}$7, then $70\text{–}80\,\mathrm{m}$8, and finally $70\text{–}80\,\mathrm{m}$9 for class logits. The full feature pipeline is raw $1\text{–}2$0D input, linear projection to $1\text{–}2$1, block 1, block 2 with $1\text{–}2$2, then the MLP head (Zeller et al., 9 Jul 2025).

4. Refinement strategy, losses, and augmentation

SemRaFiner is trained after RIT and uses three losses: cross-entropy for per-point semantic classification, Lovász loss as a convex surrogate for IoU, and a consistency loss $1\text{–}2$3 that encourages semantic homogeneity within instances (Zeller et al., 9 Jul 2025). For each instance $1\text{–}2$4, if $1\text{–}2$5 denotes the set of distinct semantic classes predicted within that instance, the consistency loss is

$1\text{–}2$6

If an instance contains only one semantic class, the loss contribution is $1\text{–}2$7; if multiple classes are present, the loss increases. The total loss is the unweighted sum of cross-entropy, Lovász, and $1\text{–}2$8 (Zeller et al., 9 Jul 2025).

The paper emphasizes that $1\text{–}2$9 is deliberately not too strong, because SemRaFiner must still be able to assign different semantics to points that should eventually be split into new instances. Removing $17\,\mathrm{Hz}$0 yields a small but measurable drop, from $17\,\mathrm{Hz}$1 to $17\,\mathrm{Hz}$2 on validation (Zeller et al., 9 Jul 2025). This suggests that the loss acts as a mild regularizer rather than as a hard constraint.

Instance refinement occurs in two forms. First, targeted augmentation teaches the network to correct static-versus-moving errors. Second, a post-processing step uses semantic predictions to split inconsistent RIT instances: if a backbone instance contains multiple semantic classes, it is separated into distinct instance IDs per class cluster (Zeller et al., 9 Jul 2025). A common misconception is that the model only predicts semantics on top of a fixed instance map. Validation results contradict that interpretation: ground-truth semantics combined with raw RIT instances without refinement yields $17\,\mathrm{Hz}$3, whereas SemRaFiner with augmentation reaches $17\,\mathrm{Hz}$4 (Zeller et al., 9 Jul 2025).

A key contribution is targeted augmentation that reproduces typical RIT errors. Two augmentation modes are used during training. With probability $17\,\mathrm{Hz}$5, static points near the boundary of ground-truth moving instances are added and marked as part of instances, so the network learns to remove them. With probability $17\,\mathrm{Hz}$6, small static-only instances containing $17\,\mathrm{Hz}$7 static points are added and treated as moving in the input, so the network learns to discard them (Zeller et al., 9 Jul 2025). Without such augmentation, the network does not encounter the failure modes of the backbone during training.

Optimization uses PyTorch, AdamW with initial learning rate $17\,\mathrm{Hz}$8, step decay by a factor of $17\,\mathrm{Hz}$9 at epoch $\mathrm{PQ}$0, $\mathrm{PQ}$1 training epochs, batch size $\mathrm{PQ}$2, and one NVIDIA A100. The default radius-transformer hyperparameters are $\mathrm{PQ}$3, $\mathrm{PQ}$4, $\mathrm{PQ}$5, $\mathrm{PQ}$6, and $\mathrm{PQ}$7 (Zeller et al., 9 Jul 2025).

5. Dataset, baselines, and empirical performance

Experiments use RadarScenes, described as the only large-scale public radar dataset with per-point labels for semantics and moving instances (Zeller et al., 9 Jul 2025). It contains $\mathrm{PQ}$8 sequences with multiple weather conditions and driving scenarios, up to four radar sensors per scene, and follows the sparse single-scan representation from prior work. The average number of points in a single test scan is approximately $\mathrm{PQ}$9. The train/validation/test split is $\mathrm{mIoU}$0 sequences (Zeller et al., 9 Jul 2025).

The comparison includes multi-scan semantic baselines such as RadarPNv1, RadarPNv2, and STA-Net, as well as single-scan baselines such as Mask3D and Gaussian Radar Transformer plus DBSCAN. SemRaFiner uses RIT for moving instances and the refinement network for semantics and instance correction (Zeller et al., 9 Jul 2025).

The principal test-set results are as follows (Zeller et al., 9 Jul 2025):

Method	Input	PQ / mIoU
Mask3D	single	56.9 / 56.1
GRT + DBSCAN	single	56.3 / 56.9
SemRaFiner	single	81.4 / 70.4
STA-Net	multi-scan	– / 70.4

Per-class panoptic quality for SemRaFiner is $\mathrm{mIoU}$1 for static, $\mathrm{mIoU}$2 for car, $\mathrm{mIoU}$3 for pedestrian, $\mathrm{mIoU}$4 for pedestrian group, $\mathrm{mIoU}$5 for bike, and $\mathrm{mIoU}$6 for truck. Its per-class semantic IoU is $\mathrm{mIoU}$7 for static, $\mathrm{mIoU}$8 for car, $\mathrm{mIoU}$9 for pedestrian, $\mathcal{P}^t$0 for pedestrian group, $\mathcal{P}^t$1 for bike, and $\mathcal{P}^t$2 for truck (Zeller et al., 9 Jul 2025).

Several comparisons are especially consequential. The overall PQ rises from approximately $\mathcal{P}^t$3 for Mask3D and GRT+DBSCAN to $\mathcal{P}^t$4 for SemRaFiner. Pedestrian PQ rises from approximately $\mathcal{P}^t$5 to $\mathcal{P}^t$6, and pedestrian-group PQ reaches $\mathcal{P}^t$7. The semantic result of $\mathcal{P}^t$8 matches the multi-scan STA-Net result, despite SemRaFiner using only single scans and about $\mathcal{P}^t$9 points rather than about $T$0 (Zeller et al., 9 Jul 2025). This suggests that a strong moving-instance backbone plus density-aware refinement is substantially more effective in this regime than generic panoptic architectures adapted from LiDAR.

Ablation studies isolate the contribution of the radius transformer. On RadarScenes validation, replacing it with a Stratified Transformer layer yields $T$1, $T$2; Point Transformer yields $T$3; KPConv yields $T$4; and the SemRaFiner radius transformer yields $T$5 (Zeller et al., 9 Jul 2025). The default setting $T$6, $T$7 is the best reported configuration. Reducing $T$8 to $T$9 lowers performance to $$\mathcal{I}^r = \{I^r_1,\dots,I^r_N\}, \quad I^r_i \in \mathbb{N}.$$00, $$\mathcal{I}^r = \{I^r_1,\dots,I^r_N\}, \quad I^r_i \in \mathbb{N}.$$01, while larger radius or neighbor counts slightly hurt performance, especially for small objects (Zeller et al., 9 Jul 2025).

The augmentation study further clarifies the refinement effect. On RadarScenes validation, ground-truth semantics combined with raw RIT instances gives $$\mathcal{I}^r = \{I^r_1,\dots,I^r_N\}, \quad I^r_i \in \mathbb{N}.$$02; SemRaFiner without augmentation gives $$\mathcal{I}^r = \{I^r_1,\dots,I^r_N\}, \quad I^r_i \in \mathbb{N}.$$03; and SemRaFiner with augmentation gives $$\mathcal{I}^r = \{I^r_1,\dots,I^r_N\}, \quad I^r_i \in \mathbb{N}.$$04 (Zeller et al., 9 Jul 2025). The paper interprets this as evidence that the network actively corrects instance assignments rather than merely attaching better semantics to fixed clusters.

Runtime and parameter counts are also reported (Zeller et al., 9 Jul 2025):

Model	Params (M)	Mean runtime (ms)
Mask3D	39.6	85.2
GRT	8.4	24.0
RIT	3.8	31.7
RIT + SemRaFiner	4.5	42.1

SemRaFiner therefore adds about $$\mathcal{I}^r = \{I^r_1,\dots,I^r_N\}, \quad I^r_i \in \mathbb{N}.$$05 million parameters and about $$\mathcal{I}^r = \{I^r_1,\dots,I^r_N\}, \quad I^r_i \in \mathbb{N}.$$06 overhead to RIT (Zeller et al., 9 Jul 2025).

6. Qualitative behavior, limitations, and research context

Qualitative examples in foggy and urban scenes show three recurring effects: better separation of close instances, cleaning of static false positives, and improved coherence for small objects such as pedestrians and bikes (Zeller et al., 9 Jul 2025). In particular, RIT may merge a car and a truck into one motion-consistent cluster, and SemRaFiner can split them using per-point semantic cues. It can also reclassify static parked cars that were erroneously marked as moving by the backbone (Zeller et al., 9 Jul 2025).

The method nonetheless depends on the quality of the moving-instance backbone. If RIT misses a moving instance entirely, SemRaFiner cannot recover it. Very sparse or extremely distant objects remain difficult to classify correctly, and some underrepresented classes, such as bike, are less accurate than cars (Zeller et al., 9 Jul 2025). The hyperparameters $$\mathcal{I}^r = \{I^r_1,\dots,I^r_N\}, \quad I^r_i \in \mathbb{N}.$$07, $$\mathcal{I}^r = \{I^r_1,\dots,I^r_N\}, \quad I^r_i \in \mathbb{N}.$$08, and augmentation settings are tuned for RadarScenes and may require adaptation for different radar configurations (Zeller et al., 9 Jul 2025).

Within radar perception, SemRaFiner sits at the intersection of radar semantic understanding, panoptic segmentation, and transformer-based point processing. Compared with prior radar work, RadarPNv1 and RadarPNv2 provide multi-scan semantic segmentation only; STA-Net also targets semantic segmentation with multi-scan aggregation; GRT performs single-scan semantic segmentation; and RIT performs moving instance segmentation exploiting Doppler (Zeller et al., 9 Jul 2025). SemRaFiner combines these lines of development into a full radar panoptic system.

The paper states that SemRaFiner demonstrates, for the first time, state-of-the-art radar panoptic segmentation on single scans, with performance comparable to multi-scan semantic-only methods and large improvements over LiDAR-panoptic-inspired baselines when applied to radar (Zeller et al., 9 Jul 2025). A plausible implication is that radar panoptic perception benefits from a decomposition that is native to the sensing modality: Doppler-powered moving instance segmentation first, then density-aware semantic refinement. The reported real-time operation and adverse-weather relevance make that decomposition particularly pertinent for autonomous driving scenarios in which cameras and LiDAR degrade (Zeller et al., 9 Jul 2025).