---
title: 'SemRaFiner: Radar Panoptic Segmentation'
url: https://www.emergentmind.com/topics/semrafiner
type: topic
---

# SemRaFiner: Radar Panoptic Segmentation

SemRaFiner is a two-stage, point-based transformer framework for panoptic segmentation in sparse and noisy automotive radar point clouds, introduced in “SemRaFiner: Panoptic Segmentation in Sparse and Noisy Radar Point Clouds” [2507.06906]. It addresses a radar-specific variant of scene understanding in which every radar point must receive both a semantic class and an instance identifier, while contending with extreme sparsity, noise, strongly varying density, ego-motion, multipath, class imbalance, and real-time constraints. The framework combines a moving-instance backbone with a density-aware semantic refinement network, and uses targeted training-time augmentation to correct typical backbone failure modes such as wrongly labeled moving points and spurious small instances [2507.06906].

## 1. Problem formulation and radar-specific challenges

SemRaFiner targets **panoptic segmentation** in **automotive radar** point clouds. In this setting, the output consists of semantic labels
\[
\mathcal{S}^{\mathrm{sem}} = \{s^{\mathrm{sem}}_1,\dots,s^{\mathrm{sem}}_N\}, \quad s^{\mathrm{sem}}_i \in \{1,\dots,C\},
\]
and instance IDs
\[
\mathcal{I}^r = \{I^r_1,\dots,I^r_N\}, \quad I^r_i \in \mathbb{N}.
\]
The semantic label specifies classes such as car, truck, pedestrian, or static, while the instance ID distinguishes individual objects of the same class [2507.06906].

The difficulty of the task is tied to the properties of automotive FMCW radar. A single scan contains only a few hundred points, approximately \(500\text{–}600\) on RadarScenes, and includes false returns and multipath reflections. Point density varies strongly with range: nearby objects may yield several points, whereas a car at \(70\text{–}80\,\mathrm{m}\) may be represented by only \(1\text{–}2\) points. Ego-motion and multipath introduce spurious detections, and small or rare classes such as pedestrians, bikes, or far vehicles are represented by very few points. The system is also required to operate faster than the radar frame rate, which is approximately \(17\,\mathrm{Hz}\) in RadarScenes [2507.06906].

Radar remains attractive because it is complementary to cameras and LiDAR. It works in adverse weather, directly measures Doppler velocity, and provides radar cross section (RCS) as an additional cue about material and size. SemRaFiner is designed to exploit these strengths, especially Doppler-based moving instance segmentation, and then refine the output into full panoptic predictions [2507.06906].

Evaluation follows panoptic metrics from the SemanticKITTI and panoptic LiDAR literature. The principal metrics are **Panoptic Quality** (\(\mathrm{PQ}\)), **mean IoU** (\(\mathrm{mIoU}\)), and per-class PQ and IoU. This choice is important because the method is not evaluated merely as semantic segmentation; it is explicitly assessed on the joint quality of semantic labeling and instance separation [2507.06906].

## 2. Two-stage system design

SemRaFiner is organized as a **two-stage, point-based transformer framework** [2507.06906]. The first stage is a **Moving Instance Segmentation Backbone**, specifically the **Radar Instance Transformer (RIT)**. The second stage is the **SemRaFiner refinement network**, which operates only on points predicted as moving.

RIT is treated as a black-box backbone. It receives the current scan \(\mathcal{P}^t\) together with \(T\) previous scans \(\mathcal{P}^{t-T},\dots,\mathcal{P}^{t-1}\), uses attention-based temporal aggregation and graph clustering internally, and outputs binary moving/static labels \(\mathcal{S}^{\mathrm{MOS}}\) and moving instance labels \(\mathcal{I}=\{I_1,\dots,I_N\}\) [2507.06906]. During training, SemRaFiner uses ground-truth moving labels for filtering; during inference, it uses the backbone predictions.

The refinement network receives only the moving points of the current scan, together with their coordinates and radar features. Its role is twofold: to assign semantic labels to moving points and to refine the instance assignments produced by RIT. Static points remain from the backbone output, and the merged result yields the full panoptic prediction [2507.06906].

The high-level workflow is therefore asymmetric. RIT is responsible for moving-object segmentation and initial instance grouping, while SemRaFiner performs density-aware feature extraction, semantic classification, and post hoc instance correction. This design suggests a deliberate decomposition of the radar panoptic problem: motion-based grouping is handled first, and semantic disambiguation is applied afterward in a radar-specific refinement stage [2507.06906].

The full system remains real time. RIT plus SemRaFiner requires approximately \(42\,\mathrm{ms}\) per scan, corresponding to about \(24\,\mathrm{Hz}\), which is faster than the RadarScenes data rate [2507.06906].

## 3. Input representation and the radius transformer

Each radar point is encoded as a \(5\)-dimensional feature vector
\[
\mathbf{x}_i = (x_i^C, y_i^C, z_i^C, \sigma_i, v_i),
\]
where \(x_i^C, y_i^C, z_i^C\) are 3D coordinates in a vehicle frame, \(\sigma_i\) is the radar cross section, and \(v_i\) is the ego-motion-compensated Doppler velocity. For the BEV representation used by the refinement stage, \(z_i^C = 0\), and the raw feature dimension is \(D=5\) [2507.06906].

The central architectural novelty is the **radius transformer layer**, which is designed for density-aware point processing. For moving points in the current scan, the network uses BEV coordinates
\[
\mathbf{P}^{\text{in}} = [\mathbf{p}_1,\dots,\mathbf{p}_{N^{\text{mov}}}]^\top \in \mathbb{R}^{N^{\text{mov}} \times 2}
\]
and input features
\[
\mathbf{X}^{\text{in}} = [\mathbf{x}_1,\dots,\mathbf{x}_{N^{\text{mov}}}]^\top \in \mathbb{R}^{N^{\text{mov}} \times D_1}.
\]
The raw radar features are first projected to a higher dimension, for example \(5 \to 64\), and then mapped to queries, keys, and values:
\[
\mathbf{Q} = \mathbf{X}^{\text{in}} \mathbf{W}_Q,\quad
\mathbf{K} = \mathbf{X}^{\text{in}} \mathbf{W}_K,\quad
\mathbf{V} = \mathbf{X}^{\text{in}} \mathbf{W}_V.
\]
The key design choice is neighborhood construction through a **ball query** rather than \(k\)-nearest neighbors. For each point,
\[
\mathbf{r}_{i,j} = \mathbf{p}_i - \mathbf{p}_j,
\]
and the neighborhood is defined as
\[
\mathcal{B}_r(\mathbf{p}_i) =
\left\{
\mathbf{p}_j \in \mathbf{P}^{\text{in}}
\mid
\|\mathbf{r}_{i,j}\| \le r
\right\}.
\]
This neighborhood is then capped at a maximum of \(N^{\text{max}}\) neighbors, with zero-padding if fewer are available [2507.06906].

This radius-based construction addresses the strongly non-uniform density of radar point clouds. In crowded regions, the radius remains fixed and the neighbor cap prevents over-connectivity; in sparse regions, the ball query still provides a physically meaningful local neighborhood. The paper explicitly contrasts this with \(k\)-NN attention, which may connect unrelated distant objects in sparse regions, and with grid-based attention, whose fixed cells may not align with object extents [2507.06906].

Relative positional encoding is implemented with two linear layers, BatchNorm, and ReLU, producing
\[
\mathbf{R} \in \mathbb{R}^{N^{\text{mov}} \times N^{\text{max}} \times D_1}.
\]
Vector attention is then formed channel-wise:
\[
\mathbf{A}_{i,j} = (\mathbf{Q}^r_{i,j} - \mathbf{K}^r_{i,j}) + \mathbf{R}_{i,j},
\]
followed by an MLP and softmax over neighbors, producing normalized weights \(\hat{\mathbf{A}}_{i,j}\). The output feature for point \(i\) is
\[
\mathbf{X}^{\text{out}}_i =
\sum_{j=1}^{N^{\text{max}}}
\hat{\mathbf{A}}_{i,j}
\odot
(\mathbf{V}^r_{i,j} + \mathbf{R}_{i,j}),
\]
with \(\odot\) denoting element-wise multiplication [2507.06906].

The layer is embedded in a residual **radius transformer block**. Two such blocks are used in SemRaFiner. After them, the network applies three MLPs in sequence: \(D_2 \to 128 \to 64\), then \(64 \to 64 \to 32\), and finally \(32 \to C\) for class logits. The full feature pipeline is raw \(5\)D input, linear projection to \(D_1=64\), block 1, block 2 with \(D_2=256\), then the MLP head [2507.06906].

## 4. Refinement strategy, losses, and augmentation

SemRaFiner is trained after RIT and uses three losses: cross-entropy for per-point semantic classification, Lovász loss as a convex surrogate for IoU, and a consistency loss \(\mathcal{L}_c\) that encourages semantic homogeneity within instances [2507.06906]. For each instance \(h\), if \(\mathcal{I}_h^{\mathrm{sem}}\) denotes the set of distinct semantic classes predicted within that instance, the consistency loss is
\[
\mathcal{L}_c =
\frac{1}{N_h}
\sum_{h=1}^{N_h}
\left(
1 - \frac{1}{|\mathcal{I}_h^{\mathrm{sem}}|}
\right).
\]
If an instance contains only one semantic class, the loss contribution is \(0\); if multiple classes are present, the loss increases. The total loss is the unweighted sum of cross-entropy, Lovász, and \(\mathcal{L}_c\) [2507.06906].

The paper emphasizes that \(\mathcal{L}_c\) is deliberately not too strong, because SemRaFiner must still be able to assign different semantics to points that should eventually be split into new instances. Removing \(\mathcal{L}_c\) yields a small but measurable drop, from \(\mathrm{PQ}=83.0\) to \(82.9\) on validation [2507.06906]. This suggests that the loss acts as a mild regularizer rather than as a hard constraint.

Instance refinement occurs in two forms. First, targeted augmentation teaches the network to correct static-versus-moving errors. Second, a post-processing step uses semantic predictions to split inconsistent RIT instances: if a backbone instance contains multiple semantic classes, it is separated into distinct instance IDs per class cluster [2507.06906]. A common misconception is that the model only predicts semantics on top of a fixed instance map. Validation results contradict that interpretation: ground-truth semantics combined with raw RIT instances without refinement yields \(\mathrm{PQ}=77.1\), whereas SemRaFiner with augmentation reaches \(\mathrm{PQ}=83.0\) [2507.06906].

A key contribution is **targeted augmentation** that reproduces typical RIT errors. Two augmentation modes are used during training. With probability \(p_I = 40\%\), static points near the boundary of ground-truth moving instances are added and marked as part of instances, so the network learns to remove them. With probability \(p_S = 40\%\), small static-only instances containing \(1\text{–}5\) static points are added and treated as moving in the input, so the network learns to discard them [2507.06906]. Without such augmentation, the network does not encounter the failure modes of the backbone during training.

Optimization uses PyTorch, AdamW with initial learning rate \(0.001\), step decay by a factor of \(0.1\) at epoch \(60\), \(80\) training epochs, batch size \(64\), and one NVIDIA A100. The default radius-transformer hyperparameters are \(D=5\), \(D_1=64\), \(D_2=256\), \(r=5.0\,\mathrm{m}\), and \(N^{\text{max}}=24\) [2507.06906].

## 5. Dataset, baselines, and empirical performance

Experiments use **RadarScenes**, described as the only large-scale public radar dataset with per-point labels for semantics and moving instances [2507.06906]. It contains \(158\) sequences with multiple weather conditions and driving scenarios, up to four radar sensors per scene, and follows the sparse single-scan representation from prior work. The average number of points in a single test scan is approximately \(539\). The train/validation/test split is \(130/6/22\) sequences [2507.06906].

The comparison includes multi-scan semantic baselines such as RadarPNv1, RadarPNv2, and STA-Net, as well as single-scan baselines such as Mask3D and Gaussian Radar Transformer plus DBSCAN. SemRaFiner uses RIT for moving instances and the refinement network for semantics and instance correction [2507.06906].

The principal test-set results are as follows [2507.06906]:

| Method | Input | PQ / mIoU |
|---|---:|---:|
| Mask3D | single | 56.9 / 56.1 |
| GRT + DBSCAN | single | 56.3 / 56.9 |
| SemRaFiner | single | **81.4 / 70.4** |
| STA-Net | multi-scan | – / 70.4 |

Per-class panoptic quality for SemRaFiner is \(99.7\) for static, \(85.7\) for car, \(59.2\) for pedestrian, \(83.1\) for pedestrian group, \(78.9\) for bike, and \(82.1\) for truck. Its per-class semantic IoU is \(99.4\) for static, \(74.9\) for car, \(42.1\) for pedestrian, \(71.0\) for pedestrian group, \(65.2\) for bike, and \(69.6\) for truck [2507.06906].

Several comparisons are especially consequential. The overall PQ rises from approximately \(56\%\) for Mask3D and GRT+DBSCAN to \(81.4\%\) for SemRaFiner. Pedestrian PQ rises from approximately \(18\text{–}30\%\) to \(59.2\%\), and pedestrian-group PQ reaches \(83.1\%\). The semantic result of \(\mathrm{mIoU}=70.4\%\) matches the multi-scan STA-Net result, despite SemRaFiner using only single scans and about \(539\) points rather than about \(3072\) [2507.06906]. This suggests that a strong moving-instance backbone plus density-aware refinement is substantially more effective in this regime than generic panoptic architectures adapted from LiDAR.

Ablation studies isolate the contribution of the radius transformer. On RadarScenes validation, replacing it with a Stratified Transformer layer yields \(\mathrm{PQ}=55.0\), \(\mathrm{mIoU}=42.6\); Point Transformer yields \(75.8/63.1\); KPConv yields \(81.0/69.6\); and the SemRaFiner radius transformer yields \(83.0/72.2\) [2507.06906]. The default setting \(r=5.0\), \(N^{\text{max}}=24\) is the best reported configuration. Reducing \(N^{\text{max}}\) to \(6\) lowers performance to \(\mathrm{PQ}=78.7\), \(\mathrm{mIoU}=66.4\), while larger radius or neighbor counts slightly hurt performance, especially for small objects [2507.06906].

The augmentation study further clarifies the refinement effect. On RadarScenes validation, ground-truth semantics combined with raw RIT instances gives \(\mathrm{PQ}=77.1\); SemRaFiner without augmentation gives \(81.5\); and SemRaFiner with augmentation gives \(83.0\) [2507.06906]. The paper interprets this as evidence that the network actively corrects instance assignments rather than merely attaching better semantics to fixed clusters.

Runtime and parameter counts are also reported [2507.06906]:

| Model | Params (M) | Mean runtime (ms) |
|---|---:|---:|
| Mask3D | 39.6 | 85.2 |
| GRT | 8.4 | 24.0 |
| RIT | 3.8 | 31.7 |
| RIT + SemRaFiner | **4.5** | **42.1** |

SemRaFiner therefore adds about \(0.7\) million parameters and about \(10.4\,\mathrm{ms}\) overhead to RIT [2507.06906].

## 6. Qualitative behavior, limitations, and research context

Qualitative examples in foggy and urban scenes show three recurring effects: better separation of close instances, cleaning of static false positives, and improved coherence for small objects such as pedestrians and bikes [2507.06906]. In particular, RIT may merge a car and a truck into one motion-consistent cluster, and SemRaFiner can split them using per-point semantic cues. It can also reclassify static parked cars that were erroneously marked as moving by the backbone [2507.06906].

The method nonetheless depends on the quality of the moving-instance backbone. If RIT misses a moving instance entirely, SemRaFiner cannot recover it. Very sparse or extremely distant objects remain difficult to classify correctly, and some underrepresented classes, such as bike, are less accurate than cars [2507.06906]. The hyperparameters \(r\), \(N^{\text{max}}\), and augmentation settings are tuned for RadarScenes and may require adaptation for different radar configurations [2507.06906].

Within radar perception, SemRaFiner sits at the intersection of radar semantic understanding, panoptic segmentation, and transformer-based point processing. Compared with prior radar work, RadarPNv1 and RadarPNv2 provide multi-scan semantic segmentation only; STA-Net also targets semantic segmentation with multi-scan aggregation; GRT performs single-scan semantic segmentation; and RIT performs moving instance segmentation exploiting Doppler [2507.06906]. SemRaFiner combines these lines of development into a full radar panoptic system.

The paper states that SemRaFiner demonstrates, for the first time, state-of-the-art radar panoptic segmentation on single scans, with performance comparable to multi-scan semantic-only methods and large improvements over LiDAR-panoptic-inspired baselines when applied to radar [2507.06906]. A plausible implication is that radar panoptic perception benefits from a decomposition that is native to the sensing modality: Doppler-powered moving instance segmentation first, then density-aware semantic refinement. The reported real-time operation and adverse-weather relevance make that decomposition particularly pertinent for autonomous driving scenarios in which cameras and LiDAR degrade [2507.06906].

Source: https://www.emergentmind.com/topics/semrafiner