---
title: RangeViT for 3D LiDAR Segmentation
url: https://www.emergentmind.com/topics/rangevit
type: topic
---

# RangeViT for 3D LiDAR Segmentation

RangeViT is a method for 3D semantic segmentation of outdoor LiDAR point clouds that adapts vision transformers (ViTs) to work within the range-projection framework. By mapping point clouds onto 2D range images and leveraging image-pretrained ViT architectures in conjunction with convolutional components, RangeViT achieves state-of-the-art accuracy among projection-based methods while retaining fast and efficient inference. The method is designed to combine the computational benefits of 2D projections with the representational power of large-scale vision transformers, yielding strong performance on datasets such as nuScenes and SemanticKITTI [2301.10222].

## 1. Range Projection and Data Encoding

RangeViT operates on input point clouds $P \in \mathbb{R}^{N \times 4}$, where each point $p = (x, y, z, i)$ comprises three Cartesian coordinates and an intensity value. The method applies a spherical or range projection to map each 3D point to 2D image coordinates $(h, w)$ for a fixed image size $H \times W$. The mapping is defined by:

- Range: $r = \sqrt{x^2 + y^2 + z^2}$
- Vertical FoV: $f_\text{up}>0$ (upper angle), $f_\text{down}<0$ (lower angle), $f_v = |f_\text{down}|+|f_\text{up}|$ (total FoV)
- Pixel assignment:
  $$
  \begin{pmatrix}
    h \\
    w
  \end{pmatrix}
  =
  \begin{pmatrix}
    \frac{1}{2}\left( 1 - \frac{\arctan2(y, x)}{\pi} \right) W \\
    \left( 1 - \frac{\arcsin(z/r) + |f_\text{down}|}{f_v} \right) H
  \end{pmatrix}
  $$

A 5-channel range image $I \in \mathbb{R}^{5 \times H \times W}$ is assembled by recording $(r, x, y, z, i)$ at each $(h, w)$ for the closest point, zero-filling empty pixels.

## 2. Model Architecture

### 2.1 Convolutional Stem
The convolutional stem replaces the standard linear patch embedding with a deeper, nonlinear tokenization more suited to the idiosyncrasies of range images. Specifically, it uses four SalsaNext residual blocks (the “context module”) to produce an intermediate tensor $t_c \in \mathbb{R}^{D_h \times H \times W}$ (default $D_h = 256$), with two initial channel expansions (5→32 and 32→$D_h$). Average pooling reduces the spatial resolution, and a $1 \times 1$ convolution projects the features to a sequence of visual tokens $v_1,\dots,v_M \in \mathbb{R}^D$.

### 2.2 ViT Encoder
Visual tokens are stacked with a class token and added to learnable positional embeddings.
Transformer blocks ($L = 12$) apply multi-head self-attention (MSA) and layer-wise feed-forward networks (FFN) with LayerNorm and residual connections:

- Input: $[v_\text{class}; v_1; \dots; v_M] \in \mathbb{R}^{(M+1) \times D}$
- Each block:
  $$
  \begin{aligned}
  T_\ell' &= \text{LayerNorm}(T_{\ell-1}) + \text{MSA}(\text{LayerNorm}(T_{\ell-1})) \\
  T_\ell &= T_\ell' + \text{FFN}(\text{LayerNorm}(T_\ell'))
  \end{aligned}
  $$

The class token is discarded at the end of encoding.

### 2.3 Convolutional Decoder and Skip Connection
Output tokens are reshaped, passed through a $1 \times 1$ convolution to expand channels, then upsampled to the original image size using PixelShuffle. The upsampled decoder outputs are concatenated with features from the convolutional stem ($t_c$), followed by additional convolutional layers (3×3 and 1×1, BatchNorm, LeakyReLU) to recover dense spatial information.

### 2.4 3D Refiner (KPConv)
Decoded 2D features are bilinearly sampled at the original 3D point coordinates to yield per-point features $p_f \in \mathbb{R}^{N \times D_h}$. These are further refined using a KPConv layer, which updates features using local 3D neighborhoods. The final point-wise linear layer infers semantic logits for each point.

## 3. Principal Design Components

Three essential design choices underlie RangeViT's accuracy and transferability:

- **Image-Pretrained ViT Backbones:** Unmodified ViT encoder architectures enable initialization from large-scale image collections such as ImageNet-21k (and subsequent fine-tuning on Cityscapes or with self-supervised DINO). This yields consistent improvements (+2–3 pp mIoU) and faster convergence compared to random initialization.
- **Convolutional Stem Instead of Linear Embedding:** The convolutional stem injects spatial locality and nonlinearity, adapting range images to match the distribution of data used in ViT pretraining. Replacement with a linear embedding results in a ~4 pp mIoU drop.
- **Convolutional Decoder with Skip Connection:** A lightweight decoder and a single skip connection (from the convolutional stem to the feature decoder) efficiently merge fine-grained and semantic representations. Omission of the skip connection or of convolutional upsampling also incurs a ~4 pp mIoU penalty.

## 4. Training Regimen and Datasets

RangeViT is evaluated on two principal datasets:

- **nuScenes:** 32×2048 range images, 28,130 train / 6,019 val, 16 classes.
- **SemanticKITTI:** 64×2048 range images, 19,130 train / 4,071 val, 19 classes (test: sequences 11–21).

ViT-S/16 models ($L=12$, $D=384$, 6 heads) are pretrained on ImageNet-21k, further fine-tuned via Segmenter on Cityscapes. Stem/decoder/KPConv are randomly initialized. Optimization uses AdamW ($\beta_1=0.9$, $\beta_2=0.999$, weight decay $1 \times 10^{-2}$), batch sizes 32 (nuScenes) or 16 (KITTI), and cosine annealing schedule. Data augmentations include random flips, 3D translations, random rotations (±5° roll/pitch/yaw), and random cropping. Inference uses sliding-window averaging.

## 5. Empirical Performance

### 5.1 Quantitative Results

**nuScenes (val set, mIoU %):**
| Method             | mIoU |
|--------------------|------|
| RangeNet++         | 65.5 |
| SalsaNext          | 72.2 |
| PolarNet           | 71.0 |
| RangeViT-IN21k     | 74.8 |
| RangeViT-CS        | 75.2 |

**SemanticKITTI (test set, mIoU %):**
| Method         | mIoU |
|----------------|------|
| RangeNet++     | 52.2 |
| SqueezeSegV3   | 55.9 |
| SalsaNext      | 59.5 |
| KPRNet         | 63.1 |
| Lite-HDSeg     | 63.8 |
| RangeViT-CS    | 64.0 |

RangeViT narrows the gap to top-performing voxel-based approaches (e.g., Cylinder3D: 76.1% nuScenes, 67.8% SemanticKITTI), while maintaining the computational speed characteristic of projection-based methods.

### 5.2 Ablation Studies and Analysis

Selected ablation results (nuScenes val, mIoU):

| Stem   | Decoder | 3D Refiner | mIoU |
|--------|---------|------------|------|
| Linear | Linear  | —          | 65.5 |
| Conv   | Linear  | —          | 69.8 |
| Conv   | UpConv  | —          | 73.8 |
| Conv   | UpConv  | KPConv     | 74.6 |

Pretraining initialization (mIoU): Random 72.4%, DINO 73.3%, IN21k 74.8%, Cityscapes 75.2%.

Partial fine-tuning, freezing attention weights and training only FFN+LN, achieves 75.5% vs 75.2% for full fine-tuning. Non-square $2 \times 8$ patches outperform standard $16 \times 16$ (75.2% vs 68.5%).

## 6. Strengths, Limitations, and Research Outlook

**Strengths:**
- Efficient reuse of image-pretrained "foundation" vision transformers for 3D LiDAR data.
- Minimal architectural changes; no custom attention mechanisms are required.
- Real-time inference capability (approximately 25 ms per frame on RTX2080).
- State-of-the-art among 2D projection-based segmentation methods.

**Known Limitations:**
- Some loss of 3D local neighborhood structure due to projection is only partly recoverable by the post-hoc KPConv refiner.
- Performance on sparse classes (e.g., bicycles, traffic cones) remains challenging.
- Limitations arise from the fixed nature of the projection and the inductive bias mismatch between image and LiDAR data.

**Future Directions:**
- Exploration of flexible tokenizers (e.g., FlexiViT, Perceiver IO) for end-to-end 3D tokenization.
- Development of alternatives to projection-based encoding and increased integration of multi-modal data (e.g., combining LiDAR with camera imagery).

RangeViT demonstrates that with a carefully architected convolutional stem, efficient decoder, and principled use of image-pretrained ViT backbones, projection-based 3D semantic segmentation can achieve accuracy that approaches more computationally intensive voxel-based methods, while preserving computational efficiency and architectural simplicity [2301.10222].

Source: https://www.emergentmind.com/topics/rangevit