UniD-Shift: Joint 2D-3D Scene Segmentation

• UniD-Shift is a multimodal framework that performs joint 2D-3D semantic segmentation by decomposing features into shared modality-invariant and private modality-specific components.
• The method fuses only the shared features through a lightweight Shared Attention Fusion module, addressing issues like sparse sampling, view distortion, and misalignment in large-scale scenes.
• Empirical evaluations on nuScenes and SemanticKITTI demonstrate competitive mIoU scores, with ablation studies highlighting the benefits of Gram alignment and decorrelation losses.

UniD-Shift is a multimodal framework for joint 2D-3D semantic segmentation of large-scale outdoor scenes. It addresses per-point semantic labeling in LiDAR point clouds while exploiting synchronized RGB imagery as an additional semantic source, and it is organized around an explicit shared-private decomposition: features from the 2D and 3D branches are split into shared modality-invariant components and private modality-specific components, only the shared components are fused, and regularization is used to enforce both semantic alignment and subspace independence (Zhang et al., 8 May 2026).

1. Problem setting and guiding premise

UniD-Shift is designed for semantic segmentation of large-scale 3D point clouds in applications such as autonomous driving and urban digital twins. The target output is a semantic label for each 3D LiDAR point, with categories such as car, road, and vegetation. The framework is motivated by four difficulties in 2D-3D fusion: sparse and non-uniform LiDAR sampling, view-dependent image distortions, noisy cross-modal alignment caused by projection and occlusion, and unstable fusion when entire feature tensors are mixed through concatenation or heavy attention. The paper also identifies domain shifts—such as differences between USA and Singapore scenes in nuScenes—as a major source of degradation for multimodal models (Zhang et al., 8 May 2026).

The central premise is that 2D images and 3D point clouds are different views of the same physical world. They therefore contain common semantic factors, such as object categories and spatial relations, but also modality-specific factors. In the formulation used by UniD-Shift, image features encode appearance, color, texture, and perspective distortions, whereas LiDAR features encode precise but sparse geometry, sampling patterns, and occlusions. Rather than performing full-feature fusion, the method decomposes each modality into a shared component and a private component. This yields a more compact latent structure in which semantic alignment is handled in the shared space and modality-specific information is preserved in private subspaces.

The framework is called unified because a single model is trained jointly for 2D and 3D semantic segmentation. The resulting latent space is structured rather than monolithic: shared factors are intended to be semantically consistent across modalities, and private factors are intended to remain decorrelated.

2. Encoder branches, projection, and point-level alignment

The overall pipeline is a dual-branch encoder-fusion-decoder architecture. The 3D branch takes a LiDAR point cloud and uses SPTNet as its backbone. The 2D branch takes synchronized RGB images and uses a SAM ViT-L vision encoder. The two branches are brought into correspondence by projecting 3D points into the image plane and sampling aligned 2D features for visible points (Zhang et al., 8 May 2026).

The 2D branch uses official SAM weights, sam_vit_1_0b3195.pth, and the SAM encoder is frozen. Only projection layers, fusion modules, and decoders are trainable. The paper attributes several effects to freezing SAM: reduced training instability, lower GPU memory usage because no gradients are stored for SAM, and preservation of SAM’s segmentation priors. Multi-scale SAM features are projected into a latent space of dimension $d=256$ through a linear layer and LayerNorm: $$F^{2D}_{proj}(u,v) = \text{LN}(W_{2D} F^{2D}_{img}(u,v) + b_{2D}),\quad F^{2D}_{proj} \in \mathbb{R}^{H \times W \times 256}.$$

For a LiDAR point $\mathbf{x}=(x,y,z)$, the 2D-3D lifting step applies camera extrinsics and intrinsics: $$\begin{bmatrix} u \ v \ 1 \end{bmatrix} \propto K \, T_{\text{cam}\leftarrow \text{lidar}} \begin{bmatrix} x \ y \ z \ 1 \end{bmatrix}.$$ If $(u,v)$ is inside the image and visible, the corresponding projected feature is sampled as

$f^{2D}_i = F^{2D}_{proj}(u_i, v_i).$

This produces a per-point 2D feature matrix $F^{2D} \in \mathbb{R}^{N_p \times 256}$.

The 3D branch uses SPTNet, described as a hybrid of sparse convolutions and transformers for large-scale outdoor point clouds. The input point cloud is voxelized into a 3D grid, only non-empty voxels are processed, and each of six SpConv blocks aggregates local neighbors. A transformer block follows each SpConv block, performing multi-head self-attention over voxel or point features and then adding the output back through a skip connection. After inverse voxel mapping, the 3D branch outputs

$F^{3D} \in \mathbb{R}^{N_p \times d},\quad d=256.$

Because both branches are projected into the same latent dimensionality and the 2D features are sampled per point, the aligned inputs to the decomposition stage are

$$F^{3D} = [f^{3D}_1,\ldots,f^{3D}_{N_p}]^\top,\quad F^{2D} = [f^{2D}_1,\ldots,f^{2D}_{N_p}]^\top.$$

3. Shared-private decomposition and Shared Attention Fusion

The defining operation in UniD-Shift is the explicit decomposition of each modality into shared and private subspaces. For each modality $m \in \{2D,3D\}$, the paper writes

$$F^{2D}_{proj}(u,v) = \text{LN}(W_{2D} F^{2D}_{img}(u,v) + b_{2D}),\quad F^{2D}_{proj} \in \mathbb{R}^{H \times W \times 256}.$$0

where $$F^{2D}_{proj}(u,v) = \text{LN}(W_{2D} F^{2D}_{img}(u,v) + b_{2D}),\quad F^{2D}_{proj} \in \mathbb{R}^{H \times W \times 256}.$$1 denotes the shared modality-invariant semantic component and $$F^{2D}_{proj}(u,v) = \text{LN}(W_{2D} F^{2D}_{img}(u,v) + b_{2D}),\quad F^{2D}_{proj} \in \mathbb{R}^{H \times W \times 256}.$$2 denotes the private modality-specific component (Zhang et al., 8 May 2026).

The decomposition is implemented by learnable linear projections with normalization: $$F^{2D}_{proj}(u,v) = \text{LN}(W_{2D} F^{2D}_{img}(u,v) + b_{2D}),\quad F^{2D}_{proj} \in \mathbb{R}^{H \times W \times 256}.$$3 The paper states that the concatenation $$F^{2D}_{proj}(u,v) = \text{LN}(W_{2D} F^{2D}_{img}(u,v) + b_{2D}),\quad F^{2D}_{proj} \in \mathbb{R}^{H \times W \times 256}.$$4 can reconstruct or approximate $$F^{2D}_{proj}(u,v) = \text{LN}(W_{2D} F^{2D}_{img}(u,v) + b_{2D}),\quad F^{2D}_{proj} \in \mathbb{R}^{H \times W \times 256}.$$5. It also uses explicit notation such as $$F^{2D}_{proj}(u,v) = \text{LN}(W_{2D} F^{2D}_{img}(u,v) + b_{2D}),\quad F^{2D}_{proj} \in \mathbb{R}^{H \times W \times 256}.$$6, $$F^{2D}_{proj}(u,v) = \text{LN}(W_{2D} F^{2D}_{img}(u,v) + b_{2D}),\quad F^{2D}_{proj} \in \mathbb{R}^{H \times W \times 256}.$$7, and $$F^{2D}_{proj}(u,v) = \text{LN}(W_{2D} F^{2D}_{img}(u,v) + b_{2D}),\quad F^{2D}_{proj} \in \mathbb{R}^{H \times W \times 256}.$$8, indicating structured latent widths for shared and fused features.

To align the shared semantic subspaces, UniD-Shift introduces a Gram matrix alignment loss. For shared features $$F^{2D}_{proj}(u,v) = \text{LN}(W_{2D} F^{2D}_{img}(u,v) + b_{2D}),\quad F^{2D}_{proj} \in \mathbb{R}^{H \times W \times 256}.$$9 and $\mathbf{x}=(x,y,z)$0, define

$\mathbf{x}=(x,y,z)$1

The alignment loss is

$\mathbf{x}=(x,y,z)$2

This regularizes the second-order statistics of the shared spaces so that semantic relations among points are aligned across modalities.

To keep private subspaces modality-specific, UniD-Shift applies a decorrelation loss. With private features $\mathbf{x}=(x,y,z)$3 and $\mathbf{x}=(x,y,z)$4,

$\mathbf{x}=(x,y,z)$5

and

$\mathbf{x}=(x,y,z)$6

The intended effect is to reduce cross-modal leakage in the private channels and improve disentanglement.

Fusion is restricted to the shared components through the Shared Attention Fusion (SAF) module. The attention direction is asymmetric: 3D shared features act as queries, and 2D shared features act as keys and values. Formally,

$\mathbf{x}=(x,y,z)$7

with attention weights

$\mathbf{x}=(x,y,z)$8

The paper also writes this as

$\mathbf{x}=(x,y,z)$9

The fused shared representation is denoted $$\begin{bmatrix} u \ v \ 1 \end{bmatrix} \propto K \, T_{\text{cam}\leftarrow \text{lidar}} \begin{bmatrix} x \ y \ z \ 1 \end{bmatrix}.$$0. The text gives

$$\begin{bmatrix} u \ v \ 1 \end{bmatrix} \propto K \, T_{\text{cam}\leftarrow \text{lidar}} \begin{bmatrix} x \ y \ z \ 1 \end{bmatrix}.$$1

but explicitly notes a typographical ambiguity in this equation, since both terms mention $$\begin{bmatrix} u \ v \ 1 \end{bmatrix} \propto K \, T_{\text{cam}\leftarrow \text{lidar}} \begin{bmatrix} x \ y \ z \ 1 \end{bmatrix}.$$2. The intended behavior is to combine 2D and 3D shared features under attention guidance, with 3D queries making the fusion point-centric and 2D features injecting image semantics. The final 3D segmentation feature is then

$$\begin{bmatrix} u \ v \ 1 \end{bmatrix} \propto K \, T_{\text{cam}\leftarrow \text{lidar}} \begin{bmatrix} x \ y \ z \ 1 \end{bmatrix}.$$3

Ablation results reported in the paper make the asymmetry explicit: using 3D queries and 2D keys yields the best performance, at $$\begin{bmatrix} u \ v \ 1 \end{bmatrix} \propto K \, T_{\text{cam}\leftarrow \text{lidar}} \begin{bmatrix} x \ y \ z \ 1 \end{bmatrix}.$$4 mIoU, compared with $$\begin{bmatrix} u \ v \ 1 \end{bmatrix} \propto K \, T_{\text{cam}\leftarrow \text{lidar}} \begin{bmatrix} x \ y \ z \ 1 \end{bmatrix}.$$5 for $$\begin{bmatrix} u \ v \ 1 \end{bmatrix} \propto K \, T_{\text{cam}\leftarrow \text{lidar}} \begin{bmatrix} x \ y \ z \ 1 \end{bmatrix}.$$6 and $$\begin{bmatrix} u \ v \ 1 \end{bmatrix} \propto K \, T_{\text{cam}\leftarrow \text{lidar}} \begin{bmatrix} x \ y \ z \ 1 \end{bmatrix}.$$7 for symmetric $$\begin{bmatrix} u \ v \ 1 \end{bmatrix} \propto K \, T_{\text{cam}\leftarrow \text{lidar}} \begin{bmatrix} x \ y \ z \ 1 \end{bmatrix}.$$8.

4. Optimization objective, training regime, and computational profile

UniD-Shift is trained jointly for 2D and 3D segmentation with segmentation, distillation, and decomposition terms. The total loss is

$$\begin{bmatrix} u \ v \ 1 \end{bmatrix} \propto K \, T_{\text{cam}\leftarrow \text{lidar}} \begin{bmatrix} x \ y \ z \ 1 \end{bmatrix}.$$9

For each modality $(u,v)$0,

$(u,v)$1

The cross-entropy term is weighted for class imbalance, and the Lovász-Softmax term serves as a surrogate for IoU. Cross-modal knowledge distillation is introduced through a bidirectional KL divergence: $(u,v)$2 The implementation hyperparameters reported are

$(u,v)$3

These terms jointly supervise per-point 3D predictions, per-pixel 2D predictions, prediction consistency, shared-space alignment, and private-space independence (Zhang et al., 8 May 2026).

Training is performed in PyTorch on $(u,v)$4 NVIDIA RTX 4090 GPUs. The main paper reports 100 epochs, while the supplementary material mentions 80 epochs with a cosine learning-rate schedule. Optimization uses SGD with momentum $(u,v)$5, weight decay $(u,v)$6, and initial learning rate $(u,v)$7. Mixed-precision training is used, and the batch size is 8 samples per GPU. Data augmentation includes random rotation, flipping, and uniform scaling for 3D inputs, while 2D inputs use photometric perturbations. Projection indices for 2D-3D correspondence are precomputed from calibration to accelerate training and inference.

The datasets are nuScenes, with 16 semantic classes and six cameras, and SemanticKITTI, with 19 semantic classes. The paper evaluates both single-domain segmentation and cross-domain transfer from nuScenes USA to Singapore.

The computational profile is dominated by the frozen SAM backbone. On SemanticKITTI, UniD-Shift with SAM has 359.8M total parameters, of which 35.2M are trainable. Reported baselines include UniSeg at 147.6M, CSFNet at 45.4M, and UniDSeg (SAM) at approximately 353M total with 38.7M trainable. Latency is 240 ms per frame on SemanticKITTI, compared with 61.1 ms for MinkowskiNet, 65.9 ms for Cylinder3D, and 63.8 ms for SPVCNN. The paper characterizes the shared-private fusion itself as lightweight relative to heavy cross-attention stacks.

5. Empirical performance, ablations, and domain shift

On nuScenes validation, UniD-Shift reports 81.0% mIoU, matching U2MKD and slightly exceeding 2DPASS at 80.5% and CSFNet (large) at 80.8%. On the nuScenes test set, UniD-Shift reports 81.2% mIoU. In the same comparison group, PMF reports 77.0%, 2D3DNet 80.0%, CMDFusion 80.8%, MSeg3D 81.1%, U2MKD 84.2%, and TASeg 84.6%, while a LiDAR-only PTv3+PPT model reaches 83.1%. The paper therefore presents UniD-Shift as competitive on nuScenes, but not the absolute top method on the test set (Zhang et al., 8 May 2026).

On SemanticKITTI validation, UniD-Shift reports 71.8% mIoU for fusion, compared with 64.9% for Cylinder3D, 63.9% for PMF, 70.0% for 2DPASS, 69.0% for MSeg3D, 71.3% for UniSeg, and 71.5% for CSFNet (large). The paper identifies 71.8% as the best reported validation mIoU in that table and notes improvements on vegetation and small objects such as traffic signs.

The ablations isolate three effects. First, backbone choice matters: a baseline with ResNet+SPVCNN+KL reaches 68.2%; replacing SPVCNN with SPTNet raises this to 70.1%; replacing ResNet with SAM in an SPTNet+SAM+KL configuration yields 73.4%. Second, shared-private fusion improves over KL-only alignment: SAM+SPTNet + KL reaches 73.4%, while SAM+SPTNet + Shared-Private Fusion reaches 74.8%. Third, the regularizers are not redundant: removing Gram alignment drops performance to 73.7%; removing decorrelation gives 74.4%; removing both is worse still. The reported pattern is that Gram alignment contributes more strongly than decorrelation.

The cross-domain experiment from nuScenes USA to Singapore is the most direct test of generalization under distribution shift. UniD-Shift reports 73.3% for the 2D branch, 68.8% for the 3D branch, and 74.5% mIoU for the multimodal model. Reported multimodal baselines are xMUDA at 69.4%, UniDSeg at 72.9%, MM2D3D at 72.4%, and UniDxMD at 74.3%. The paper identifies UniD-Shift’s 74.5% as the best cross-domain multimodal mIoU in that comparison and interprets this as evidence that the shared-private decomposition improves transferability.

Qualitative results are described as showing cleaner object boundaries, fewer fragmented patches, and more consistent regions than SPVCNN, PointTransformerV2, 2DPASS, and CSFNet, especially in sparse zones and dense urban scenes with complex depth variation. The principal failure mode reported is on very distant, low-resolution objects, where both LiDAR sampling density and image detail are weak.

6. Interpretability, limitations, and position in the literature

Interpretability in UniD-Shift is architectural rather than post hoc. The model explicitly separates shared and private features, fuses only shared components, and regularizes the decomposition with Gram alignment for the shared space and decorrelation for the private space. The paper associates shared features with semantic categories and cross-modal consistency, private 2D features with texture, appearance, and lighting variation, and private 3D features with LiDAR sampling patterns and fine geometry (Zhang et al., 8 May 2026).

Within the 2D-3D segmentation literature, the method is positioned against 2DPASS, PMF, CSFNet, MSeg3D, UniSeg, xMUDA, UniDSeg, and UniDxMD. The paper’s stated distinction is that many prior methods either fuse complete feature representations through concatenation or heavy cross-attention, or treat fusion and domain adaptation as separate problems. UniD-Shift instead combines a frozen SAM-based vision encoder, an SPTNet geometric encoder, a lightweight SAF module that fuses only shared features, and a regularized shared-private decomposition within a single framework.

Several limitations are stated directly. The method depends on accurate LiDAR-camera calibration and temporal synchronization; misalignment degrades correspondence quality. The visual backbone is heavy: SAM ViT-L is frozen but still increases memory footprint and inference cost. Performance remains weak on very sparse far-range targets. Suggested future directions are more compact fusion structures, lighter encoders, adaptive alignment mechanisms for imperfect calibration and synchronization, extension to more modalities such as radar, temporal modeling for 4D segmentation, and more advanced unsupervised domain adaptation.

In that sense, UniD-Shift is best understood not as a generic fusion heuristic but as a structured multimodal segmentation framework in which semantic alignment is confined to a designated shared subspace and modality-specific evidence is preserved explicitly in private channels. Its empirical profile is correspondingly specific: competitive single-domain performance on nuScenes, best reported validation performance in the cited SemanticKITTI comparison, and strong cross-domain robustness on nuScenes USA-to-Singapore.