MSeg3D: Multi-Modal 3D Segmentation

• MSeg3D is a multi-modal framework that integrates LiDAR and camera images to deliver state-of-the-art 3D semantic segmentation, especially for small and distant objects.
• It employs a three-stage fusion pipeline—geometry-based fusion, cross-modal completion, and semantic-based aggregation—to effectively align heterogeneous sensor data.
• Asymmetric data augmentation and comprehensive loss functions ensure robustness against sensor failures, outperforming LiDAR-only methods in autonomous driving scenarios.

MSeg3D is a multi-modal 3D semantic segmentation framework designed for autonomous driving that jointly leverages LiDAR point clouds and multi-camera images. Addressing intrinsic challenges of fusing heterogeneous modalities, the approach introduces a staged architecture for feature extraction, alignment, and semantic aggregation, producing state-of-the-art segmentation accuracy, particularly on small and distant objects where LiDAR-only methods fail. MSeg3D integrates novel multi-modal fusion strategies, asymmetric data augmentation, and a comprehensive supervisory regime, demonstrating robustness under missing camera views and multi-frame LiDAR settings (Li et al., 2023).

1. Model Architecture and Intra/Inter-Modal Processing

MSeg3D processes input data from two primary sensors:

• LiDAR: $N_p$ points $$P_\mathrm{in} \in \mathbb{R}^{N_p \times C_\mathrm{in}}$$ (e.g., $(x, y, z)$, intensity)
• Cameras: $N_c$ images $$X_\mathrm{in} \in \mathbb{R}^{N_c \times 3 \times H_\mathrm{in} \times W_\mathrm{in}}$$

Feature extraction proceeds via jointly-trained intra-modal backbones:

• LiDAR backbone: Sparse 3D U-Net (e.g., from OpenPCDet) on voxelized input, yielding per-voxel features $V \in \mathbb{R}^{N_v \times C_\mathrm{voxel}}$.
• Camera backbone: 2D CNN (default HRNet-w48) yields multi-scale image feature maps $$X \in \mathbb{R}^{N_c \times C_\mathrm{img} \times H \times W}$$.

Subsequently, all segmentation points are fused in a three-stage inter-modal pipeline:

1. GF-Phase (Geometry-based feature fusion)
2. Cross-Modal Feature Completion
3. SF-Phase (Semantic-based feature fusion)

The final point-wise logits are produced by a point head operating on fused features.

2. Multi-Modal Feature Fusion Pipeline

2.1 GF-Phase: Geometry-Based Fusion

For each 3D point,

• LiDAR features: Derived from trilinear interpolation of features from the 3 nearest voxels, producing $$f_{\mathrm{lidar},i} \in \mathbb{R}^{C_\mathrm{voxel}}$$.
• Camera features: Bilinearly interpolated using known 3D-2D projections, $f_{\mathrm{cam},i} \in \mathbb{R}^{C_\mathrm{img}}$, with out-of-FOV points zero-masked by $B_i \in \{0,1\}$.

Both features are projected to $C_\mathrm{int}$ dimensions, concatenated, and fed to an MLP to yield $$f_{\mathrm{gfused},i} \in \mathbb{R}^{C_\mathrm{gfused}}$$.

2.2 Cross-Modal Feature Completion

For points outside camera FOV, MSeg3D hallucinated “pseudo-camera” features $$F_{\mathrm{pcam}} = \mathcal{H}_\mathrm{pcam}(F_\mathrm{lidar})$$ using an MLP. Training enforces feature distribution alignment (mean squared error) on those within FOV. At inference, missing camera features are replaced by these predictions, bridging field-of-view gaps.

2.3 SF-Phase: Semantic-Based Fusion

Semantic aggregation pools information across class categories:

• LiDAR semantic aggregation: Voxel segmentation head produces per-class scores; per-class voxel softmax and weighted averaging generate $$E_{\mathrm{lidar}} \in \mathbb{R}^{N_\mathrm{cls} \times C_\mathrm{voxel}}$$.
• Camera semantic aggregation: FCN-style head and per-class pixel softmax compute $$E_{\mathrm{cam}} \in \mathbb{R}^{N_\mathrm{cls} \times C_\mathrm{img}}$$.
• Both $E_{\mathrm{lidar}}$ and $E_{\mathrm{cam}}$ are projected, concatenated, and refined with $K$ blocks of Multi-Head Self/ Cross-Attention (MHSA/MHCA) and FFN layers (SFFM). Output $$F_\mathrm{sfused} \in \mathbb{R}^{N_p \times C_\mathrm{sfused}}$$ is used for final segmentation.

3. Multi-Modal and Asymmetric Data Augmentation

A key property is the use of asymmetric data augmentation:

• LiDAR-only augmentations: Large 3D rotations ($\pm 45^\circ$), translation ($\sigma = 0.5\,\mathrm{m}$), scaling ($0.95$--$1.05$), random flipping.
• Camera-only augmentations: Scale ($1.0$--$1.5$), small rotation ($\pm 1^\circ$), random crop to $(H_\mathrm{in}, W_\mathrm{in})$, color jitter, JPEG compression, etc.
• Symmetric augmentations: e.g., horizontal flip. The point-to-pixel correspondence index is kept fixed. Permuting these augmentations maximizes cross-modal diversity, improving generalization.

4. Loss Functions and Supervision Regime

MSeg3D incorporates a comprehensive set of loss functions:

• $L_\mathrm{point}$: Per-point segmentation (cross-entropy + Lovász-softmax) over 3D ground truth labels.
• $L_\mathrm{p2v}$: Voxel-level segmentation supervision, using point→voxel majority voting.
• $L_\mathrm{point2pixel}$: Segmentation supervision on downsampled camera images, projecting point labels to pixels.
• $L_\mathrm{pixel2point}$: Cross-modal completion loss aligning hallucinated pseudo-camera features with image features for points within FOV.

The overall loss, $$L = \alpha_1 L_\mathrm{point} + \alpha_2 L_\mathrm{p2v} + \alpha_3 L_\mathrm{point2pixel} + \alpha_4 L_\mathrm{pixel2point}$$, with weights $(1.0, 1.0, 0.5, 1.0)$, enforces consistency and mutual learning across modalities.

5. Empirical Performance, Robustness, and Ablations

Quantitative Results

Dataset	LiDAR-only SOTA	MSeg3D Result	Notes
nuScenes (test)	81.12 mIoU	81.14 mIoU, fwIoU 91.35	+4 pts over PMF; +8 pts for pedestrians; +6 pts for cones
Waymo (test)	71.13	70.51	+3 pts over other fusion (multi-cam rear missing)
SemanticKITTI (val)	64.9	66.7 mIoU¹	Evaluated inside camera FOV

¹ mIoU¹ is computed only inside camera FOV.

Ablation Studies

Variant	nuScenes mIoU	Waymo mIoU
GF-Phase only	72.4	63.9
+Cross-modal completion	+4.0 pts	+3.2 pts
+SF-Phase	+3.5 pts	+2.8 pts
+Asymmetric Augmentation	+0.9 pts	+3.6 pts

Improvements accumulate; the gap between “all points” and “inside FOV” drops below 1 point when all fusions are included.

Robustness Experiments

• Camera malfunction: With 6→0 cameras, mIoU drops from 80.0 to 74.5, yet still exceeds the LiDAR-only baseline (72.0).
• Multi-frame LiDAR: With 25 accumulating frames, MSeg3D reaches 81.1 mIoU (vs. LiDAR-only saturating at 75.8); on Waymo, LiDAR-only at 69.5@10f vs. MSeg3D at 70.2.

MSeg3D preserves accuracy with sensor failures and leverages longer temporal contexts beyond single-frame inference.

6. Implementation Details and Training Protocols

• Backbones: LiDAR U-Net (4× down/up, example $C_\mathrm{voxel} = 128$), camera backbone HRNet-w48 ($\sim$87M params), possible substitutions: SegFormer, ResNet101.
• Fusion dimensions: $C_\mathrm{int} = 64$, $C_\mathrm{gfused} = 128$, $C_\mathrm{sfused} = 256$, $K = 6$ SFFM blocks (6 heads).
• Optimization: SGD with momentum 0.9, weight decay $1\times 10^{-4}$, lr $0.01$ (decay by $10\times$ at epoch 15), batch size 32 (16×V100s, 24 epochs); AdamW alternative: lr $1\times 10^{-3}$.
• Voxel size: $d\approx 0.1\,\mathrm{m}$; image downsampling $H/H_\mathrm{in} = 1/4$, $W/W_\mathrm{in} = 1/4$.

The implementation supports large-scale scene inference and efficient batch training.

7. Connections to the Literature and Impact

MSeg3D advances multi-modal 3D segmentation, addressing three previously underexplored challenges in the literature: modality heterogeneity, limited FOV intersections, and data augmentation mismatch. By structuring a two-stage fusion (geometry-driven, then semantic-driven), and applying cross-modal completion, MSeg3D achieves robust, state-of-the-art segmentation on nuScenes, Waymo, and SemanticKITTI (Li et al., 2023). These design principles enable significant improvement over LiDAR-only and prior fusion baselines, especially critical for accurate autonomous vehicle perception of small and distant objects.

MSeg3D's strategies distinguish it from contemporaneous work such as MoonSeg3R (Du et al., 17 Dec 2025), which focuses on monocular online zero-shot segmentation, and WildSeg3D (Guo et al., 11 Mar 2025), which leverages feed-forward multi-view 2D-based 3D segmentation, with neither achieving MSeg3D's tight fusion of real-world point cloud and image data for LiDAR-camera systems.