Unsupervised Image-to-LiDAR Distillation

• Unsupervised image-to-LiDAR knowledge distillation is a framework that transfers rich semantic and geometric information from pre-trained 2D models to 3D LiDAR networks without relying on labeled data.
• It employs methods such as contrastive learning, feature/logit alignment, and self-calibrated convolution to bridge modality gaps and enhance performance in sparse data regimes.
• The approach leverages unlabeled image–LiDAR pairs and advanced visual foundation models to achieve notable gains in 3D segmentation and detection across diverse datasets.

Unsupervised image-to-LiDAR knowledge distillation refers to frameworks that transfer rich semantic or geometric information from 2D image-based vision models to 3D LiDAR-based networks, without relying on human-labeled 3D data. This process enables the learning of high-quality LiDAR representations by harnessing unlabeled or weakly labeled image–LiDAR pairs, sometimes leveraging powerful visual foundation models (VFMs), multi-modal rendering, domain adaptation, or cross-modal contrastive learning. The resulting unsupervised pre-training or adaptation pipelines can substantially improve 3D segmentation and detection performance, particularly in data-scarce or domain-shifted regimes.

1. Core Architectures and Correspondence Mechanisms

Image-to-LiDAR distillation pipelines are generally structured around two primary network branches: a 2D image-based teacher (often frozen, pre-trained, or foundation model) and a 3D point-cloud/LiDAR student network. A crucial step is to establish correspondences between 3D points (or voxels) and their projections in the 2D images. Geometric calibration between sensors enables either pixel–point associations (direct projection), clustering-based correspondences (superpixels and superpoints), or even higher-level volumetric alignments such as NeRF ray traversals.

• In NeRF-based self-supervision (Timoneda et al., 5 Nov 2024), the 3D backbone is a sparse U-Net whose features are processed both by a standard segmentation head and a volumetric NeRF MLP. The NeRF head predicts densities and semantic logits along rays projected from camera viewpoints, enabling pixel-level semantic rendering from LiDAR features.
• Several pipelines construct point-to-pixel pairs for direct contrastive or mean squared error distillation, using either raw features or class logits (Jo et al., 16 Jan 2025, Kang et al., 30 Aug 2025, Zhang et al., 18 Mar 2024).
• Cross-attention-based strategies extract higher-level "concept tokens" from the 3D student via transformers, which are further aligned with semantic tokens from 2D models such as CLIP (Yao et al., 2022).

2. Unsupervised Distillation Methodologies

Unsupervised transfer relies on losses that do not require 3D manual labels. Key approaches include:

• Contrastive Distillation: InfoNCE or related losses encourage 3D features of projected point–pixel pairs to be close, using all other batch samples as negatives. Careful pair mining and sampling strategies address spatial or class imbalance (Zhang et al., 23 May 2024, Jo et al., 16 Jan 2025).
• Feature/Logit Alignment: MSE or KL divergence losses align pooled or per-point 3D features/logits with their 2D counterparts, sometimes at superpixel/superpoint granularity or via domain adaptation modules (Kang et al., 30 Aug 2025).
• Pseudo-label Generation: 2D foundation models (e.g. SAM, DINOv2) generate masks or class labels, which are fused with 3D-rendered predictions to produce pseudo-labels for self-supervised 2D or 3D learning (Timoneda et al., 5 Nov 2024, Zhang et al., 23 May 2024).
• Cross-modal Attention: Transformers with cross-modal attention mechanisms extract and align semantically meaningful representations across modalities and domains (Yao et al., 2022).

3. Domain Adaptation and Self-Calibration

Domain adaptation is critical due to modal gaps (appearance vs structure) as well as domain shifts (different sensors, environments).

• Self-Calibrated Convolution: FSKD and UDAKD (Kang et al., 30 Aug 2025) deploy stacked 3D self-calibrated convolution blocks in the 3D backbone. These context-aware gates suppress modality-specific artifacts and promote domain-invariant feature learning.
• Backbone Freezing and Universal Representations: Methods such as SALUDA (Michele et al., 21 Nov 2025) pretrain large sparse U-Nets on multiple datasets using multimodal distillation, then freeze the backbone and only retrain a lightweight segmentation head per target scenario, enabling a "train once, adapt everywhere" paradigm.
• Spatial/Temporal Quantization and Data Utilization: Adjusting voxelization to Cartesian rather than cylindrical coordinates and mining positive pairs from unsynced sensor data—via cluster tracking and point registration—significantly improves spatial fidelity and data efficiency (Jo et al., 16 Jan 2025).

4. Detailed Loss Functions and Optimization Objectives

The training objectives are constructed to propagate supervisory signal from images to LiDAR while incorporating multi-level and multi-modal alignment:

• NeRF Self-Supervised Loss: The rendered semantics from the NeRF head are supervised by SAM pseudo-labels via 2D cross-entropy and Lovász-Softmax losses on confident segments. The full objective sums supervised and self-supervised 3D/2D losses, weighted and annealed appropriately:

$$L = \beta\,L_{3D\_vox} + \gamma\,L_{3D\_NeRF} + \lambda\,L_{2D\_NeRF}$$

• Contrastive/InfoNCE Loss: For learned 3D and 2D features $(f_k, g_k)$, the distillation is:

$$L_\mathrm{distill} = -\frac{1}{M} \sum_{k=1}^M \log \frac{\exp(\langle f_k, g_k \rangle / \tau)}{\sum_{d=1}^M \exp(\langle f_d, g_k \rangle / \tau)}$$

• Semantic Consistency with vMF: Features are regularized to cluster around their class mean on the hypersphere, pulling intra-class embeddings together (Zhang et al., 23 May 2024).
• Feature and Semantic Distillation: When pseudo-labels are available, a combined objective aligns both features and logits via MSE and KL divergence, respectively, with strong weighting on feature alignment (Kang et al., 30 Aug 2025).
• Volumetric Rendering Chains: NeRF-style chains in (Timoneda et al., 5 Nov 2024) compute accumulated pixel logits and semantics along rays, incorporating density-based volumetric weighting.

5. Use of Unlabeled Data and LiDAR-only Inference

Unsupervised pipelines maximally exploit unlabeled LiDAR scans and image–LiDAR pairs:

• Both labeled and unlabeled 3D data are processed by a shared backbone. For unlabeled samples, self-supervised NeRF rendering or contrastive/distillation losses propagate gradients.
• During inference, all image-specific components (NeRF head, ray casting, 2D models) are dropped, resulting in a LiDAR-only pipeline. The student network operates as a conventional segmentation or detection model (Timoneda et al., 5 Nov 2024, Kang et al., 30 Aug 2025).
• Some methods leverage out-of-sync sensor data and compensate spatial misalignments through positive pair mining with unsupervised registration (Jo et al., 16 Jan 2025).

6. Quantitative Benchmark Gains and Comparative Analysis

Unsupervised image-to-LiDAR distillation unlocks substantial improvements over both random initialization and previous cross-modal methods on standard benchmarks. Selected results:

Method	Dataset	Setting	Baseline (mIoU)	Distill (mIoU)	Gain
NeRF self-supervision (Timoneda et al., 5 Nov 2024)	nuScenes	1% labels	50.9	55.9	+5.0
	SemanticKITTI	1% labels	45.4	47.7	+2.3
OLIVINE (Zhang et al., 23 May 2024)	nuScenes (lin prob)	full	44.95	50.09	+5.14
	nuScenes (1%)	fine-tune	38.30	50.58	+12.28
HVDistill (Zhang et al., 18 Mar 2024)	nuScenes (1%)	fine-tune	28.3	42.7	+14.4
SALUDA (Michele et al., 21 Nov 2025)	N→K (x domain)	lin probe	44.6	52.1	+7.5
UDAKD/FSKD (Kang et al., 30 Aug 2025)	nuScenes (1%)	few-shot	38.3	44.7	+6.4
	nuScenes (0%)	zero-shot	56.8 (2DPASS)	63.7	+6.9

State-of-the-art approaches combine architectural optimizations, balanced pair mining, careful data processing, and exploitation of 2D foundation models, leading to consistent 2–12+ mIoU increases in few-shot, zero-shot, and cross-domain configurations.

7. Limitations and Considerations

Despite significant advances, certain limitations persist:

• Quality of transferred labels and features remains sensitive to the accuracy and semantic granularity of 2D foundation models and pseudo-labelers.
• Static–dynamic object distinction and sparse regions (e.g., sky or heavily occluded areas) may degrade pseudo-label quality, requiring further geometric reasoning or motion modeling (Najibi et al., 2023).
• Some approaches require highly synchronized sensor data and precise calibration; errors in these steps propagate through the distillation pipeline.
• Domain shifts (e.g., changes in LiDAR sensor, weather, or urban/rural environments) can still impair adaptation unless robust domain-agnostic backbones are used and frozen appropriately (Michele et al., 21 Nov 2025).

A plausible implication is that future research will emphasize even richer cross-modal representations (e.g., with language or radar supervision), improved geometric alignment and reasoning, and more flexible handling of sensory misalignment and missing data. Unsupervised image-to-LiDAR distillation remains a highly active research area, with ongoing developments in foundation model utilization, domain transfer resilience, and efficient multi-modal training.