Depth Distillation Techniques

• Depth Distillation is a technique that transfers depth-related representations from a rich teacher model to a compact student model, enhancing geometric perception.
• It leverages feature-level, output-level, and temporal consistency losses to robustly train student models for tasks like monocular depth estimation and multi-modal fusion.
• Empirical results demonstrate significant parameter reduction and improved accuracy in complex applications such as radar-camera integration, thermal imaging, and event-based sensing.

Depth Distillation is a specialized branch of knowledge distillation focusing on transferring depth-related representations, priors, or predictions from a teacher model—often larger, more accurate, or multimodal—into a more compact, efficient, or otherwise constrained student model. This strategy is central to advancing monocular depth estimation, multi-modal fusion (e.g., radar, event, or thermal cameras), dense completion, and related geometric perception tasks. Contemporary depth distillation techniques address both computational efficiency and the injection of geometric priors by leveraging innovative loss formulations, cross-modal supervision, and the transfer of explainability or uncertainty information.

1. Distillation Principles and Motivation

The core objective of depth distillation is to enhance the performance of a student model on depth estimation tasks using privileged information available to a teacher during training. This privileged information may take the form of more informative sensor modalities (e.g., stereo vision, LiDAR, radar, RGB, events), semantic structure (e.g., segmentation, keypoints), or specialized architectures with extensive supervision or capacity. The motivation derives from several pressing factors:

• Data Scarcity and Cost: Collecting dense metric ground-truth for depth remains considerably expensive or impossible for certain modalities, such as event cameras (Ren et al., 10 Mar 2026), transparent surfaces (Huang et al., 2024), and thermal imagery (Zuo et al., 21 Apr 2025).
• Model Compression and Efficiency: State-of-the-art models, such as those used in autonomous driving, often exceed the computational resources of edge devices. Distillation enables substantial parameter reduction while retaining a large fraction of the teacher’s accuracy (Sun et al., 15 Oct 2025).
• Cross-Modal and Cross-Task Transfer: Depth is a geometric cue useful across tasks, impacting not only estimation itself but also associated semantic segmentation, keypoint detection, and place recognition (Anand et al., 2024, Nedov et al., 11 Jun 2026).
• Robustness Under Domain Shifts: The distillation framework can address domain gaps, simulate out-of-distribution data, and adapt modality-specific priors from robust RGB or multi-sensor teachers (Hu et al., 2022, He et al., 26 Feb 2025).

2. Loss Functions and Distillation Strategies

Various loss function designs and alignment principles have emerged in depth distillation, each targeting a distinct operational facet of depth-related representation.

2.1 Feature-Level Alignment

• Saliency/Explainability Transfer: Grad-CAM-based alignment between student and teacher saliency maps captures the spatial focus of the depth estimator, enforcing explainability-aligned transfer (Sun et al., 15 Oct 2025). The cosine similarity between ℓ₂-normalized activations contextualizes where the network "looks" for depth cues.
• Self-Attention and Attention-Based Distillation: Spatial and channel attention modules propagate geometric dependencies or 3D-aware positional information (Wu et al., 2022, Wu et al., 2023).
• Cross-Modal Matching: Channels or patch-level features are matched via attention or correlation modules to bridge representational discrepancies between teacher and student networks (Huang et al., 2024, Sun et al., 15 Oct 2025, Zhang et al., 2023).

2.2 Output/Prediction-Level Distillation

• Soft Classification and Depth Distribution: Regression is recast as soft classification over discretized depth bins, transferring the full distributional structure (not just mean estimates) through Kullback–Leibler divergence on predicted depth probabilities (Sun et al., 15 Oct 2025).
• Scale-Invariant or Affine-Invariant Losses: To mitigate errors from scale ambiguity in monocular depth or cross-domain transfer, losses are often made invariant to global scale/shift transformations (Liang et al., 21 Mar 2025, He et al., 26 Feb 2025). Recent work analyzes the noise amplification induced by global normalization and proposes hybrid or context-specific alternatives (He et al., 26 Feb 2025).
• Uncertainty-Aware or Self-Adaptive Losses: Pixel-level uncertainty, estimated or propagated during training, weighs distillation terms more heavily on hard or reliable regions (Sun et al., 2024, Wu et al., 2023, Zuo et al., 21 Apr 2025).
• Cross-Task Distillation: Knowledge conversion across tasks is achieved by architectural translators (e.g., depth→segmentation), enabling semantic priors to regularize depth estimation (Cai et al., 2021).

2.3 Temporal and Geometric Consistency

• Temporal and Multi-Frame Alignment: For sequences or event streams, explicit losses enforce that depth changes over time in the student mirror those in the teacher (Ren et al., 10 Mar 2026, Bartolomei et al., 18 Sep 2025).
• Multi-View and Stereo Supervision: Reprojection, multi-view photo-consistency, and stereo guidance use geometric constraints to improve student robustness and avoid teacher artifacts (Liu et al., 2022, Guo et al., 2023).
• Monitored/Selective Distillation: Adaptive, per-pixel selection (i.e., positive congruent learning) ensures the student does not inherit teacher error modes—a confidence or “monitor” gates distillation only when the teacher is reliable (Liu et al., 2022, Guo et al., 2023).

3. Architectural Approaches and Modalities

Depth distillation spans a diversity of architectures and sensor/processing contexts, each motivating unique design choices:

• RGB/Camera-Only Distillation: Off-the-shelf monocular foundation models or stereo teachers drive monocular students, incorporating attention-based or shift-invariant priors (Wu et al., 2023, He et al., 26 Feb 2025).
• Radar-Camera and Multi-Modal Fusion: Fusion architectures leverage complementary geometric cues from radar and cameras, using confidence-aware gated fusion and feature modulation layers to enable real-time operation in adverse weather (Sun et al., 2024, Sun et al., 15 Oct 2025).
• Event Cameras: Cross-modal distillation from robust RGB-based teachers (vision foundation models—VFMs) to event-based depth estimators integrates spatio-temporal, output, and distributional alignment losses (Ren et al., 10 Mar 2026, Bartolomei et al., 18 Sep 2025).
• Thermal Cameras: Cross-modal pipelines utilize confidence-aware per-pixel weighting to compensate for domain gaps and predictor uncertainty, fine-tuning in both supervised and self-supervised regimes (Zuo et al., 21 Apr 2025).
• Depth Completion and Transparent Surfaces: Distillation bridges the gap between sparse/missing measurements (LiDAR holes, transparent objects) using ensemble, CRF, and attention modules fused with teacher guidance (Huang et al., 2024, Liang et al., 21 Mar 2025).
• Dynamic Network Depth: Depth-level dynamic neural networks train multiple sub-nets of varying depth by embedding knowledge distillation at split points (Zhao et al., 2021).

4. Training Protocols and Optimization

Depth distillation frameworks typically share several optimization and procedural elements:

• End-to-End or Multi-Stage: Some approaches pretrain on synthetic/complementary data (e.g., simulated LiDAR via teacher depth) and subsequently fine-tune with real-world supervision or scale-invariant alignment (Liang et al., 21 Mar 2025, Hu et al., 2022).
• Teacher Freezing and Forward Hooks: The teacher model is generally frozen; feature-level or attention alignment proceeds via forward hooks or gradient detachment to avoid updating teacher parameters (Sun et al., 15 Oct 2025).
• Composite and Modular Losses: Final objectives combine supervised, distillation, attention/feature, uncertainty, and task-alignment losses with schedule- or validation-tuned weighting (Sun et al., 2024, Zuo et al., 21 Apr 2025, Wu et al., 2023).
• Batch Sampling and Curriculum: Hardness-aware sampling, curriculum over class groupings (e.g., for cross-task distillation), and hybrid multi-teacher scheduling are applied to maximize supervision diversity (Cai et al., 2021, He et al., 26 Feb 2025).

5. Empirical Results and Real-World Impact

Depth distillation has demonstrated substantial practical impact across a range of datasets and tasks:

• Radar-Camera Depth: XD-RCDepth achieves a 29.7% reduction in parameters relative to baseline with only minor accuracy loss, further closing the gap to a heavy teacher when both explainability and distribution alignment are used (MAE on nuScenes reduced by 7.91%) (Sun et al., 15 Oct 2025).
• Thermal Monocular Depth: Confidence-aware distillation reduces AbsRel by up to 22.88% (ViViD++), and enables self-supervised transfer to novel domains without additional annotation (Zuo et al., 21 Apr 2025).
• Event-Based Depth: Cross-modal and tri-level (output/feature/temporal) distillation achieves >50% reduction in mean error versus previous SOTA on EventScape, while yielding superior temporal consistency (Ren et al., 10 Mar 2026).
• Monocular Completion: Synthetic LiDAR pretraining plus scale- and shift-invariant fine-tuning places student models at 1st on KITTI, outperforming both purely supervised and vanilla monocular approaches (Liang et al., 21 Mar 2025).
• Image-Only Estimation: Multi-teacher, cross-context, and normalization-free distillation achieves new SOTA on NYUv2, KITTI, DIODE, ETH3D, consistently outperforming both individual teacher networks and prior ensemble methods (He et al., 26 Feb 2025).
• Place Recognition and Keypoint Detection: Depth-aware distillation into pre-trained recognition backbones or keypoint detectors noticeably improves robustness to environmental variation and background noise (Anand et al., 2024, Nedov et al., 11 Jun 2026).

6. Limitations, Challenges, and Future Directions

• Teacher Reliability and Domain Bias: The upper limit of distillation is dictated by teacher reliability; failure modes in domains where the teacher is uncertain (e.g., out-of-distribution, complex transparent surfaces, severe weather) propagate to the student (Zuo et al., 21 Apr 2025, Ren et al., 10 Mar 2026).
• Normalization-Induced Noise: Global depth normalization (SSI) can unintentionally amplify pseudo-label noise, motivating local or hybrid normalization schemes and careful tuning (He et al., 26 Feb 2025).
• Capacity and Modality Gap: Large gaps in representational capacity or modality (e.g., RGB-to-event or RGB-to-thermal) require carefully designed transformation networks, confidence estimation, or feature-adaptation layers (Bartolomei et al., 18 Sep 2025, Zuo et al., 21 Apr 2025).
• Scalability with Large Unlabeled Data: Data-free distillation via OOD simulation offers a path to adapt models without domain-labeled data, but is limited by simulation fidelity and the complexity of domain adaptation transformations (Hu et al., 2022).
• Temporal and Multi-Teacher Generalization: There remains substantial opportunity to further exploit temporal priors, self-supervised video sequences, and multi-teacher or ensemble frameworks for robust generalization (Ren et al., 10 Mar 2026, He et al., 26 Feb 2025).

Depth distillation is thus a mature and rapidly evolving framework, providing a principled approach for injecting geometric priors, distributing semantic or temporal structure, and achieving resource-efficient deployment of depth-perception models across a diversity of real-world settings. Future advances are likely to further integrate self-supervised, temporally coherent, and multi-modal signals while mitigating propagation of teacher bias and adapting to challenging domain shifts.