Papers
Topics
Authors
Recent
Search
2000 character limit reached

Depth-Aware Distillation (DAD) Methods

Updated 29 September 2026
  • Depth-Aware Distillation (DAD) is a family of teacher-student learning techniques that use depth, geometry, or spatial structure as a key component for knowledge transfer, enhancing the performance of student models without needing the privileged modalities at inference.
  • DAD methods exploit asymmetries between teacher and student such as privileged input, capacity, and modality to improve the student's ability to understand and generate depth-related features in tasks like 3D detection and depth estimation.
  • Examples of DAD include ADD, CADiT, and GeoDistill, which demonstrate selective geometric transfer, technology used to improve student predictions for depth completion, stereo-assisted 3D lane and visual place recognition, and more by minimizing errors via photometric agreement, epistemic uncertainty, and adaptation.

Depth-Aware Distillation (DAD) is a family of teacher–student learning methods in which depth, geometry, spatial structure, or a depth-derived reliability signal constitutes a central component of knowledge transfer. DAD commonly uses privileged training-time information—such as ground-truth depth, stereo imagery, LiDAR, radar, multiview observations, monocular-depth predictions, or a geometric foundation model—to improve a student that receives fewer modalities or has lower capacity at inference. The teacher and student may exchange depth predictions, intermediate geometric features, spatial relations, attention patterns, depth distributions, or geometry-conditioned descriptors. DAD is therefore a methodological category rather than a single canonical architecture or loss.

1. Scope and conceptual foundations

Monocular and multimodal perception are underconstrained because image appearance does not uniquely determine metric range, surface geometry, occlusion structure, or spatial scale. A depth-aware teacher can provide information unavailable to the deployment-time student, while distillation constrains the student to absorb useful geometric regularities without requiring the privileged modality during inference.

The defining property of DAD is not merely that a model predicts depth. A method is depth-aware when depth or geometry materially determines the distillation target, feature transformation, reliability weighting, spatial correspondence, or auxiliary structural objective. This distinguishes DAD from generic feature distillation applied to a depth-estimation task. For example, Channel-Aware Distillation Transformer (CADiT) transfers depth-estimation representations through channel correlations, but its explicit operation is a cross-network channel-reweighting mechanism rather than a geometric constraint such as surface-normal, epipolar, or metric-scale consistency (Zhang et al., 2023). Conversely, Attention-Based Depth Distillation (ADD) uses accurate depth to construct 3D-aware positional encodings that govern feature adaptation before teacher–student matching (Wu et al., 2022).

DAD methods generally exploit one or more of the following asymmetries:

  • Privileged input: the teacher receives ground-truth depth, stereo pairs, LiDAR, radar, or adjacent views unavailable to the student.
  • Capacity asymmetry: a large teacher transfers knowledge to a lightweight student.
  • Representation asymmetry: a geometric teacher transfers depth-aware features into an RGB, thermal, segmentation, lane-detection, or place-recognition model.
  • Reliability asymmetry: teacher information is selectively transferred according to photometric agreement, uncertainty, modality compatibility, or depth-distribution structure.
  • Inference asymmetry: teacher networks, depth sensors, distillation branches, and auxiliary supervision are removed after training.

DAD should not be conflated with methods whose names merely contain “depth-aware.” PQDAST, for example, uses a Depth Anything consistency loss to preserve the scene layout of stylized images, but depth is an auxiliary RGB-domain regularizer rather than an explicitly depth-conditioned teacher–student representation (Ioannou et al., 24 Feb 2025). Similarly, Depth3DLane uses depth-prior feature distillation and a hierarchical depth-aware head, but the paper does not provide a complete formal distillation loss or explicit depth target for its auxiliary head (Lyu et al., 25 Apr 2025).

2. Teacher–student configurations and privileged information

The teacher design determines the type of geometry transferred and the alignment burden imposed on training.

ADD uses identical monocular detector architectures for teacher and student. The teacher receives the image plus accurate ground-truth depth, while the student retains the baseline monocular inputs, including estimated or learned depth representations. The teacher is trained independently, frozen, and then used to supervise the student through intermediate features and decoder responses. MonoDETR uses object-wise depth ground truth for the teacher, whereas CaDDN and PatchNet use pixel-wise depth ground truth (Wu et al., 2022). The identical architecture provides corresponding image locations, feature scales, decoder layers, channels, and query semantics, avoiding the modality and representation gaps associated with stereo- or LiDAR-based teachers.

Other configurations deliberately use different modalities. ADU-Depth trains a teacher on concatenated left–right RGB images and dense ground-truth depth, while its student receives a single RGB image. Both use a NeWCRFs-style encoder–decoder with a Swin Transformer encoder and neural-window fully connected CRF-style decoder, but the teacher accepts six input channels and the student three. The student uses attention adaptation, uncertainty modeling, and focal-depth response distillation to approximate stereo-informed representations (Wu et al., 2023).

MonoTher-Depth transfers an RGB foundation model into a thermal student. Both networks use the metric version of Depth Anything, consisting of a DINOv2 feature extractor, DPT decoder, and metric-bins module. The RGB teacher is used only during training; the thermal student receives a single thermal image during deployment. RGB depth is geometrically reprojected into the thermal camera, and a confidence network determines which RGB predictions should supervise the thermal model (Zuo et al., 21 Apr 2025).

LiRCDepth uses CaFNet, a 62.25-million-parameter radar–camera teacher, to supervise a 12.65-million-parameter MobileNetV2 student. The student receives RGB and radar inputs at inference but uses a much lighter encoder, fusion mechanism, and decoder. Distillation covers camera features, radar features, pairwise decoder relationships, and intermediate depth maps (Ma et al., 2024). XD-RCDepth further compresses the student to 8.89 million parameters and uses FiLM-based radar-conditioned fusion. Its distillation includes Grad-CAM saliency transfer and depth-distribution distillation (Sun et al., 15 Oct 2025).

A different configuration appears in GeoLaV, where a frozen 3D-aware teacher such as π3\pi^3 supplies geometry-rich features to a text-driven video segmentation model. The teacher does not directly supervise scalar depth; instead, the student memory representation is aligned with features encoding depth relationships, viewpoint consistency, correspondence, and scene structure (Zhu et al., 23 Jun 2026). In geometry-conditioned visual place recognition, Depth Anything V2 acts as a geometric foundation model, while DINOv2 and SALAD provide the appearance representation. Per-image normalized depth is embedded into the visual token space and selectively added to appearance tokens (Nedov et al., 23 Sep 2026).

3. Distillation targets and mathematical mechanisms

DAD distillation targets range from dense scalar depth to high-level descriptors. The principal mechanisms can be organized as follows.

Feature and response distillation

ADD distills both multi-level convolutional features and decoder-query responses. For feature level kk, the student feature FkF^k is transformed into a depth-aware representation F3dkF^k_{3d} and matched to the teacher feature TkT^k. Foreground and background regions receive different weights:

Lfeat=∑k[αI∥Mk(F3dk−Tk)∥22+βI∥(1−Mk)(F3dk−Tk)∥22].\mathcal L_{\mathrm{feat}} = \sum_k \left[ \alpha_I \left\| M^k(F^k_{3d}-T^k) \right\|_2^2 + \beta_I \left\| (1-M^k)(F^k_{3d}-T^k) \right\|_2^2 \right].

The reported values are αI=1.0\alpha_I=1.0 and βI=0.1\beta_I=0.1. For transformer decoder queries, Hungarian matching produces a foreground-query mask MfM_f, and target-aware cross-attention retrieves teacher query information. The response loss uses αv=1.0\alpha_v=1.0 and kk0 for foreground and background queries, respectively (Wu et al., 2022).

CADiT instead forms a channel-correlation map between flattened student and teacher features:

kk1

then reconfigures student features by

kk2

and minimizes

kk3

The method also applies multi-scale kk4 matching to teacher and student depth outputs. Its total objective combines self-supervised photometric reconstruction, edge-aware smoothness, CADiT feature transfer, and output depth distillation with kk5 (Zhang et al., 2023).

Depth-level and reprojection-aware distillation

MAL constructs a depth pseudo-target by comparing teacher- and student-derived reprojections. At each pixel, the depth associated with the lower image-reprojection error is selected:

kk6

The student is then trained toward the selected target through an kk7 depth loss. This is prediction-level DAD: it is spatially adaptive, reprojection-aware, and not unconditional teacher imitation. MAL also introduces temporal hints that associate moving instances across three adjacent frames using Hungarian matching and approximately linear motion, producing motion-rectified reconstruction candidates (Dong et al., 2024).

Stereo-assisted monitored distillation for depth completion uses an ensemble of completion teachers—PENet, MSG_CHN, ENet, and NLSPN—and a stereo teacher, PSMNet. Candidate teacher depths are selected pixelwise according to photometric reconstruction error. The resulting adaptive target avoids transferring systematic completion-teacher errors when another candidate, including stereo depth, better reconstructs the target image (Guo et al., 2023).

Uncertainty-aware distillation

ADU-Depth predicts per-pixel log variance kk8. Feature residuals are weighted using a heteroscedastic Gaussian negative-log-likelihood:

kk9

whereas response residuals use a Laplace-based form:

FkF^k0

The uncertainty map reduces the influence of regions where monocular observations cannot reliably reproduce stereo knowledge, such as distant background, boundaries, occlusions, and weakly observed areas. Focal-depth response distillation additionally emphasizes cases with poor teacher–student depth agreement (Wu et al., 2023).

MonoTher-Depth learns a confidence map for RGB depth using RGB–thermal feature similarity, warped depth discrepancy, RGB depth, thermal depth, and RGB appearance. The confidence-weighted consistency loss is

FkF^k1

The method removes the top FkF^k2 of residuals and retains pixels in the top FkF^k3 of feature similarity. Its ablation shows that unweighted distillation can be worse than no distillation, supporting selective teacher supervision (Zuo et al., 21 Apr 2025).

Distributional depth distillation

XD-RCDepth converts scalar teacher and student depths into soft distributions over fixed depth-bin centers. With logits defined by negative distance to bin centers,

FkF^k4

the temperature-scaled distributions are

FkF^k5

Depth-distribution distillation minimizes the forward KL divergence:

FkF^k6

The paper does not report the numerical temperature, bin count, or bin spacing. The distribution is constructed deterministically from scalar depth predictions; it is not produced by a separate uncertainty head (Sun et al., 15 Oct 2025).

4. Geometry-conditioned feature adaptation

A central design question is whether teacher and student tensors should be matched directly. Many DAD methods instead use geometry to transform the student before matching.

ADD derives a 3D-aware positional embedding from ground-truth depth. A pointwise MLP implemented with FkF^k7 convolutions transforms the depth representation, and a channel-wise argmax selects a discretized depth-position index. A learned embedding dictionary converts that index into positional features, which are added to flattened student features before self-attention. Depth therefore enters the attention queries and keys as a geometric positional signal rather than merely as a regression target (Wu et al., 2022).

ADU-Depth uses self-attention adaptation on student encoder and decoder features before squared matching with stereo-teacher features. The adaptation is intended to capture long-range dependencies and reduce the stereo–monocular domain gap (Wu et al., 2023).

Depth3DLane transfers Depth Anything V2 features from layers associated with FkF^k8 and FkF^k9 into StudentS32 and StudentS64 modules. The resulting depth-aware features are concatenated with Spatial Transformation Pyramid features before BEV lane prediction. Its Hierarchical Depth-Aware Head separately learns multiscale depth features through an encoder–decoder, while a CRF uses appearance and teacher-derived depth cues to encourage lane continuity (Lyu et al., 25 Apr 2025).

GeoDistill uses pooled bidirectional attention between appearance and geometry branches. Appearance queries attend to pooled geometry keys and values, and geometry queries attend to pooled appearance keys and values. A pixelwise reliability estimator then controls geometry injection:

F3dkF^k_{3d}0

The teacher is removed after training; the deployed model infers a task-oriented relative geometry representation from RGB alone (Wang et al., 3 Sep 2026).

In visual place recognition, DAD embeds normalized depth into the same token space as DINOv2 appearance features. The conditioned tokens are

F3dkF^k_{3d}1

where F3dkF^k_{3d}2 is a trainable channel-wise gate initialized to F3dkF^k_{3d}3. A two-stage objective first aligns the geometry-conditioned descriptor with the frozen appearance teacher and subsequently refines the representation using retrieval loss (Nedov et al., 23 Sep 2026).

5. Objectives, supervision, and training schedules

DAD objectives usually combine baseline task supervision with one or more distillation terms. ADD uses

F3dkF^k_{3d}4

with F3dkF^k_{3d}5. The response branch is disabled when the detector lacks an encoder–decoder transformer head, and F3dkF^k_{3d}6 when the baseline uses a pretrained depth estimator rather than jointly learning depth (Wu et al., 2022).

DDND combines self-supervised photometric and smoothness losses with CADiT and multiscale output distillation:

F3dkF^k_{3d}7

where F3dkF^k_{3d}8. The system uses view synthesis, minimum photometric reprojection, auto-masking, and edge-aware inverse-depth smoothness, but no ground-truth depth term is reported (Zhang et al., 2023).

LiRCDepth combines uncertainty-rectified depth supervision with four distillation terms:

F3dkF^k_{3d}9

The camera-feature, radar-feature, decoder-structure, and inter-depth terms transfer distinct forms of teacher information. The student uses accumulated and single-frame LiDAR-derived supervision, with uncertainty rectification to reduce the influence of registration errors in accumulated depth (Ma et al., 2024).

GeoLaV uses segmentation and cosine feature-similarity losses. During monocular geometry pretraining, pixels are lifted into 3D using predicted depth or point maps, transformed through virtual camera poses, reprojected, and inpainted to produce synthetic videos. During real-video fine-tuning, the student memory features are aligned with frozen 3D and visual foundation model features. No direct scalar depth-regression loss is defined (Zhu et al., 23 Jun 2026).

The training-only nature of DAD is a recurrent design pattern. At inference, ADD removes the teacher, ground-truth depth, distillation branches, and ground-truth-derived positional encodings. ADU-Depth removes the stereo teacher. MonoTher-Depth retains only the thermal student. MAL removes instance segmentation, teacher execution, motion rectification, and distillation-target construction. GeoDistill removes Depth Anything V2 and all sensor depth. This separation permits privileged supervision without imposing the teacher’s computational or sensor requirements on deployment.

6. Empirical evidence, limitations, and methodological distinctions

DAD has been evaluated across 3D detection, depth estimation, depth completion, thermal perception, radar–camera fusion, lane detection, video segmentation, salient-object detection, and visual place recognition.

In monocular 3D detection on KITTI, MonoDETR + ADD improves validation TkT^k0 from TkT^k1 to TkT^k2 for Easy, Moderate, and Hard difficulty. CaDDN and PatchNet also improve, and the MonoDETR-based system reports test TkT^k3 of TkT^k4 with no additional inference computational cost (Wu et al., 2022).

For nano-drone depth estimation, DDND reduces Abs Rel from TkT^k5 without KD to TkT^k6 with KD, while retaining a 310K-parameter student and achieving 1.24 FPS on GAP8 at TkT^k7 resolution. The complete system improves Sq Rel, RMSE, RMSE log, and threshold accuracy, although it remains less accurate than the 3.1-million-parameter teacher (Zhang et al., 2023).

ADU-Depth reports KITTI Eigen-split values of Abs Rel TkT^k8, RMSE TkT^k9, and Lfeat=∑k[αI∥Mk(F3dk−Tk)∥22+βI∥(1−Mk)(F3dk−Tk)∥22].\mathcal L_{\mathrm{feat}} = \sum_k \left[ \alpha_I \left\| M^k(F^k_{3d}-T^k) \right\|_2^2 + \beta_I \left\| (1-M^k)(F^k_{3d}-T^k) \right\|_2^2 \right].0. On DrivingStereo, it reports Abs Rel Lfeat=∑k[αI∥Mk(F3dk−Tk)∥22+βI∥(1−Mk)(F3dk−Tk)∥22].\mathcal L_{\mathrm{feat}} = \sum_k \left[ \alpha_I \left\| M^k(F^k_{3d}-T^k) \right\|_2^2 + \beta_I \left\| (1-M^k)(F^k_{3d}-T^k) \right\|_2^2 \right].1 and RMSE Lfeat=∑k[αI∥Mk(F3dk−Tk)∥22+βI∥(1−Mk)(F3dk−Tk)∥22].\mathcal L_{\mathrm{feat}} = \sum_k \left[ \alpha_I \left\| M^k(F^k_{3d}-T^k) \right\|_2^2 + \beta_I \left\| (1-M^k)(F^k_{3d}-T^k) \right\|_2^2 \right].2. Its ablations show that removing focal-depth distillation, uncertainty modeling, or attention adaptation degrades performance (Wu et al., 2023).

In depth completion, the stereo-assisted monitored-distillation method achieves KITTI MAE Lfeat=∑k[αI∥Mk(F3dk−Tk)∥22+βI∥(1−Mk)(F3dk−Tk)∥22].\mathcal L_{\mathrm{feat}} = \sum_k \left[ \alpha_I \left\| M^k(F^k_{3d}-T^k) \right\|_2^2 + \beta_I \left\| (1-M^k)(F^k_{3d}-T^k) \right\|_2^2 \right].3, RMSE Lfeat=∑k[αI∥Mk(F3dk−Tk)∥22+βI∥(1−Mk)(F3dk−Tk)∥22].\mathcal L_{\mathrm{feat}} = \sum_k \left[ \alpha_I \left\| M^k(F^k_{3d}-T^k) \right\|_2^2 + \beta_I \left\| (1-M^k)(F^k_{3d}-T^k) \right\|_2^2 \right].4, iMAE Lfeat=∑k[αI∥Mk(F3dk−Tk)∥22+βI∥(1−Mk)(F3dk−Tk)∥22].\mathcal L_{\mathrm{feat}} = \sum_k \left[ \alpha_I \left\| M^k(F^k_{3d}-T^k) \right\|_2^2 + \beta_I \left\| (1-M^k)(F^k_{3d}-T^k) \right\|_2^2 \right].5, and iRMSE Lfeat=∑k[αI∥Mk(F3dk−Tk)∥22+βI∥(1−Mk)(F3dk−Tk)∥22].\mathcal L_{\mathrm{feat}} = \sum_k \left[ \alpha_I \left\| M^k(F^k_{3d}-T^k) \right\|_2^2 + \beta_I \left\| (1-M^k)(F^k_{3d}-T^k) \right\|_2^2 \right].6, compared with baseline MAE Lfeat=∑k[αI∥Mk(F3dk−Tk)∥22+βI∥(1−Mk)(F3dk−Tk)∥22].\mathcal L_{\mathrm{feat}} = \sum_k \left[ \alpha_I \left\| M^k(F^k_{3d}-T^k) \right\|_2^2 + \beta_I \left\| (1-M^k)(F^k_{3d}-T^k) \right\|_2^2 \right].7 and RMSE Lfeat=∑k[αI∥Mk(F3dk−Tk)∥22+βI∥(1−Mk)(F3dk−Tk)∥22].\mathcal L_{\mathrm{feat}} = \sum_k \left[ \alpha_I \left\| M^k(F^k_{3d}-T^k) \right\|_2^2 + \beta_I \left\| (1-M^k)(F^k_{3d}-T^k) \right\|_2^2 \right].8. The method is trained without dense ground-truth depth and uses stereo supervision only during training (Guo et al., 2023).

For thermal depth estimation, MonoTher-Depth achieves an approximately Lfeat=∑k[αI∥Mk(F3dk−Tk)∥22+βI∥(1−Mk)(F3dk−Tk)∥22].\mathcal L_{\mathrm{feat}} = \sum_k \left[ \alpha_I \left\| M^k(F^k_{3d}-T^k) \right\|_2^2 + \beta_I \left\| (1-M^k)(F^k_{3d}-T^k) \right\|_2^2 \right].9 reduction in Abs Rel during unlabeled ViViD++ adaptation, from αI=1.0\alpha_I=1.00 to αI=1.0\alpha_I=1.01. Its ablation shows that removing confidence modeling produces worse performance than no distillation, demonstrating that teacher reliability is consequential under cross-modal domain shift (Zuo et al., 21 Apr 2025).

In radar–camera depth estimation, LiRCDepth reduces MAE from αI=1.0\alpha_I=1.02 without KD to αI=1.0\alpha_I=1.03 with KD at the 50 m nuScenes evaluation range, while reducing parameters from 62.25M for CaFNet to 12.65M. XD-RCDepth reduces the student to 8.89M parameters and reports MAE reductions of 7.91%, 7.96%, and 7.97% at 50 m, 70 m, and 80 m, respectively, relative to its non-distilled student (Ma et al., 2024, Sun et al., 15 Oct 2025).

Depth3DLane reports OpenLane F1 of 61.5%, close-range αI=1.0\alpha_I=1.04-error of 0.145 m, far-range αI=1.0\alpha_I=1.05-error of 0.133 m, and 88 FPS. The αI=1.0\alpha_I=1.06 distillation combination obtains the best general balance in its scale ablation, while the paper does not provide the explicit feature-distance loss, feature dimensions, or distillation weights (Lyu et al., 25 Apr 2025).

In video segmentation, GeoLaV improves Ref-Youtube-VOS from 65.1 to 70.5 in the reported αI=1.0\alpha_I=1.07 ablation when both monocular geometry pretraining and geometry-aware distillation are used. Its zero-shot performance reaches 47.0 on Ref-Youtube-VOS and 31.6 on MeViS in the supplied comparison (Zhu et al., 23 Jun 2026).

For RGB-D salient-object detection, GeoDistill ignores dataset depth and uses Depth Anything V2 only during training. It achieves the best or tied-best result in 26 of 36 metric–dataset comparisons and reduces ReDWeb-S relative MAE by 13.4%. The method’s depth branch is relative and task-oriented rather than metric (Wang et al., 3 Sep 2026).

In forest visual place recognition, the earlier depth-aware descriptor method improves WildCross inter-sequence Recall@1 from 49.38% to 62.69% relative to its frozen appearance baseline, while the geometry-conditioned VPR formulation improves a matched SALAD baseline from 61.41% to 66.37% and Recall@5 from 65.86% to 72.49% (Nedov et al., 11 Jun 2026, Nedov et al., 23 Sep 2026).

Despite these results, several limitations recur. Teacher errors can be propagated when reliability is not modeled or when the teacher is systematically biased. Monocular depth remains scale-ambiguous and may hallucinate geometry. Direct scalar depth matching can be sensitive to representation conventions, such as depth versus inverse depth, and may lack scale alignment. Cross-modal methods require calibration, overlap, or modality compatibility during training. Photometric selection can fail in textureless, reflective, transparent, or repeatedly textured regions. Many papers omit exact tensor dimensions, attention-head counts, normalization layers, temperature values, loss weights, teacher-update mechanisms, or mask definitions. Several methods report no FLOP, memory, energy, or complete deployment analysis.

DAD also encompasses materially different objectives. It includes explicit privileged-depth transfer, stereo-to-monocular adaptation, confidence-weighted RGB-to-thermal transfer, reprojection-based pseudo-label selection, distributional depth matching, geometry-conditioned token modulation, and implicit 3D feature alignment. These mechanisms are related through their use of geometric information in teacher supervision, but they should not be treated as interchangeable. A method that merely adds a depth consistency regularizer, such as PQDAST, is narrower than a system that explicitly distills depth-conditioned features, responses, distributions, or geometric relations.

The general methodological principle is therefore selective geometric transfer: use a privileged or stronger source of depth and structure during training, transform the student representation before matching when modalities differ, weight supervision according to reliability, and remove the privileged branches at inference. DAD is strongest when depth functions not only as an auxiliary prediction target, but as a mechanism that determines what information is transferred, where it is transferred, and how strongly the student should trust it.

Topic to Video (Beta)

No one has generated a video about this topic yet.

Whiteboard

No one has generated a whiteboard explanation for this topic yet.

Follow Topic

Get notified by email when new papers are published related to Depth-Aware Distillation (DAD).