---
title: Depth-Aware Distillation (DAD) Methods
url: https://www.emergentmind.com/topics/depth-aware-distillation-dad
type: topic
---

# Depth-Aware Distillation (DAD) Methods

Depth-Aware Distillation (DAD) is a family of teacher–student learning methods in which depth, geometry, spatial structure, or a depth-derived reliability signal constitutes a central component of knowledge transfer. DAD commonly uses privileged training-time information—such as ground-truth depth, stereo imagery, LiDAR, radar, multiview observations, monocular-depth predictions, or a geometric foundation model—to improve a student that receives fewer modalities or has lower capacity at inference. The teacher and student may exchange depth predictions, intermediate geometric features, spatial relations, attention patterns, depth distributions, or geometry-conditioned descriptors. DAD is therefore a methodological category rather than a single canonical architecture or loss.

## 1. Scope and conceptual foundations

Monocular and multimodal perception are underconstrained because image appearance does not uniquely determine metric range, surface geometry, occlusion structure, or spatial scale. A depth-aware teacher can provide information unavailable to the deployment-time student, while distillation constrains the student to absorb useful geometric regularities without requiring the privileged modality during inference.

The defining property of DAD is not merely that a model predicts depth. A method is depth-aware when depth or geometry materially determines the distillation target, feature transformation, reliability weighting, spatial correspondence, or auxiliary structural objective. This distinguishes DAD from generic feature distillation applied to a depth-estimation task. For example, Channel-Aware Distillation Transformer (CADiT) transfers depth-estimation representations through channel correlations, but its explicit operation is a cross-network channel-reweighting mechanism rather than a geometric constraint such as surface-normal, epipolar, or metric-scale consistency [2303.10386]. Conversely, Attention-Based Depth Distillation (ADD) uses accurate depth to construct 3D-aware positional encodings that govern feature adaptation before teacher–student matching [2211.16779].

DAD methods generally exploit one or more of the following asymmetries:

- **Privileged input**: the teacher receives ground-truth depth, stereo pairs, LiDAR, radar, or adjacent views unavailable to the student.
- **Capacity asymmetry**: a large teacher transfers knowledge to a lightweight student.
- **Representation asymmetry**: a geometric teacher transfers depth-aware features into an RGB, thermal, segmentation, lane-detection, or place-recognition model.
- **Reliability asymmetry**: teacher information is selectively transferred according to photometric agreement, uncertainty, modality compatibility, or depth-distribution structure.
- **Inference asymmetry**: teacher networks, depth sensors, distillation branches, and auxiliary supervision are removed after training.

DAD should not be conflated with methods whose names merely contain “depth-aware.” PQDAST, for example, uses a Depth Anything consistency loss to preserve the scene layout of stylized images, but depth is an auxiliary RGB-domain regularizer rather than an explicitly depth-conditioned teacher–student representation [2502.16996]. Similarly, Depth3DLane uses depth-prior feature distillation and a hierarchical depth-aware head, but the paper does not provide a complete formal distillation loss or explicit depth target for its auxiliary head [2504.18325].

## 2. Teacher–student configurations and privileged information

The teacher design determines the type of geometry transferred and the alignment burden imposed on training.

ADD uses identical monocular detector architectures for teacher and student. The teacher receives the image plus accurate ground-truth depth, while the student retains the baseline monocular inputs, including estimated or learned depth representations. The teacher is trained independently, frozen, and then used to supervise the student through intermediate features and decoder responses. MonoDETR uses object-wise depth ground truth for the teacher, whereas CaDDN and PatchNet use pixel-wise depth ground truth [2211.16779]. The identical architecture provides corresponding image locations, feature scales, decoder layers, channels, and query semantics, avoiding the modality and representation gaps associated with stereo- or LiDAR-based teachers.

Other configurations deliberately use different modalities. ADU-Depth trains a teacher on concatenated left–right RGB images and dense ground-truth depth, while its student receives a single RGB image. Both use a NeWCRFs-style encoder–decoder with a Swin Transformer encoder and neural-window fully connected CRF-style decoder, but the teacher accepts six input channels and the student three. The student uses attention adaptation, uncertainty modeling, and focal-depth response distillation to approximate stereo-informed representations [2309.14744].

MonoTher-Depth transfers an RGB foundation model into a thermal student. Both networks use the metric version of Depth Anything, consisting of a DINOv2 feature extractor, DPT decoder, and metric-bins module. The RGB teacher is used only during training; the thermal student receives a single thermal image during deployment. RGB depth is geometrically reprojected into the thermal camera, and a confidence network determines which RGB predictions should supervise the thermal model [2504.16127].

LiRCDepth uses CaFNet, a 62.25-million-parameter radar–camera teacher, to supervise a 12.65-million-parameter MobileNetV2 student. The student receives RGB and radar inputs at inference but uses a much lighter encoder, fusion mechanism, and decoder. Distillation covers camera features, radar features, pairwise decoder relationships, and intermediate depth maps [2412.11507]. XD-RCDepth further compresses the student to 8.89 million parameters and uses FiLM-based radar-conditioned fusion. Its distillation includes Grad-CAM saliency transfer and depth-distribution distillation [2510.13565].

A different configuration appears in GeoLaV, where a frozen 3D-aware teacher such as $\pi^3$ supplies geometry-rich features to a text-driven video segmentation model. The teacher does not directly supervise scalar depth; instead, the student memory representation is aligned with features encoding depth relationships, viewpoint consistency, correspondence, and scene structure [2606.24464]. In geometry-conditioned visual place recognition, Depth Anything V2 acts as a geometric foundation model, while DINOv2 and SALAD provide the appearance representation. Per-image normalized depth is embedded into the visual token space and selectively added to appearance tokens [2609.27370].

## 3. Distillation targets and mathematical mechanisms

DAD distillation targets range from dense scalar depth to high-level descriptors. The principal mechanisms can be organized as follows.

### Feature and response distillation

ADD distills both multi-level convolutional features and decoder-query responses. For feature level $k$, the student feature $F^k$ is transformed into a depth-aware representation $F^k_{3d}$ and matched to the teacher feature $T^k$. Foreground and background regions receive different weights:

$$
\mathcal L_{\mathrm{feat}}
=
\sum_k
\left[
\alpha_I
\left\|
M^k(F^k_{3d}-T^k)
\right\|_2^2
+
\beta_I
\left\|
(1-M^k)(F^k_{3d}-T^k)
\right\|_2^2
\right].
$$

The reported values are $\alpha_I=1.0$ and $\beta_I=0.1$. For transformer decoder queries, Hungarian matching produces a foreground-query mask $M_f$, and target-aware cross-attention retrieves teacher query information. The response loss uses $\alpha_v=1.0$ and $\beta_v=0.5$ for foreground and background queries, respectively [2211.16779].

CADiT instead forms a channel-correlation map between flattened student and teacher features:

$$
CCM=\operatorname{Softmax}(F_S^{\mathsf T}F_T),
$$

then reconfigures student features by

$$
F_S'=F_S+F_S CCM
$$

and minimizes

$$
\mathcal L_{\mathrm{CADiT}}=\left\|F_S'-F_T\right\|_2.
$$

The method also applies multi-scale $L_1$ matching to teacher and student depth outputs. Its total objective combines self-supervised photometric reconstruction, edge-aware smoothness, CADiT feature transfer, and output depth distillation with $\alpha_d=0.1$ [2303.10386].

### Depth-level and reprojection-aware distillation

MAL constructs a depth pseudo-target by comparing teacher- and student-derived reprojections. At each pixel, the depth associated with the lower image-reprojection error is selected:

$$
D_{td}(p)=
\begin{cases}
D_t(p), & \text{if teacher-based reprojection has lower loss},\\
D_s(p), & \text{if student-based reprojection has lower loss}.
\end{cases}
$$

The student is then trained toward the selected target through an $L_1$ depth loss. This is prediction-level DAD: it is spatially adaptive, reprojection-aware, and not unconditional teacher imitation. MAL also introduces temporal hints that associate moving instances across three adjacent frames using Hungarian matching and approximately linear motion, producing motion-rectified reconstruction candidates [2402.11507].

Stereo-assisted monitored distillation for depth completion uses an ensemble of completion teachers—PENet, MSG_CHN, ENet, and NLSPN—and a stereo teacher, PSMNet. Candidate teacher depths are selected pixelwise according to photometric reconstruction error. The resulting adaptive target avoids transferring systematic completion-teacher errors when another candidate, including stereo depth, better reconstructs the target image [2303.15840].

### Uncertainty-aware distillation

ADU-Depth predicts per-pixel log variance $s_i=\log\sigma_i^2$. Feature residuals are weighted using a heteroscedastic Gaussian negative-log-likelihood:

$$
\frac{1}{2}\exp(-s_i)r_i^2+\frac{1}{2}s_i,
$$

whereas response residuals use a Laplace-based form:

$$
\sqrt{2}\exp(-s_i)|r_i|+s_i.
$$

The uncertainty map reduces the influence of regions where monocular observations cannot reliably reproduce stereo knowledge, such as distant background, boundaries, occlusions, and weakly observed areas. Focal-depth response distillation additionally emphasizes cases with poor teacher–student depth agreement [2309.14744].

MonoTher-Depth learns a confidence map for RGB depth using RGB–thermal feature similarity, warped depth discrepancy, RGB depth, thermal depth, and RGB appearance. The confidence-weighted consistency loss is

$$
L_{\mathrm{cons}}
=
\frac{1}{M}
\sum_j
\operatorname{sg}(\hat W_r^j)
\left|
\operatorname{sg}(\hat D_r^j)-\breve D_{tr}^j
\right|.
$$

The method removes the top $20\%$ of residuals and retains pixels in the top $80\%$ of feature similarity. Its ablation shows that unweighted distillation can be worse than no distillation, supporting selective teacher supervision [2504.16127].

### Distributional depth distillation

XD-RCDepth converts scalar teacher and student depths into soft distributions over fixed depth-bin centers. With logits defined by negative distance to bin centers,

$$
z_i^{(\cdot)}(p)=-\left|d^{(\cdot)}(p)-c_i\right|,
$$

the temperature-scaled distributions are

$$
\mathbf p^S(p)=\operatorname{softmax}\left(\frac{\mathbf z^S(p)}{\tau}\right),
\qquad
\mathbf q^T(p)=\operatorname{softmax}\left(\frac{\mathbf z^T(p)}{\tau}\right).
$$

Depth-distribution distillation minimizes the forward KL divergence:

$$
\mathcal L_{D^2\text{-KD}}
=
\frac{\tau^2}{|\Omega|}
\sum_{p\in\Omega}
D_{\mathrm{KL}}\left(\mathbf q^T(p)\,\|\,\mathbf p^S(p)\right).
$$

The paper does not report the numerical temperature, bin count, or bin spacing. The distribution is constructed deterministically from scalar depth predictions; it is not produced by a separate uncertainty head [2510.13565].

## 4. Geometry-conditioned feature adaptation

A central design question is whether teacher and student tensors should be matched directly. Many DAD methods instead use geometry to transform the student before matching.

ADD derives a 3D-aware positional embedding from ground-truth depth. A pointwise MLP implemented with $1\times1$ convolutions transforms the depth representation, and a channel-wise argmax selects a discretized depth-position index. A learned embedding dictionary converts that index into positional features, which are added to flattened student features before self-attention. Depth therefore enters the attention queries and keys as a geometric positional signal rather than merely as a regression target [2211.16779].

ADU-Depth uses self-attention adaptation on student encoder and decoder features before squared matching with stereo-teacher features. The adaptation is intended to capture long-range dependencies and reduce the stereo–monocular domain gap [2309.14744].

Depth3DLane transfers Depth Anything V2 features from layers associated with $S32$ and $S64$ into StudentS32 and StudentS64 modules. The resulting depth-aware features are concatenated with Spatial Transformation Pyramid features before BEV lane prediction. Its Hierarchical Depth-Aware Head separately learns multiscale depth features through an encoder–decoder, while a CRF uses appearance and teacher-derived depth cues to encourage lane continuity [2504.18325].

GeoDistill uses pooled bidirectional attention between appearance and geometry branches. Appearance queries attend to pooled geometry keys and values, and geometry queries attend to pooled appearance keys and values. A pixelwise reliability estimator then controls geometry injection:

$$
F_i=
\rho_i
\left(
\bar A_i\odot(1+r_i\odot q_i)
+
r_i\odot\bar G_i
\right).
$$

The teacher is removed after training; the deployed model infers a task-oriented relative geometry representation from RGB alone [2609.03378].

In visual place recognition, DAD embeds normalized depth into the same token space as DINOv2 appearance features. The conditioned tokens are

$$
\mathbf x
=
E_v(I)+\boldsymbol{\alpha}\odot E_g(D),
$$

where $\boldsymbol{\alpha}$ is a trainable channel-wise gate initialized to $0.01$. A two-stage objective first aligns the geometry-conditioned descriptor with the frozen appearance teacher and subsequently refines the representation using retrieval loss [2609.27370].

## 5. Objectives, supervision, and training schedules

DAD objectives usually combine baseline task supervision with one or more distillation terms. ADD uses

$$
\mathcal L
=
\mathcal L_{\mathrm{cls}}
+
\mathcal L_{\mathrm{reg}}
+
\mathcal L_{\mathrm{depth}}
+
\alpha\mathcal L_{\mathrm{feat}}
+
\beta\mathcal L_{\mathrm{ed}},
$$

with $\alpha=\beta=1.0$. The response branch is disabled when the detector lacks an encoder–decoder transformer head, and $\mathcal L_{\mathrm{depth}}=0$ when the baseline uses a pretrained depth estimator rather than jointly learning depth [2211.16779].

DDND combines self-supervised photometric and smoothness losses with CADiT and multiscale output distillation:

$$
\mathcal L
=
\mathcal L_{ss}
+
\alpha_d\mathcal L_{\mathrm{CADiT}}
+
\mathcal L_{\mathrm{Out}},
$$

where $\alpha_d=0.1$. The system uses view synthesis, minimum photometric reprojection, auto-masking, and edge-aware inverse-depth smoothness, but no ground-truth depth term is reported [2303.10386].

LiRCDepth combines uncertainty-rectified depth supervision with four distillation terms:

$$
L_{\mathrm{total}}
=
L_{\mathrm{Depth}}
+
\gamma_1L_{\mathrm{KD}^{I}}
+
\gamma_2L_{\mathrm{KD}^{R}}
+
\gamma_3L_{\mathrm{KD}^{Dec}}
+
\gamma_4L_{\mathrm{KD}^{D}}.
$$

The camera-feature, radar-feature, decoder-structure, and inter-depth terms transfer distinct forms of teacher information. The student uses accumulated and single-frame LiDAR-derived supervision, with uncertainty rectification to reduce the influence of registration errors in accumulated depth [2412.11507].

GeoLaV uses segmentation and cosine feature-similarity losses. During monocular geometry pretraining, pixels are lifted into 3D using predicted depth or point maps, transformed through virtual camera poses, reprojected, and inpainted to produce synthetic videos. During real-video fine-tuning, the student memory features are aligned with frozen 3D and visual foundation model features. No direct scalar depth-regression loss is defined [2606.24464].

The training-only nature of DAD is a recurrent design pattern. At inference, ADD removes the teacher, ground-truth depth, distillation branches, and ground-truth-derived positional encodings. ADU-Depth removes the stereo teacher. MonoTher-Depth retains only the thermal student. MAL removes instance segmentation, teacher execution, motion rectification, and distillation-target construction. GeoDistill removes Depth Anything V2 and all sensor depth. This separation permits privileged supervision without imposing the teacher’s computational or sensor requirements on deployment.

## 6. Empirical evidence, limitations, and methodological distinctions

DAD has been evaluated across 3D detection, depth estimation, depth completion, thermal perception, radar–camera fusion, lane detection, video segmentation, salient-object detection, and visual place recognition.

In monocular 3D detection on KITTI, MonoDETR + ADD improves validation $AP_{3D}$ from $28.84/20.61/16.38$ to $30.71/21.94/18.42$ for Easy, Moderate, and Hard difficulty. CaDDN and PatchNet also improve, and the MonoDETR-based system reports test $AP_{3D}$ of $25.61/16.81/13.79$ with no additional inference computational cost [2211.16779].

For nano-drone depth estimation, DDND reduces Abs Rel from $0.157$ without KD to $0.147$ with KD, while retaining a 310K-parameter student and achieving 1.24 FPS on GAP8 at $128\times160$ resolution. The complete system improves Sq Rel, RMSE, RMSE log, and threshold accuracy, although it remains less accurate than the 3.1-million-parameter teacher [2303.10386].

ADU-Depth reports KITTI Eigen-split values of Abs Rel $0.049$, RMSE $2.080$, and $\delta_1=0.976$. On DrivingStereo, it reports Abs Rel $0.028$ and RMSE $1.146$. Its ablations show that removing focal-depth distillation, uncertainty modeling, or attention adaptation degrades performance [2309.14744].

In depth completion, the stereo-assisted monitored-distillation method achieves KITTI MAE $217.881$, RMSE $800.503$, iMAE $0.920$, and iRMSE $2.203$, compared with baseline MAE $223.021$ and RMSE $808.190$. The method is trained without dense ground-truth depth and uses stereo supervision only during training [2303.15840].

For thermal depth estimation, MonoTher-Depth achieves an approximately $22.88\%$ reduction in Abs Rel during unlabeled ViViD++ adaptation, from $0.153$ to $0.118$. Its ablation shows that removing confidence modeling produces worse performance than no distillation, demonstrating that teacher reliability is consequential under cross-modal domain shift [2504.16127].

In radar–camera depth estimation, LiRCDepth reduces MAE from $1.638$ without KD to $1.514$ with KD at the 50 m nuScenes evaluation range, while reducing parameters from 62.25M for CaFNet to 12.65M. XD-RCDepth reduces the student to 8.89M parameters and reports MAE reductions of 7.91%, 7.96%, and 7.97% at 50 m, 70 m, and 80 m, respectively, relative to its non-distilled student [2412.11507; 2510.13565].

Depth3DLane reports OpenLane F1 of 61.5%, close-range $z$-error of 0.145 m, far-range $z$-error of 0.133 m, and 88 FPS. The $S32+S64$ distillation combination obtains the best general balance in its scale ablation, while the paper does not provide the explicit feature-distance loss, feature dimensions, or distillation weights [2504.18325].

In video segmentation, GeoLaV improves Ref-Youtube-VOS from 65.1 to 70.5 in the reported $\mathcal J\&\mathcal F$ ablation when both monocular geometry pretraining and geometry-aware distillation are used. Its zero-shot performance reaches 47.0 on Ref-Youtube-VOS and 31.6 on MeViS in the supplied comparison [2606.24464].

For RGB-D salient-object detection, GeoDistill ignores dataset depth and uses Depth Anything V2 only during training. It achieves the best or tied-best result in 26 of 36 metric–dataset comparisons and reduces ReDWeb-S relative MAE by 13.4%. The method’s depth branch is relative and task-oriented rather than metric [2609.03378].

In forest visual place recognition, the earlier depth-aware descriptor method improves WildCross inter-sequence Recall@1 from 49.38% to 62.69% relative to its frozen appearance baseline, while the geometry-conditioned VPR formulation improves a matched SALAD baseline from 61.41% to 66.37% and Recall@5 from 65.86% to 72.49% [2606.13206; 2609.27370].

Despite these results, several limitations recur. Teacher errors can be propagated when reliability is not modeled or when the teacher is systematically biased. Monocular depth remains scale-ambiguous and may hallucinate geometry. Direct scalar depth matching can be sensitive to representation conventions, such as depth versus inverse depth, and may lack scale alignment. Cross-modal methods require calibration, overlap, or modality compatibility during training. Photometric selection can fail in textureless, reflective, transparent, or repeatedly textured regions. Many papers omit exact tensor dimensions, attention-head counts, normalization layers, temperature values, loss weights, teacher-update mechanisms, or mask definitions. Several methods report no FLOP, memory, energy, or complete deployment analysis.

DAD also encompasses materially different objectives. It includes explicit privileged-depth transfer, stereo-to-monocular adaptation, confidence-weighted RGB-to-thermal transfer, reprojection-based pseudo-label selection, distributional depth matching, geometry-conditioned token modulation, and implicit 3D feature alignment. These mechanisms are related through their use of geometric information in teacher supervision, but they should not be treated as interchangeable. A method that merely adds a depth consistency regularizer, such as PQDAST, is narrower than a system that explicitly distills depth-conditioned features, responses, distributions, or geometric relations.

The general methodological principle is therefore selective geometric transfer: use a privileged or stronger source of depth and structure during training, transform the student representation before matching when modalities differ, weight supervision according to reliability, and remove the privileged branches at inference. DAD is strongest when depth functions not only as an auxiliary prediction target, but as a mechanism that determines what information is transferred, where it is transferred, and how strongly the student should trust it.

Source: https://www.emergentmind.com/topics/depth-aware-distillation-dad