---
title: Adverse Weather Distillation
url: https://www.emergentmind.com/topics/adverse-weather-distillation
type: topic
---

# Adverse Weather Distillation

Adverse weather distillation denotes a family of distillation procedures used to transfer clean-scene, clear-weather, or teacher-model knowledge into models that must operate under haze, rain, snow, fog, night, or related degradations. In the recent literature, the term covers both conventional knowledge distillation—such as teacher–student alignment of outputs, features, cost volumes, or detector responses—and data distillation procedures that construct supervisory pairs from degraded imagery itself. The underlying motivation is consistent across tasks: adverse weather weakens direct supervision, violates photometric assumptions, or induces severe domain shift, so training is reformulated around invariances between clean and degraded observations or around pseudo-labels generated in a more reliable domain [2509.19165] [2505.12199] [2409.17001] [2402.18493] [2306.09008] [1904.04605] [2403.07292].

## 1. Terminological scope and task coverage

Within the literature, adverse weather distillation is not restricted to a single task or a single form of supervision. In self-supervised stereo matching, RoSe names a second training stage “adverse-weather distillation,” in which a teacher trained with scene-correspondence priors on clear/adverse pairs produces pseudo-disparities on clear stereo inputs, and a fresh student is trained on mixed clear or degraded inputs to match those predictions [2509.19165]. In monocular depth estimation, ACDepth uses a multi-granularity knowledge distillation strategy in which a student absorbs knowledge from a clear-trained teacher model and pretrained Depth Anything V2, with feature-wise distillation, ordinal guidance, and feature consistency across degradation types [2505.12199]. In adverse weather removal, distillation appears both as continual knowledge replay on a unified network structure and as soft residual transfer from CLIP features into a restoration backbone [2403.07292] [2306.09008].

The same logic extends beyond restoration and depth. In adverse weather optical flow, synthetic-domain motion statistics are distilled into a real-domain network by aligning cost-volume correlation histograms and by using pseudo-labels from a synthetic-degraded encoder [2409.17001]. In LiDAR-based 3D object detection, Sunny-to-Rainy Knowledge Distillation aligns RoI instance features and final detector responses between a sunny teacher and rainy student while adding a noise-aware correction term [2402.18493]. Earlier real-image deraining work uses “data distillation” rather than teacher–student logits: a rainy image is paired first with a coarsely derained soft label and then with a clean image onto which the extracted rain residual is re-applied, yielding a hard rainy–clean pair for shared-network training [1904.04605].

| Setting | Source of distilled knowledge | Distillation target |
|---|---|---|
| Continual all-in-one weather removal | Frozen old model and replay buffer | Predictions and principal features |
| Stereo matching | Clear-pair teacher pseudo-disparity | Student disparity on mixed inputs |
| Monocular depth estimation | Clear-trained teacher and Depth Anything V2 | Multi-scale features, ordinal relations, feature consistency |
| Optical flow | Synthetic-degraded encoder | Cost-volume histogram and pseudo-flow |
| 3D object detection | Sunny detector | RoI features and detector responses |
| Real-image deraining | Filtered rainy image and extracted rain residual | Soft and hard supervisory pairs |

A common misconception is that adverse weather distillation is merely “logit matching under bad weather.” The surveyed methods contradict that interpretation. Their distilled objects include per-pixel restorations, compressed mid-level embeddings, residual spatial features, disparity maps, cost-volume statistics, RoI features, and even re-synthesized rainy images [2403.07292] [2509.19165] [2306.09008] [2409.17001] [2402.18493] [1904.04605].

## 2. Distillation targets and objective functions

A central axis of variation is the object being matched. In continual all-in-one adverse weather removal, the old model at stage $t-1$ and the current model at stage $t$ are both evaluated on a stored degraded sample $\bar I$, yielding
$$
y^- = \phi^{(t-1)}(F^{(t-1)}(\bar I)), \qquad
y^+ = \phi^t(F^t(\bar I)).
$$
Prediction-level distillation combines an $\ell_1$ term with a contrastive reconstruction regularizer,
$$
L_{KD}(\bar I)=\|y^- - y^+\|_1 + \beta_2 \cdot L_{CT}(y^+,y^-,\bar I),
$$
while principal-feature distillation aligns compressed features,
$$
L_{PKD}(\bar I)=\|\psi(F^{(t-1)}(\bar I))-\psi(F^t(\bar I))\|_1.
$$
The total continual objective adds these replay losses to the standard single-weather restoration loss on current-task samples [2403.07292].

RoSe’s stereo formulation is structurally simpler at the final stage. After Step 1 self-supervised scene-correspondence learning, the model is frozen as a teacher $f$. For a clear pair $c_i$, the teacher produces a masked pseudo-disparity $D_i^*=f(c_i)$. A re-initialized student $f^*$ is trained on mixed inputs $m_i \in \{c_i,h_i\}$ with
$$
L_{kd}=\frac{1}{N}\sum_p |f^*(m_i)_p-f(c_i)_p| \odot M_v(p),
$$
where $M_v$ is obtained from a left–right consistency check. The key property is that the student is supervised by the teacher’s clear-scene prediction even when the input is foggy, rainy, or nocturnal [2509.19165].

ACDepth introduces a broader distillation stack. Its overall objective is
$$
L=L_d+\lambda_1 L_r+\lambda_2 L_c,
$$
with $\lambda_1=0.01$ and $\lambda_2=0.02$. The feature-wise term $L_d$ aligns multi-scale teacher and student features on mixed clear or degraded samples. The consistency term $L_c$ explicitly ties student degraded features to stop-gradient teacher-clear and student-clear features. The ordinal guidance distillation term $L_r$ focuses the model on uncertain regions defined by the normalized inverse-depth disagreement between teacher and student, with threshold $\gamma=0.95$ and ordinal tolerance $\tau=0.15$ [2505.12199].

Other domains use different alignment spaces. In CLIP-based adverse weather removal, the teacher signal is the residual feature
$$
R_j^C = F_j^C(I_c)-F_j^C(I_w),
$$
computed from CLIP image-encoder features on clean and weathered images; the SAR encoder’s residuals are matched to this quantity via an $\ell_1$ loss after channel-wise normalization [2306.09008]. In CH$^2$DA-Flow, the distillation target is not the output flow alone but the distribution of sampled cost-volume correlations. Synthetic and real degraded histograms are aligned with
$$
L^{kl}_{corr}=\sum_{i=1}^k p_{r,i}\log\frac{p_{r,i}}{p_{s,i}},
$$
and pseudo-flow supervision is added through $L^{self}_{flow}=\|F_r-F_{pse}\|_1$ [2409.17001]. In SRKD for 3D object detection, instance-feature matching, response distillation, and noise-aware prediction correction are combined as
$$
L=L_{cls}+L_{reg}+\eta_1 L_{ins}+\eta_2 L_{rsp}+\eta_3 L_{napc},
$$
with $\eta_1=2.0$, $\eta_2=0.5$, and $\eta_3=2.0$ [2402.18493].

These formulations show that adverse weather distillation is often closer to structured correspondence transfer than to classical classification KD. RoSe explicitly distinguishes its procedure from standard classification distillation by noting that it distills per-pixel restoration outputs rather than class probabilities and adds a contrastive term to pull the new output toward the old output while pushing it away from the degraded input [2403.07292]. A plausible implication is that weather-robust distillation methods tend to encode restoration or geometry priors directly in the supervisory object, rather than relying on softened categorical outputs.

## 3. Teacher construction, paired data, and pseudo-supervision

Because real paired labels under adverse weather are scarce, most methods devote substantial effort to constructing teacher signals or pseudo-pairs whose geometry remains valid. RoSe begins from clear-weather stereo datasets—DrivingStereo, MS2, and KITTI—and trains three CycleGAN-Turbo translators $T_{\rm night}$, $T_{\rm fog}$, and $T_{\rm rain}$ on unpaired samples. Each clear pair $c_i$ is converted into $h_i^{c}$ in the target style, and because the translator changes appearance rather than geometry, the original ground-truth disparity is preserved. This yields Adverse-DrivingStereo with $84\,665\times4$ images in Clear/Fog/Rain/Night, Adverse-MS2 with $79\,659\times4$, and Adverse-KITTI with $394\times4$ [2509.19165].

ACDepth similarly treats adverse weather generation as a prerequisite for robust distillation, but uses a one-step diffusion model built on Stable Diffusion Turbo with LoRA adapters. For each weather condition $c$, a clear image $e_i$ and prompt $P_c$ are processed to obtain
$$
h_i^c = F_c(\mathrm{SDT}(e_i,P_c)).
$$
Cycle-consistency, adversarial, and identity regularization losses are jointly optimized so that the translated image preserves scene content while adopting target weather statistics. Separate LoRA sets are trained for day$\rightarrow$night, day$\rightarrow$rain, and related conditions, using approximately $500$ clear/rain/night examples per domain and $4k$ for RobotCar night [2505.12199].

In adverse weather optical flow and LiDAR detection, teacher generation is explicitly domain-adaptive. CH$^2$DA-Flow bridges clean, synthetic degraded, and real degraded domains. Synthetic fog and rain are used in Clean-Degraded Motion Adaptation, and the resulting synthetic-degraded encoder becomes the source model for Synthetic-Real Motion Adaptation on real degraded imagery [2409.17001]. SRKD augments sunny Waymo point clouds with DRET, a rain simulation pipeline that combines particle-based splashes in Unity3D with physically based LiDAR scattering and attenuation, producing approximately $10\,K$ rainy scans offline [2402.18493].

The oldest of the surveyed methods, “Rain O’er Me,” uses no external teacher at all. Instead, it creates supervision from the rainy image itself. A rainy image $X$ is downsampled, super-resolved by a pretrained SRDN, combined by element-wise minimum, and refined with a guided filter:
$$
X_L=\mathrm{Downsample}(X;s=8),\quad
X_H=\mathrm{SRDN}(X_L),\quad
Y_S=\min(X,X_H),\quad
Y_g=\mathrm{GF}(X,Y_S).
$$
The result $Y_g$ is a blurred rain-free soft label. The network predicts a rain residual $\hat R$, enhances it to $\hat R_e$, and adds it to a different clean image $C$ to form $D=C+\hat R_e$, thereby generating a hard rainy–clean pair for a second supervisory branch [1904.04605].

Continual adverse weather removal replaces synthetic pairing with replay. Its memory buffer $M$ stores only degraded old images, not clean targets, and at the end of each task a uniform random subset of newly seen data is added with equal budget per task. Distillation is then performed against the frozen old model on each replayed sample [2403.07292]. This suggests that in continual settings, the pseudo-labeling problem is shifted from “how to obtain a clean target” to “how to preserve the old model’s behavior on degraded inputs.”

## 4. Architectural couplings between distillation and weather robustness

Adverse weather distillation is rarely a stand-alone auxiliary loss; it is usually embedded in architectures designed to expose weather-invariant or degradation-sensitive representations. The continual all-in-one restoration framework adopts FFA-Net as a unified backbone, abstracts it into a feature extractor $F(\cdot)$ and image projector $\phi(\cdot)$, and adds an auxiliary auto-encoder $\Phi$ pretrained on features of all stored old samples. Its encoder $\psi(\cdot)$ is then frozen and used as a PCA-style projector from $\mathbb{R}^{H\times W\times C}$ to $\mathbb{R}^{H\times W\times \hat C}$, with implementation based on multi-head channel self-attention and a learnable channel selection layer so as to mimic PCA at linear instead of quadratic cost [2403.07292].

RoSe couples distillation to a feature extractor enhanced with frozen visual foundation model priors. Outputs from pre-trained ViT blocks are fused with an FPN encoder at scales $\{1/16,1/8,1/4\}$, and an Anti-Adverse Feature Enhancement Module operates through instance normalization, batch normalization, channel-attention fusion, and Fourier-domain amplitude filtering:
$$
X_{IN}=\mathrm{IN}(X), \qquad X_{BN}=\mathrm{BN}(X),
$$
$$
A=\sigma(\mathrm{Conv}([X_{IN},X_{BN}])), \qquad
X'=A\odot X_{IN}+(1-A)\odot X_{BN}.
$$
The subsequent FFT/iFFT path filters degradation-related amplitude components while preserving phase and structure [2509.19165].

In CLIP-based adverse weather removal, the Spatially-Adaptive Residual Encoder and CLIP Weather Prior module make distillation explicitly architectural. Each SAR Transformer block applies multi-head self-attention followed by a SARFFN. The SAR module predicts location-varying combinations of basis depthwise kernels,
$$
w_i=\sigma(\mathrm{Conv}(DW(F_i^\ell))) \in \mathbb{R}^{N_{dw}\times H_i\times W_i},
$$
$$
k_i(x,y)=\sum_{j=1}^{N_{dw}} w_{i,j}(x,y)\otimes k_{i,j},
$$
so the residual branch concentrates on degraded regions. The CWP module then injects sample-specific CLIP priors $\theta_c$ and distribution-specific embeddings $\theta_{i,l}$ through cross-attention, with a side cross-entropy weather-classification loss to regularize the prior [2306.09008].

CH$^2$DA-Flow and SRKD exhibit analogous couplings in motion and detection systems. CH$^2$DA-Flow uses a FlowFormer backbone with cost-volume correlations, a DispNet stereo network, and PoseNet, so that synthetic$\rightarrow$real distillation operates directly on motion statistics in the cost space [2409.17001]. SRKD uses identical teacher and student detector architectures, but ties distillation to object-level similarity measures, high-confidence response selection, and explicit noise ratios inside predicted boxes [2402.18493]. A plausible implication is that adverse weather distillation is most effective when the intermediate representation chosen for transfer corresponds to the physical locus of the weather corruption: image residuals for restoration, disparities for stereo, cost volumes for flow, and point-density responses for LiDAR detection.

## 5. Quantitative behavior across representative systems

The strongest quantitative evidence in the surveyed literature comes from settings where weaker replay or weaker self-supervision is directly compared to stronger distillation. In continual all-in-one adverse weather removal on the three-task sequence Haze $\rightarrow$ Rain $\rightarrow$ Snow, Joint-M, which uses replay without distillation, reaches $24.94$ dB average PSNR; adding only prediction KD raises this to $30.36$ dB; adding principal-feature KD further raises it to $30.70$ dB, only $0.99$ dB below the “oracle” Individual-task upper bound of $31.69$ dB. On the two-task sequence Haze $\rightarrow$ Rain, the full model scores $29.99$ dB versus $26.56$ dB for AFC and $30.46$ dB for Individual-task training. The same study reports stable results down to $250$–$500$ exemplars, and that after a second task the first-task performance drops by only approximately $0.1$ dB, while after the third task each earlier task retains approximately $99\%$ of its original PSNR [2403.07292].

RoSe reports clear gains from its Step 2 adverse-weather distillation. On DrivingStereo weather validation, the mixed-training Step 2 model achieves Clear $0.78$ EPE and $1.60\%$ Bad-3.0, Fog $0.85$ and $1.71\%$, Rain $0.88$ and $1.88\%$, and Night $1.01$ and $3.14\%$. In zero-shot generalization, it attains Bad-3.0 of $3.63/4.85/4.75$ on DrivingStereo Clear/Fog/Rain and $20.8$ on MS2 Night. Ablations attribute approximately $10$–$20\%$ relative improvement to Step 2 distillation after prior gains from VFM fusion, scene-correspondence losses, and AFEM [2509.19165].

ACDepth reports improvements in monocular depth under rain and night on nuScenes. Against md4all-DD, rainy absRel decreases from $0.1414$ to $0.1377$ and night absRel from $0.1921$ to $0.1873$, corresponding to $-2.61\%$ and $-2.50\%$. Its ablations show that on nuScenes night, a baseline without distillation has absRel $0.2419$, adding $L_d$ reduces this to $0.1883$, adding $L_r$ gives $0.1879$, and adding $L_c$ gives $0.1872$; on rain, the sequence is $0.1572 \rightarrow 0.1404 \rightarrow 0.1414 \rightarrow 0.1377$ [2505.12199].

For weather removal, CLIP-based residual distillation also shows measurable gains. On Snow100K-L, Test1, and RainDrop, the full method achieves mean PSNR $29.21$ dB and SSIM $0.9393$, compared with TransWeather’s $28.44/0.9296$. In ablations, a baseline at $29.04$ PSNR improves to $29.23$ with CutMix, $29.43$ with CWP, and $29.71$ with SAR plus CLIP-SRD. On SPA-rain without retraining, it reports approximately $31.78/0.9419$ versus TransWeather’s $30.65/0.9337$ [2306.09008].

Motion and detection studies report similarly consistent trends. CH$^2$DA-Flow achieves EPE approximately $0.74$ on Weather-GOF rain versus $1.07$ for the best prior method, and $2.82$ on DenseFog versus $4.32$; on Real-Weather World it obtains EPE approximately $2.3$–$2.4$ and F1-all approximately $22$–$24\%$, compared with $3.6$–$7.0$ EPE and $32$–$61\%$ F1 for direct baselines [2409.17001]. In LiDAR 3D detection on WOD-DA, DSVT improves from All(L2) $60.4/57.9$ mAP/mAPH to $63.6/61.1$ with DRET-Aug plus SRKD; Voxel-RCNN improves from $55.8/53.7$ to $60.1/57.3$; PV-RCNN++ improves from $56.8/54.5$ to $59.8/57.6$. The same framework also slightly improves sunny All(L2-mAP), for example DSVT $73.4 \rightarrow 73.7$ [2402.18493].

“Rain O’er Me” does not report paired PSNR/SSIM on real scenes because no paired real ground truth exists, but emphasizes model compactness and speed: $51{,}552$ parameters, CPU inference time $1.45$ s, and GPU inference time $0.15$ s on a $512\times512$ image [1904.04605].

## 6. Conceptual distinctions, limitations, and open issues

Several conceptual distinctions recur across the literature. First, adverse weather distillation is not equivalent to domain adaptation, although the two are frequently combined. CH$^2$DA-Flow explicitly frames its method as cumulative homogeneous-heterogeneous adaptation, yet the synthetic$\rightarrow$real transfer stage is implemented through knowledge distillation on cost-volume histograms and pseudo-flow labels [2409.17001]. SRKD likewise relies on rainy augmentation and detector supervision, but its key cross-weather transfer mechanism is teacher–student alignment between sunny and rainy scans [2402.18493].

Second, the “teacher” need not be an externally supervised large model. It may be a frozen previous checkpoint in continual learning, a clear-weather model in cross-weather stereo or depth, a synthetic-domain encoder in optical flow, a pretrained multimodal encoder such as CLIP, or a soft label obtained by filtering the rainy image itself [2403.07292] [2509.19165] [2505.12199] [2306.09008] [1904.04605]. This suggests that adverse weather distillation is better understood as a transfer principle—moving supervision into a representation or domain where it is more stable—than as a fixed architecture.

Third, the surveyed results also delimit the method’s current boundaries. RoSe’s night results remain worse than its clear-weather results, with Bad-3.0 increasing from $1.60\%$ on Clear to $3.14\%$ on Night even after Step 2 distillation [2509.19165]. ACDepth’s ablation on nuScenes rain shows that adding the ordinal guidance term after $L_d$ changes absRel from $0.1404$ to $0.1414$, so the benefit of individual submodules can be non-monotonic until the full objective is assembled [2505.12199]. The CLIP-based restoration paper identifies a key challenge as reconciling CLIP’s global alignment objective with pixel-level restoration, and therefore delays the distillation loss until epoch $200$ [2306.09008]. “Rain O’er Me” notes that extremely heavy rain or rain plus mist may still leave residual fog, and that the soft-label path can over-smooth textures [1904.04605]. SRKD reports no inference-time penalty, but does incur approximately $20$–$50\%$ extra GPU training time and $10$–$15\%$ more VRAM, while DRET is not end-to-end and requires offline preprocessing [2402.18493].

A final misconception is that distillation under adverse weather always requires task IDs or specialized branches. The continual all-in-one weather removal framework explicitly states that no task ID or specialized branch is needed at test time, because haze, rain, snow, or mixtures are processed by the same $F$–$\phi$ pipeline [2403.07292]. Comparable unification is visible in RoSe’s mixed-input student and in ACDepth’s mixed clear plus synthetic degradation training [2509.19165] [2505.12199]. A plausible implication is that, as the field matures, the most durable use of adverse weather distillation may be not as a bolt-on regularizer but as a way to train unified backbones whose intermediate representations are anchored by clean-scene priors even when the observable input is severely degraded.

Source: https://www.emergentmind.com/topics/adverse-weather-distillation