---
title: 'DisDop: Distillation with Domain Priors'
url: https://www.emergentmind.com/topics/disdop
type: topic
---

# DisDop: Distillation with Domain Priors

DisDop is a two-stage framework for **open-vocabulary aerial object detection (OVAOD)** that distills **multi-level domain priors** from remote sensing foundation models into a lightweight detector. It is introduced as **“Distillation with Domain Priors for Open-Vocabulary Aerial Object Detection”** and targets a setting in which detectors must recognize aerial objects from natural-language category descriptions, including categories unseen during training. Its central premise is that open-vocabulary detectors designed for natural images transfer poorly to aerial imagery because of domain shifts in scale, orientation, object density, and local appearance; DisDop addresses this by transferring visual, textual, and contextual priors from **RemoteCLIP** and **DINOv3** into an **LAE**-based detector [2605.24639].

## 1. Problem formulation and design rationale

DisDop is situated in the OVAOD setting, where the detector is not restricted to a predefined closed label set and must instead use text descriptions to detect both seen and unseen aerial categories. The motivating difficulty is that aerial imagery differs substantially from natural-image corpora: the paper highlights a **broader range of variations in scales**, **orientation differences**, **lack of distinctive appearance features**, and the scarcity of large-scale labeled drone-view data. This creates a domain gap for open-vocabulary detectors pretrained on natural scenes [2605.24639].

The framework is motivated by a second mismatch as well: even strong vision-language models and detection systems often inherit priors from natural-image data, whereas aerial imagery benefits from priors learned directly in remote sensing regimes. DisDop therefore uses **remote sensing foundation models** rather than relying only on models trained on natural images. In the paper’s formulation, **RemoteCLIP** contributes cross-modal alignment and text-space semantics, while **DINOv3** contributes fine-grained local and spatial feature structure. This suggests a division of labor in which language-aligned semantics and local patch discrimination are distilled separately but coordinated within a single detector [2605.24639].

The resulting notion of **“multi-level domain priors”** has three components. **Visual priors** come from the RemoteCLIP vision encoder and DINOv3 vision encoder; **textual priors** come from the RemoteCLIP text encoder and capture inter-category semantic relations; and **contextual priors** inject global image context into local instance supervision, particularly for small objects. The framework is built on the **LAE detector**, with a **Swin Transformer-Tiny** student backbone, and is designed so that the heavy teacher models are needed during distillation rather than inference [2605.24639].

## 2. Overall architecture and two-stage training procedure

DisDop is trained in **two sequential stages**. **Stage I** performs **Visual Prior Distillation (VPD)** on **unlabeled LAE-1M** images. In this stage, the student backbone is trained to match a fused teacher representation derived from **RemoteCLIP-ViT-L/14** and **DINOv3-ViT-L/16**. Only the student backbone is updated; the teacher side is frozen. The goal is to inject domain-specific visual priors into the backbone before full detector training [2605.24639].

**Stage II** trains the full detector on **labeled LAE-1M**. The paper states that the objective combines the standard LAE detection losses with a textual/contextual distillation loss, denoted conceptually as
\[
\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{det}} + \mathcal{L}_{\text{distill}}.
\]
The exact decomposition of \(\mathcal{L}_{\text{det}}\) and the weighting between detection and distillation losses are not provided in the supplied text, and the paper excerpt likewise does not specify the exact Stage II coefficient schedule [2605.24639].

The architecture distinguishes where distillation occurs. **Visual distillation** acts at the **backbone** level. **Textual prior distillation** acts on the detector’s **output category features** \(f_c^i\). **Contextual prior distillation** modifies the visual supervision target \(f_{eh}^j\) used in Stage II. The paper does not state that distillation is performed explicitly at the proposal level, neck level, or box-regression head, so the framework is best described as backbone-level visual distillation plus representation-level semantic and contextual alignment [2605.24639].

Several implementation parameters are explicit. The teachers are **RemoteCLIP-ViT-L/14** and **DINOv3-ViT-L/16**; the student is **Swin Transformer-Tiny**; Stage I runs for **15,000 iterations** with **batch size 8** and **AdamW**; the experiments use **8 NVIDIA A100 GPUs**. The paper does not specify the learning rate, weight decay, the exact values of \(\lambda_{\text{cosine}}\) and \(\lambda_{\text{attn}}\), the initialization of \(\mu\), or the precise Stage II loss weight, and these omissions matter for full reimplementation [2605.24639].

## 3. Visual prior distillation and teacher fusion

A central claim of DisDop is that **RemoteCLIP** and **DINOv3** provide complementary visual priors. RemoteCLIP is strong in **cross-modal alignment** and thus useful for open-vocabulary semantics, but image-level contrastive training leaves it weaker at **local feature extraction**. DINOv3 is strong in **local spatial representation**, preserving semantic consistency across patches, but does not natively supply text alignment. DisDop fuses the two before distilling them into the student backbone [2605.24639].

For an input image, the framework extracts patch/token features
\[
\mathbf{F}_{\text{CLIP}}
\]
from RemoteCLIP and
\[
\mathbf{F}_{\text{DINO}}
\]
from DINOv3. It then computes a DINOv3 self-similarity matrix
\[
\mathbf{S} = \frac{\mathbf{F}_{\text{DINO}} \cdot \mathbf{F}_{\text{DINO}}^T}{\|\mathbf{F}_{\text{DINO}}\| \cdot \|\mathbf{F}_{\text{DINO}}^T\|} \in \mathbb{R}^{N \times N},
\]
where \(N\) is the number of visual patch tokens. This matrix captures patchwise semantic and spatial relations and is used to structure the RemoteCLIP features [2605.24639].

A distinctive component is **semantic-adaptive outlier detection (SAOD)**. For each patch \(i\), the method defines a DINO-guided semantic neighborhood
\[
\mathcal{N}_i = \{ j \mid S_{ij} > \gamma_{\text{lof}} \},
\]
where \(\gamma_{\text{lof}}\) is a similarity threshold. Within this neighborhood, **Local Outlier Factor (LOF)** is applied to the RemoteCLIP features, producing a context-dependent outlier mask
\[
\mathbf{M} \in \mathbb{R}^{N \times N},
\]
with \(M_{ij}=1\) indicating that patch \(j\) is an outlier relative to patch \(i\)’s neighborhood. The paper stresses that outlier status is relative rather than global: a patch may be anomalous for one semantic center and normal for another [2605.24639].

After filtering weak or anomalous relations, attention weights are defined by
\[
\mathbf{A}_{ij} = \frac{\exp(S_{ij}/\tau)}{\sum_k \exp(S_{ik}/\tau)}
\]
when
\[
S_{ij} > \gamma \quad \text{and} \quad M_{ij}=0,
\]
and \(\mathbf{A}_{ij}=0\) otherwise. The fused teacher feature is then
\[
\mathbf{F}_{\text{teacher}} = \mathbf{A} \cdot \mathbf{F}_{\text{CLIP}}.
\]
In effect, DINOv3 supplies the affinity structure and RemoteCLIP supplies the semantic features, yielding a teacher representation intended to reinforce intra-object consistency while preserving open-vocabulary semantics [2605.24639].

The distilled student is trained with two losses. The **cosine similarity loss** is
\[
\mathcal{L}_{\text{cosine}} = \frac{1}{N} \sum_{i=1}^{N} \left( 1 - \frac{ \mathbf{f}_{\text{teacher}}^{(i)} \cdot \mathbf{f}_{\text{student}}^{(i)} }{ \|\mathbf{f}_{\text{teacher}}^{(i)}\| \cdot \|\mathbf{f}_{\text{student}}^{(i)}\| } \right),
\]
and the **attention distillation loss** is
\[
\mathcal{L}_{\text{attn}} = \mathrm{KL}\left(\mathbf{P}_{\text{teacher}} \,\|\, \mathbf{P}_{\text{student}}\right),
\]
where
\[
\mathbf{P}_{\text{teacher}} = \mathrm{softmax}\left( \mathbf{F}_{\text{teacher}} \mathbf{F}_{\text{teacher}}^T / \tau_t \right),
\quad
\mathbf{P}_{\text{student}} = \mathrm{softmax}\left( \mathbf{F}_{\text{student}} \mathbf{F}_{\text{student}}^T / \tau_s \right),
\]
with \(\tau_t=0.1\) and \(\tau_s=1.0\). The total backbone distillation objective is
\[
\mathcal{L}_{\text{backbone}} = \lambda_{\text{cosine}} \mathcal{L}_{\text{cosine}} + \lambda_{\text{attn}} \mathcal{L}_{\text{attn}}.
\]
The specific values of \(\lambda_{\text{cosine}}\) and \(\lambda_{\text{attn}}\) are not reported in the provided text [2605.24639].

## 4. Textual priors, semantic geometry, and contextual enhancement

The paper argues that pointwise alignment between detector features and visual targets is insufficient for open-vocabulary generalization because unseen-category recognition depends on preserving **semantic topology** across categories. A baseline pointwise distillation objective is written as
\[
L_{KD} = \frac{1}{N} \sum_{i=1}^{N} \left( f_c^i - f_v^i \right)^2,
\]
where \(f_c^i\) is the detector output category feature for ROI \(i\) and \(f_v^i\) is the RemoteCLIP visual feature for ROI \(i\). The paper criticizes this form because it handles small objects poorly and ignores inter-category relations [2605.24639].

DisDop instead uses **textual prior distillation** based on pairwise relationships among **RemoteCLIP text embeddings** \(t^i\). The core objective is
\[
L_{\text{distill}} =
\frac{1}{N^2} \sum_{i=1}^{N}\sum_{j=1}^{N}
\left(
\left\| f_c^i - f_{eh}^j \right\|_{cos}
-
\left\| t^i - t^j \right\|_{cos}
\right)^2.
\]
Here \(f_c^i\) is the detector output feature for instance \(i\), \(f_{eh}^j\) is an enhanced visual feature for instance \(j\), and \(t^i,t^j\) are text embeddings from RemoteCLIP’s text encoder. The semantic relation modeling is therefore implicit in pairwise cosine distances rather than in an explicit graph or prompt-ensemble mechanism. The paper gives the example that “bridge” and “ship” should be closer than “bridge” and “windmill,” illustrating how textual geometry supervises representation geometry [2605.24639].

The **contextual prior** is introduced because aerial images often contain tiny objects whose local ROI crops do not carry enough semantic evidence. The paper argues that global scene context can disambiguate such cases. It therefore defines an enhanced visual target
\[
f_{eh}^j = LN\left( \mu f_g^j + (1 - \mu) f_v^j \right),
\]
where \(f_v^j\) is the local visual feature from the ROI crop, \(f_g^j\) is a global visual feature extracted from the entire image containing the ROI, \(\mu \in [0,1]\) is a learnable balancing parameter, and \(LN(\cdot)\) is layer normalization. In the framework description, this contextual prior modifies the training target used in relational distillation rather than introducing a separate inference-time context head [2605.24639].

This design indicates that textual and contextual priors are coupled in Stage II: the model is not only encouraged to match local visual targets but to organize its category features according to a language-derived geometry, while that geometry is anchored to supervision targets enhanced with global scene context. A plausible implication is that DisDop is particularly tuned to the two central aerial-detection failure modes named in the paper: unseen-category semantics and small-object ambiguity [2605.24639].

## 5. Training data, benchmarks, and evaluation protocol

DisDop uses **LAE-1M** for training in both stages: **unlabeled LAE-1M** in Stage I and **labeled LAE-1M** in Stage II. Evaluation is conducted on three aerial benchmarks: **DIOR**, **DOTAv2.0**, and **LAE-80C**. The paper reports **open-set detection** results in Table 1 and **closed-set fine-tuning** results in Table 2. For **DOTAv2.0**, only **horizontal bounding box detection** is evaluated [2605.24639].

The reported metrics are **\(AP_{50}\)** and **\(mAP\)**. The supplied text does not specify seen/unseen class splits, whether the setting is generalized zero-shot versus strict unseen-only, or the exact prompt construction used for category texts. The paper excerpt also does not report per-seen and per-unseen AP values. These omissions are material because they limit full protocol reconstruction and make the benchmark definition less transparent than the model formulation itself [2605.24639].

The hyperparameters explicitly reported include the Stage I teacher-fusion thresholds
\[
\gamma = 0.5,\quad \gamma_{\text{lof}} = 0.8,\quad \tau_{\text{lof}} = 1.2,
\]
and the attention-distillation temperatures
\[
\tau_t = 0.1,\quad \tau_s = 1.0.
\]
Best-performing configuration choices vary by setting, but the paper excerpt states that the detector is built on **LAE**, the student backbone is **Swin Transformer-Tiny**, and teacher models are **RemoteCLIP-ViT-L/14** and **DINOv3-ViT-L/16** [2605.24639].

## 6. Empirical performance, ablations, and limitations

The main open-set detection results show consistent gains over prior baselines. On **DIOR**, the paper reports: **GLIP + DVC: 82.8 \(AP_{50}\)**, **GroundingDINO + DVC: 83.6**, **LAE-DINO: 85.5**, and **DisDop: 87.2**. On **DOTAv2.0**, the reported numbers are **43.0**, **46.0**, **46.8**, and **47.5 \(mAP\)**, respectively. On **LAE-80C**, they are **16.5**, **17.7**, **20.2**, and **22.5 \(mAP\)**, with the largest relative gain occurring on LAE-80C [2605.24639].

In closed-set fine-tuning, DisDop also improves over the strongest listed baseline. On **DIOR**, the sequence is **GLIP-FT: 88.9**, **GroundingDINO-FT: 91.1**, **LAE-DINO-FT: 92.2**, and **DisDop-FT: 92.6**. On **DOTAv2.0**, it is **51.5**, **55.1**, **57.9**, and **58.6**, respectively. These numbers indicate that the domain-prior distillation strategy is not confined to open-set transfer, but also benefits conventional fine-tuning [2605.24639].

| Benchmark | Baseline | DisDop |
|---|---:|---:|
| DIOR open-set \(AP_{50}\) | LAE-DINO 85.5 | **87.2** |
| DOTAv2.0 open-set \(mAP\) | LAE-DINO 46.8 | **47.5** |
| LAE-80C open-set \(mAP\) | LAE-DINO 20.2 | **22.5** |
| DIOR closed-set \(AP_{50}\) | LAE-DINO-FT 92.2 | **92.6** |
| DOTAv2.0 closed-set \(mAP\) | LAE-DINO-FT 57.9 | **58.6** |

The ablation studies are designed to isolate the effect of the framework’s major components. With **LAE-DINO** as baseline, adding only **VPD** improves DIOR from **85.5** to **86.6**, DOTAv2.0 from **46.8** to **47.2**, and LAE-80C from **20.2** to **21.5**. Adding only **TCPD** yields **86.2**, **47.1**, and **20.9**. Combining both gives the full **87.2**, **47.5**, and **22.5**. This shows that both visual and textual/contextual distillation contribute, while the visual branch provides the larger single gain in the reported table [2605.24639].

Teacher-fusion ablations further support the role of **SAOD**. The paper reports, for the visual-distillation stage, **w/o Outlier Filtering: 86.4 / 47.1 / 21.3**, **Global LOF: 86.3 / 47.0 / 21.2**, and **SAOD (Ours): 86.6 / 47.2 / 21.5** on DIOR, DOTAv2.0, and LAE-80C, respectively. A separate ablation on relational-distillation scope gives **Point-Wise: 85.8 / 46.8 / 20.4**, **Within-Image: 86.1 / 46.9 / 20.7**, and **Within-Batch (Ours): 86.2 / 47.1 / 20.9**, supporting the paper’s claim that batch-level semantic relations are denser and more useful than within-image ones in aerial scenes [2605.24639].

Several limitations remain explicit. The supplied text states that the paper does **not** provide exact layer mappings between teacher and student, exact projection/alignment head architectures, the full weighted Stage II objective, prompt templates for text embeddings, the exact inference-time classification formulation, or seen/unseen split definitions for the open-vocabulary benchmarks. These omissions do not alter the conceptual identity of DisDop, but they do constrain reproducibility from the paper text alone. A plausible implication is that code consultation would be necessary for faithful reimplementation, particularly in Stage II where the textual and contextual priors interface with LAE’s detector-specific components [2605.24639].

Source: https://www.emergentmind.com/topics/disdop