VG-DETR: Vision Foundation-Guided Detector

• The paper presents VG-DETR, a detector that fuses a frozen VFM with a DINO-based mean-teacher framework to enhance pseudo-label quality in source-free adaptation.
• Its VFM-guided Pseudo-label Mining re-evaluates low-confidence detections via semantic similarity with class-wise prototypes, addressing noisy pseudo-labels.
• Dual-level alignment at both instance and image scales boosts cross-domain detection performance in remote sensing scenarios with minimal labeled samples.

Vision Foundation-Guided Detection Transformer (VG-DETR) is a semi-supervised, source-free object detector for remote sensing imagery that integrates a frozen Vision Foundation Model (VFM) into a DETR-style adaptation pipeline built on DINO and a mean-teacher framework (Han et al., 15 Aug 2025). The method is designed for cross-domain detection when source-domain images are unavailable during adaptation and only a small fraction of target-domain samples are labeled. Its defining components are VFM-guided Pseudo-label Mining (VPM), which uses VFM semantic priors to re-evaluate low-confidence pseudo-labels, and Dual-level VFM-guided Alignment (DVA), which aligns detector representations with VFM embeddings at both instance and image levels (Han et al., 15 Aug 2025).

1. Definition, naming, and conceptual scope

The exact method name is Vision foundation model-Guided DEtection TRansformer (VG-DETR). In the paper that introduces the acronym, the base detector is DINO, the VFM is DINOv2 ViT-L, and the full training procedure is organized as semi-supervised source-free adaptation with a teacher and student of identical architecture, both initialized from source-trained weights (Han et al., 15 Aug 2025). The method does not replace the detector backbone with the VFM; instead, it uses the VFM as a frozen semantic guide through offline feature extraction, prototype construction, pseudo-label assessment, and representation alignment.

Within the broader DETR literature, closely related systems exist under different names rather than under the exact acronym “VG-DETR.” Frozen-DETR uses frozen foundation models as plug-and-play modules for encoder and decoder enhancement, RT-DETRv4 uses a frozen VFM teacher to supervise the deep semantic bottleneck $F_5$ during training, and VFM$^4$SDG injects VFM-derived priors into both encoder and query pathways for domain generalization (Fu et al., 2024, Liao et al., 29 Oct 2025, Zhang et al., 23 Apr 2026). This suggests that “VG-DETR” can function as both a specific method name and a broader descriptive label for DETR-family detectors guided by frozen vision foundation representations.

2. Problem setting and motivation

VG-DETR studies source-free object detection (SFOD) in remote sensing images under domain shift. The source domain $D_s$ is available only through a pretrained source model, while target-domain adaptation is conducted without any source images. The paper further adopts a semi-supervised source-free regime in which 1%, 5%, or 10% of target-domain samples are labeled (Han et al., 15 Aug 2025).

The method is motivated by the failure mode of self-training under severe remote-sensing domain gaps. The paper identifies pseudo-label noise as the principal cause of training collapse in source-free adaptation, especially in remote sensing scenes characterized by dense object distributions, large scale variation, complex/cluttered backgrounds, many small objects, and cross-satellite, synthetic-to-real, and cross-modal shifts. Under such conditions, confidence thresholding is brittle: a low threshold admits many false positives, while a high threshold discards many true positives. VG-DETR addresses this by injecting external semantic priors from a frozen VFM, with the small labeled target subset providing anchors for prototype construction and stabilization (Han et al., 15 Aug 2025).

The source-free condition is enforced explicitly: adaptation uses only target-domain data, initialized from a source-trained detector, with no source images during target adaptation. The paper evaluates this setting on three remote-sensing transfer scenarios: cross-satellite, synthetic-to-real, and cross-modal adaptation (Han et al., 15 Aug 2025).

3. Architecture and training pipeline

The detector core is DINO inside a mean-teacher semi-supervised framework. For unlabeled target images, the teacher predicts on weakly augmented inputs, pseudo-labels are generated from teacher outputs, and the student is trained on the corresponding strongly augmented images. For labeled target images, the student is optimized with the standard supervised DINO detection loss (Han et al., 15 Aug 2025).

The teacher parameters are updated by EMA:

$$\Theta ^{t}_{i} \leftarrow \alpha \Theta^{t}_{i-1}+(1-\alpha) \Theta^{s}_{i},$$

where $\alpha = 0.999$ (Han et al., 15 Aug 2025). The student detection loss is decomposed into supervised and unsupervised terms:

$$\mathcal{L}_{\text {det }} = \mathcal{L}_{\text {det }}^{\text {sup }} + \mathcal{L}_{\text {det }}^{\text {unsup }} = \mathcal{L}_{\text {det }}(P_{s}^{\text {sup}}, Y^{\text {sup}}) + \mathcal{L}_{\text {det }}(P_{s}^{\text {unsup}}, P_{t}^{\text {unsup}}).$$

The VFM is incorporated in a “free lunch” manner. All target images are processed by a frozen DINOv2 ViT-L offline, producing feature maps

$F^i \in \mathbb{R}^{H \times W \times d}.$

These feature maps are stored on disk and reused during training, so the VFM is not used as the detector backbone and does not alter detector inference (Han et al., 15 Aug 2025).

The overall pipeline has two stages. In the offline stage, the frozen VFM extracts target-domain feature maps, ground-truth boxes from labeled target samples are used with ROI Align to obtain object instance features, and K-means is applied to form class-wise reference prototypes and background prototypes. In the online stage, teacher-generated pseudo-labels are filtered and recovered by VPM, the student is trained on labeled and pseudo-labeled target data, DVA aligns detector features to VFM representations, and the teacher is updated by EMA (Han et al., 15 Aug 2025).

4. VFM-guided pseudo-label mining

VPM is designed to improve both the quality and quantity of pseudo-labels. It operates on the low-confidence “gray zone” between a lower threshold and an upper threshold. The upper threshold is dynamically determined based on prior work, while the lower threshold is chosen by ablation. Predictions in this interval are not discarded immediately; instead, they are re-evaluated by VFM semantic similarity to target-domain reference prototypes (Han et al., 15 Aug 2025).

Prototype construction begins by applying ROI Align to the VFM feature maps of labeled target images, producing instance features

$F^i_{ins} \in \mathbb{R}^{N_i \times d}.$

For each category, K-means clustering is performed with objective

$$\sum_{k=1}^{K} \sum_{j=1}^{N_k} \| f_j - \mathbf{C}_k \|_2^2,$$

and cluster centroids

$$\mathbf{C}_k = \frac{1}{N_k} \sum_{i \in \mathcal{C}_k} f_i.$$

The resulting prototype bank is

$^4$0

where $^4$1 denotes background (Han et al., 15 Aug 2025).

For an unlabeled image, teacher-predicted boxes are used as proposals on the stored VFM feature map, and cosine similarity is computed between each ROI feature $^4$2 and the reference prototypes:

$^4$3

The decision rule is explicit. If the maximum similarity exceeds 0.5, and the associated prototype class matches the detector’s predicted class, the prediction is retained as a reliable pseudo-label; otherwise it is rejected (Han et al., 15 Aug 2025). The paper emphasizes that this rule-based interaction does not fuse confidence and similarity into a learned scalar. Rather, detector confidence defines the candidate set, and VFM similarity decides which low-confidence predictions should be recovered.

Background prototypes are also constructed, using background regions generated by “flipping” ground-truth boxes and removing overlapping regions based on IoU. This provides an explicit semantic background reference for rejecting noisy detections (Han et al., 15 Aug 2025).

5. Dual-level VFM-guided alignment

DVA aligns detector representations to VFM embeddings at two levels: instance-level alignment and image-level alignment. The detector-side object query features are denoted

$^4$4

and class assignments are obtained from detector classification logits

$^4$5

by sigmoid and argmax over classes (Han et al., 15 Aug 2025).

At the instance level, detector query features are grouped by predicted class and then softly clustered into $^4$6 prototypes per class using Sinkhorn-based soft clustering. These detector prototypes are aligned to the offline VFM reference prototypes with a contrastive objective in which same-class, same-component pairs are positives and all others are negatives:

$^4$7

The projection $^4$8 is a 3-layer perceptron with ReLU (Han et al., 15 Aug 2025).

At the image level, the detector backbone feature map

$^4$9

is aligned with the stored VFM feature map

$D_s$0

after channel projection by a $D_s$1 convolution and spatial matching by bilinear interpolation. The similarity loss is

$D_s$2

This encourages location-wise cosine agreement between detector and VFM feature maps (Han et al., 15 Aug 2025).

The complete objective is

$D_s$3

with $D_s$4 and $D_s$5 (Han et al., 15 Aug 2025). A central practical point is that DVA does not treat VFM guidance as a backbone replacement; it uses the VFM as an external semantic reference that regularizes the detector’s internal representations.

6. Empirical results, ablations, and limitations

VG-DETR is evaluated on three remote-sensing transfer scenarios using $D_s$6: xView $D_s$7 DOTA1.0, SRSD $D_s$8 DIOR, and HRRSD $D_s$9 SSDD (Han et al., 15 Aug 2025).

Scenario	VG-DETR $$\Theta ^{t}_{i} \leftarrow \alpha \Theta^{t}_{i-1}+(1-\alpha) \Theta^{s}_{i},$$0 at 1% / 5% / 10% labels	Representative comparison
xView $$\Theta ^{t}_{i} \leftarrow \alpha \Theta^{t}_{i-1}+(1-\alpha) \Theta^{s}_{i},$$1 DOTA	70.5 / 77.5 / 78.4	Semi-DETR: 67.7 / 75.1 / 76.8
SRSD $$\Theta ^{t}_{i} \leftarrow \alpha \Theta^{t}_{i-1}+(1-\alpha) \Theta^{s}_{i},$$2 DIOR	60.7 / 65.9 / 67.4	Semi-DETR: 59.9 / 65.4 / 67.0
HRRSD $$\Theta ^{t}_{i} \leftarrow \alpha \Theta^{t}_{i-1}+(1-\alpha) \Theta^{s}_{i},$$3 SSDD	61.4 / 70.6 / 78.0	MCL: 60.7 / 69.7 / 77.1

On the cross-satellite setting, the gains over semi-supervised DINO+MT are +3.3, +2.9, and +2.1 at 1%, 5%, and 10% labels, respectively. On synthetic-to-real adaptation, the gains over Semi-DETR are smaller but still positive. On the cross-modal setting, VG-DETR achieves the best reported results at all three label ratios (Han et al., 15 Aug 2025).

The ablation study on xView $$\Theta ^{t}_{i} \leftarrow \alpha \Theta^{t}_{i-1}+(1-\alpha) \Theta^{s}_{i},$$4 DOTA with 5% labels isolates the contribution of each component:

Method	$$\Theta ^{t}_{i} \leftarrow \alpha \Theta^{t}_{i-1}+(1-\alpha) \Theta^{s}_{i},$$5
Source-only	51.2
Fine-tuning	70.5
+ MT	74.6
+ VPM	76.5
+ DVA-Instance	74.5
+ DVA-Image	75.2
+ DVA-Instance + Image	75.6
Full VG-DETR	77.5

These ablations show that VPM alone gives +1.9 over the MT baseline, that image-level alignment is more effective than instance-level alignment alone, and that the best result requires combining VPM with both alignment levels (Han et al., 15 Aug 2025). Threshold analysis further shows that VPM outperforms both fixed-threshold and dynamic-threshold pseudo-label selection, reaching 76.5 versus 74.8 for the dynamic-threshold baseline. Prototype-number ablation finds the best performance at $$\Theta ^{t}_{i} \leftarrow \alpha \Theta^{t}_{i-1}+(1-\alpha) \Theta^{s}_{i},$$6 prototypes per class (Han et al., 15 Aug 2025).

The paper also identifies several constraints. VG-DETR relies on a small labeled target subset, so it is not a fully unsupervised SFOD method. Because VFM feature maps have limited spatial resolution, evaluation inaccuracies may occur for extremely small objects and densely packed regions. The preprocessing is offline, which assumes access to all target images in advance (Han et al., 15 Aug 2025).

7. Relation to other VFM-guided DETR systems

A common misconception is to equate any pretrained DETR with a vision-foundation-guided DETR. A plain DETR baseline for polyp detection, for example, uses ResNet-50, ordinary COCO pretraining, and dataset-specific augmentation, but it does not use CLIP-, DINO-, MAE-, or SAM-style guidance; it is therefore a conventional DETR baseline rather than a VFM-guided design (Liu et al., 2021). By contrast, several later systems explicitly inject frozen foundation-model priors into DETR-family detectors.

Frozen-DETR enhances DINO-family detectors by using frozen foundation models as plug-and-play feature enhancers. The class token is injected into decoder self-attention as an image query, and patch tokens are appended as an additional encoder scale. On COCO val with DINO-det-4scale + R50 + 12 epochs, it improves the baseline from 49.0 AP to 51.9 AP with one foundation model and to 53.8 AP with two foundation models (Fu et al., 2024).

RT-DETRv4 uses a frozen DINOv3-ViT-B teacher to supervise the AIFI output feature $$\Theta ^{t}_{i} \leftarrow \alpha \Theta^{t}_{i-1}+(1-\alpha) \Theta^{s}_{i},$$7 through a Deep Semantic Injector (DSI) and balances the auxiliary loss with Gradient-guided Adaptive Modulation (GAM). The VFM branch is removed at inference, so the detector keeps its original runtime path. The reported COCO results are 49.7/53.5/55.4/57.0 AP for the S/M/L/X variants at 273/169/124/78 FPS, respectively (Liao et al., 29 Oct 2025).

VFM$$\Theta ^{t}_{i} \leftarrow \alpha \Theta^{t}_{i-1}+(1-\alpha) \Theta^{s}_{i},$$8SDG addresses single-domain generalized object detection by using frozen DINOv3 ViT-L/16 as a cross-domain stability prior. It distills VFM token relations into the encoder through Cross-domain Stable Relational Prior Distillation and enhances queries with semantic prototypes and global VFM context through Semantic-Contextual Prior-based Query Enhancement. On the SDGOD benchmark, it improves Co-DETR from 44.2 to 50.8 average mAP and DINO from 41.4 to 49.4 (Zhang et al., 23 Apr 2026).

Relative to these systems, VG-DETR is distinguished by its source-free remote-sensing setting, its reliance on a small labeled target subset, and its use of the VFM primarily as an offline semantic assessor and alignment target rather than as an online backbone or an inference-time guidance branch (Han et al., 15 Aug 2025). This suggests a useful taxonomy within VFM-guided DETR research: some methods use the VFM as a training-only teacher, some as a parallel frozen enhancer, and some, like VG-DETR, as an offline source of semantic prototypes and dense feature references.