VFM$^{4}$SDG: Unveiling the Power of VFMs for Single-Domain Generalized Object Detection

Abstract: In real-world scenarios, continual changes in weather, illumination, and imaging conditions cause significant domain shifts, leading detectors trained on a single source domain to degrade severely in unseen environments. Existing single-domain generalized object detection (SDGOD) methods mainly rely on data augmentation or domain-invariant representation learning, but pay limited attention to detector mechanisms, leaving clear limitations under complex domain shifts. Through analytical experiments, we find that performance degradation is dominated by increasing missed detections, which fundamentally arises from reduced cross-domain stability of the detector: object-background and inter-instance relations become less stable in the encoding stage, while semantic-spatial alignment of query representations also becomes harder to maintain in the decoding stage. To this end, we propose VFM$<sup>{4}$SDG, a dual-prior learning framework for SDGOD, which introduces a frozen vision foundation model (VFM) as a transferable cross-domain stability prior into detector representation learning and query modeling. In the encoding stage, we propose Cross-domain Stable Relational Prior Distillation to enhance the robustness of object-background and inter-instance relational modeling. In the decoding stage, we propose Semantic-Contextual Prior-based Query Enhancement, which injects category-level semantic prototypes and global visual context into queries to improve their semantic recognition and spatial localization stability in unseen domains. Extensive experiments show that the proposed method consistently outperforms existing SOTA methods on standard SDGOD benchmarks and two mainstream DETR-based detectors, demonstrating its effectiveness, robustness, and generality.

• The paper introduces a dual-prior framework (CSRPD & SCPQE) to stabilize encoder and decoder stages for robust object detection.
• It leverages frozen self-supervised VFMs to distill cross-domain relational priors and enhance semantic anchoring, achieving significant mAP improvements.
• Quantitative and qualitative evaluations under diverse conditions validate the method's ability to reduce false negatives and improve detection consistency.

VFM$^{4}$SDG: Leveraging Vision Foundation Model Priors for Robust Single-Domain Generalized Object Detection

Problem Formulation and Motivation

Object detectors deployed in real-world visual perception systems frequently encounter domain shifts induced by environmental and imaging conditions such as varying weather, lighting, or sensor properties. In the single-domain generalization (SDG) setting, a detector must be trained solely on labeled data from a single source domain and expected to generalize to distinctly shifted, unseen domains—without recourse to target domain supervision. Existing SDG object detection (SDGOD) methods primarily emphasize data augmentation or domain-invariant feature learning, often neglecting the mechanistic stability of detection models under domain shift. Analysis in this work reveals that under domain shift, the core failure mode is a persistent increase in false negatives, with class confusion and false positives remaining relatively stable.

Figure 1: Under increasing domain shift, false negatives dominate performance degradation, while VFM$^{4}$SDG markedly slows their growth relative to baseline methods.

This empirical observation identifies the critical need to stabilize object-background and inter-instance relational representations during encoding, as well as semantic-spatial alignment during query-based decoding in Transformer-based detectors. Rather than relying on appearance diversification alone, model robustness must be reconceived as the preservation of cross-domain relational and query stability.

VFM$^{4}$SDG: Dual-Prior Learning with Vision Foundation Models

VFM$^{4}$SDG introduces a dual-prior learning framework built atop DETR-style detectors. It explicitly injects frozen Vision Foundation Models (VFMs)—specifically, self-supervised models like DINOv3—as cross-domain stability priors in both the representation learning (encoding) and query modeling (decoding) stages.

Figure 2: The VFM$^{4}$SDG pipeline with dual-prior integration: CSRPD for encoder relational distillation and SCPQE for decoder query enhancement.

Cross-domain Stable Relational Prior Distillation (CSRPD)

In the encoding stage, VFM$^{4}$SDG enforces the consistency between the detector encoder’s object-background and inter-instance relation matrices and those extracted from the frozen VFM. This is achieved via multi-scale token-level relational distillation, wherein pairwise cosine similarities (relational matrices) are computed across spatial positions. A Smooth-$\ell_1$ loss penalizes deviations between student and VFM teacher relational matrices across all selected feature pyramid levels. This operation focuses not on aligning absolute semantics, but on transferring global and local structural object-background relationships known to be stable across diverse domains in large-scale pre-trained VFMs.

Semantic-Contextual Prior-based Query Enhancement (SCPQE)

For the query-based decoding stage, VFM$^{4}$SDG introduces two types of priors:

• Semantic Identity Guidance (SIGA): Category prototypes are extracted offline from frozen VFM object features pooled over ground truth regions in the source domain. Through a learnable projection and multi-head cross-attention, these prototypes provide robust semantic anchors to each decoder query, mitigating source-domain bias.
• Contextual Spatial Grounding (CSGA): Global dense image features from the VFM, projected into the query space, supply spatial context via additional cross-attention, improving stability of spatial localization under domain shift.

Queries, after sequential injection of semantic and contextual priors, proceed into the standard detector decoder. These mechanisms combine to yield stabilized semantic-spatial query anchoring, crucial for reliability in unseen target domains.

Quantitative and Qualitative Results

Extensive experimentation evaluates VFM$^{4}$SDG under both DINO- and Co-DETR detector frameworks on the standard SDGOD urban scenes benchmark. The method is shown to achieve consistent and strong mAP@50 improvements (DINO: +8.0%, Co-DETR: +6.6% over respective baselines), outperforming both previous SOTA data augmentation approaches and domain-invariant learning schemes by clear margins. Particularly notable are the performance margins under severe domain shift (e.g., Night-Rainy: Co-DETR baseline 18.3% → 29.0% with VFM$^{4}$SDG).

Qualitative visualizations further corroborate these findings, highlighting better foreground detection consistency and reduced missed targets under heavy fog, rain, and nighttime conditions. VFM$^{4}$0SDG exhibits improved completeness of detection, especially on small and distant objects, where baselines exhibit miss-dominated errors.

Figure 3: Qualitative detection results across domains demonstrate VFM$^{4}$1SDG’s reduction of missed and false positive detections, with stronger localization consistency.

Figure 4: Encoder feature visualization reveals that VFM$^{4}$2SDG’s distillation reduces spurious background activations and sharpens semantic focus in contrast to the baseline.

Ablation and Design Analysis

Component-level ablation evidences the cumulative and complementary nature of relational and query priors: CSRPD, SIGA, and CSGA each deliver nontrivial gains individually, but their combination yields maximum performance (e.g., Co-DETR: avg. mAP up to 50.8%). Relational distillation outperforms direct semantic KD for generalization, emphasizing the need for robust cross-domain structure over absolute feature alignment. Empirical analysis shows multi-scale (as opposed to highest-level-only) relational distillation is necessary to fully exploit structural priors. The dual-prior approach remains effective across a range of VFM architectures, demonstrating method generality.

Model complexity and inference measurement reveal a moderate parameter and runtime impact versus baseline, with performance not attributable merely to increased parameter count.

Implications and Future Directions

VFM$^{4}$3SDG’s results challenge prevailing paradigms in SDGOD by reframing generalization as a problem of mechanistic stability—specifically, the stability of relational representations and query anchoring under domain shift. Injecting frozen VFM priors enables significant robustness improvements without recourse to expanded domain supervision or brute-force data augmentation. This underscores the critical role for self-supervised representation learning at scale, not as a replacement backbone but as a provider of transferable stability priors.

Open directions include:

• Designing lightweight, real-time modules for dual-prior integration to reduce computational overhead for practical deployment.
• Exploring more structured, possibly causal, relational priors in both encoding and query modeling stages.
• Extending dual-prior strategies to other dense vision tasks (segmentation, tracking) and detection frameworks.

Figure 5: Failure cases under extreme degradation reveal remaining challenges: minuscule and occluded objects in heavy noise remain hard to detect, indicating open research problems for robust perception.

Conclusion

VFM$^{4}$4SDG establishes that cross-domain stability in detection is best achieved by explicitly preserving object-background/inter-instance relational structures and robust semantic-spatial query alignment. By treating large-scale, self-supervised VFMs as frozen sources of transferable priors, and integrating these into both the encoding and query stages, strong and consistent single-domain generalization is realized. This work sets a new direction for leveraging foundation models—not as mere backbones, but as sources of robust, mechanistically-informed inductive biases for robust visual perception under domain shift.