DINO-YOLO: Hybrid Transformer-Enhanced YOLO

• DINO-YOLO is a hybrid object detection framework that combines YOLO’s speed with DINOv3’s rich, self-supervised features to excel in data-scarce environments.
• It utilizes dual-point integration—injecting transformer features at the input stage and mid-backbone—to enhance semantic context in object detection.
• Empirical evaluations on civil engineering datasets reveal significant accuracy improvements and real-time inference capabilities even with limited annotated data.

DINO-YOLO is a hybrid object detection architecture that integrates the data efficiency and semantic richness of self-supervised Vision Transformers, specifically DINOv3, with the speed and localization strengths of modern convolutional YOLO (You Only Look Once) frameworks. It is designed to address detection tasks in civil engineering domains where large-scale annotated datasets are often unavailable, offering state-of-the-art performance on datasets with fewer than 10,000 images while maintaining real-time inference speeds on commodity and field-deployable hardware (P et al., 29 Oct 2025).

1. Architectural Overview

DINO-YOLO leverages a twofold approach: the fast, anchor-free detection pipeline of YOLOv12, and the semantically grounded, augmentation-invariant features from large-scale, self-supervised DINOv3 Vision Transformers. The architecture introduces frozen DINOv3 feature modules at two critical junctures:

• P0 Input Preprocessing: The input RGB image undergoes transformation via DINOv3's patch-embedding pipeline, generating 768-dimensional semantically rich features which are projected and upsampled back to image resolution (640×640), forming the initial input for YOLO's first convolutional stage.
• P3 Mid-Backbone Enhancement: At the 80×80 spatial resolution with 512 channels in the YOLOv12 backbone, features are projected and tokenized, processed by a frozen DINOv3 transformer block, and then re-fused with the YOLO activations via a gated residual connection.

YOLOv12 provides the overall object detection workflow:

• Input: 640×640 RGB images
• Backbone: Downsampling with conv/C3 modules from 160×160 to 20×20
• Neck: Feature Pyramid Network (FPN) for multi-scale aggregation
• Detection Head: Anchor-free, single feed-forward scale-specific heads (P3/8, P4/16, P5/32)
• Model footprint: ~220M total parameters (47M trainable), 135 GFLOPs at 640² resolution

DINOv3, pretrained on 1.7B unlabeled images using the DINO self-distillation paradigm, provides feature representations robust to augmentation, varying in size from ViT-B/16 (86M params) to ViT-L/16 (307M).

2. Dual-Point Feature Integration

As the core innovation, DINO-YOLO injects self-supervised transformer features at two points:

P0: Semantic Input Preprocessor

The initial input transformation, $f_\text{input}(x)$, is replaced by a learnable projection of DINOv3 features:

$$f_\text{input}(x) = T_\text{proj}(\text{DINOv3}(x))$$

Where DINOv3 produces patch embeddings $\in \mathbb{R}^{H \times W \times 768}$, and $T_\text{proj}$ maps to $\mathbb{R}^{3 \times 640 \times 640}$. This results in semantically contextualized edge and texture features for downstream convolutional layers.

P3: Mid-Backbone Semantic Fusion

At the output of YOLOv12's fifth layer, the feature map $f_3 \in \mathbb{R}^{512 \times 80 \times 80}$ is projected and tokenized:

1. Project $f_3$: $512 \rightarrow 768$ channels
2. Flatten and feed tokens into a frozen DINOv3 transformer block
3. Reshape outputs and perform gated residual fusion
4. Project fused features back: $768 \rightarrow 512$ channels

This ensures that the regions of the backbone most influential for object detection heads are augmented with global self-attention semantics derived from DINOv3.

3. Loss Functions and Inference Performance

DINO-YOLO adopts the standard YOLOv12 detection loss, a weighted combination of:

• $\mathcal{L}_\text{loc}$: Localization loss via CIoU or GIoU between predicted and ground-truth boxes
• $$f_\text{input}(x) = T_\text{proj}(\text{DINOv3}(x))$$0: Binary cross-entropy for objectness
• $$f_\text{input}(x) = T_\text{proj}(\text{DINOv3}(x))$$1: Cross-entropy for class assignment

$$f_\text{input}(x) = T_\text{proj}(\text{DINOv3}(x))$$2

Main evaluation metrics include mAP@0.5 (mean average precision at IoU threshold 0.5). The inference cost is described as:

$$f_\text{input}(x) = T_\text{proj}(\text{DINOv3}(x))$$3

Despite 2–4× increased latency relative to baseline YOLOv12, DINO-YOLO maintains real-time throughput (30–47 FPS) on an RTX 5090 GPU, with mixed precision and TensorRT acceleration supporting further improvements.

4. Empirical Validation and Ablation Studies

Extensive validation on civil engineering datasets with varying annotation scales demonstrates significant accuracy improvements:

Dataset	YOLOv12 Baseline	DINO-YOLO Best	Absolute Gain	FPS (DINO-YOLO)
Tunnel Segment Crack (648)	48.31% (L)	54.28% (L-ViT-B, Dual)	+12.4%	~47
Construction PPE (1K)	49.04% (M)	55.77% (M-ViT-L, Dual)	+13.7%	~36
KITTI (7K)	41.54% (S)	72.06% (L-ViT-B, Dual)	+73.5%	~30

Ablation across five YOLO scales and nine DINOv3 variants establishes:

• Medium-scale YOLOv12 (M) with DualP0P3 integration of ViT-L/16 achieves 55.77% [email protected].
• Small-scale YOLOv12 (S) requires Triple Integration (P0P3P4) with ViT-B/16 to reach 53.63%, but is overwhelmed by larger ViT backbones.
• Large-scale YOLOv12 (L) is best served by DualP0P3 with ViT-B/16 (53.08%, +16.9% over baseline); triple integration significantly degrades accuracy.
• Dual-point injection (P0 and P3) generally outperforms single-point strategies unless backbone capacity or DINO variant is mismatched.

No single universal integration strategy exists; optimal configurations depend on precise matching of YOLO backbone capacity and DINOv3 size.

5. Deployment in Civil Engineering Applications

DINO-YOLO is explicitly engineered for data-constrained field environments:

• Sustains 30–47 FPS on the NVIDIA RTX 5090 with an 8–12 GB peak VRAM footprint
• Supports infrastructure cost reduction: a single server can monitor 15–20 camera streams, compared to 5–7 for heavier models
• FP16 and TensorRT deployment can further increase throughput to ≥60 FPS
• Jetson AGX Orin edge nodes can run small/medium DualP0P3 setups at 10–15 FPS for mobile or distributed sensing scenarios

Application-specific use-cases validated include:

• Tunnel crack inspection with conveyor-belt camera arrays
• Construction PPE monitoring for worker safety compliance with multi-stream dashboards
• Autonomous vehicle navigation in construction yards across 8,000+ categories

6. Connections to Related Work

DINO-YOLO advances hybrid detection ideas alluded to in comparative studies of Transformercore and YOLO-based detectors. Previous work demonstrates that pure YOLO architectures excel in orientation-aware detection on visually complex domains, while Transformers offer global context advantageous for structured layouts (Aguilar, 25 Jun 2025). DINO-YOLO distinguishes itself by not unifying detection heads or language grounding, but by strategic, frozen injection of pretrained transformer features at points where convolutional backbones are subject to underfitting due to limited labels. This approach avoids overfitting, preserves inference speed, and leverages both local and global inductive biases in detection pipelines.

7. Implications, Limitations, and Prospective Directions

The empirical evidence shows that injecting frozen, self-supervised DINOv3 features at the input (P0) and mid-backbone (P3) stages of YOLOv12 significantly improves sample efficiency and generalization, particularly for sub-10,000 image civil engineering datasets that are otherwise prone to overfitting or low recall with standard detectors. The 2–4× inference latency overhead remains manageable given modern GPU capabilities, establishing a practical tradeoff for real-world field deployment (P et al., 29 Oct 2025).

A plausible implication is that similar dual-point transformer feature integration could be beneficial in other high-variance, data-constrained object detection settings, provided that backbone-transformer capacity is balanced. However, generalization to domains with abundant annotated data, to ultra-low-latency or mobile-only applications, or to settings requiring open-vocabulary detection and oriented bounding box regression (OBB) as in document layout analysis remains to be systematically explored.

For civil engineering, DINO-YOLO represents a deployable, state-of-the-art detection framework that balances data efficiency, accuracy, and operational speed in resource-constrained environments.