YOLO-UniOW: Open-world & Uniform Object Detection

• The paper outlines two YOLO-UniOW variants that address uniform object detection with orientation awareness and open-world detection via CLIP-based alignment.
• The model employs a dual-head design, adapting YOLO architecture for efficient grid-based prediction and low-rank adapter modules for image and text similarity.
• Innovative wildcard learning and directed IoU enhance unknown object detection and optimize performance on benchmarks like LVIS and nuScenes.

YOLO-UniOW is a family of object detection models addressing challenges in open-world and open-vocabulary detection while maintaining high efficiency and speed. The term encompasses two distinct research lines: (1) YOLO-UniOW (YUDO), an architecture for uniform directed object detection with orientation awareness; (2) YOLO-UniOW, a universal open-world detector that integrates efficient CLIP-based region–text alignment and wildcard learning for out-of-distribution awareness. Both leverage the real-time strengths of YOLO architectures, applying specialized adaptations for their respective problem domains.

1. Model Architecture and Variants

YOLO-UniOW encompasses two model architectures documented in recent literature:

• YOLO-UniOW (YUDO) (Nedeljković, 2023): This model is tailored for tasks requiring detection of uniform-size, oriented objects, such as honeybees. Its base is YOLOv7-tiny (~6M parameters, 13.1 GFLOPS), employing a single, anchor-free detection head. The per-grid cell output is a vector: $(x, y, \theta, obj, abd\_cls, bee\_cls)$ where $(x, y)$ are center offsets, $\theta$ the orientation, and object/class scores are independent logistic probabilities. Object width and height are omitted due to uniformity assumptions.
• YOLO-UniOW (Universal Open-World Detector) (Liu et al., 2024):
  • YOLOv10 backbone and neck for image feature extraction.
  • Dual detection heads (one-to-one for boxes/objectness, one-to-many for class scores).
  • An Adaptive Decision Learning (AdaDL) module leveraging CLIP’s latent space with low-rank adapters for efficient region–text alignment.
  • A wildcard learning mechanism that enables unknown object detection and dynamic vocabulary expansion without replay.

2. Output Parameterization and Head Design

YOLO-UniOW (YUDO)

• Grid and Output:

16×16 grid, single detection head, per-cell prediction for position and direction.

• Coordinate Handling:

$$\hat{x}_i = \sigma(t_{x,i}) + c_x,\quad \hat{y}_i = \sigma(t_{y,i}) + c_y$$

Center-offsets normalized; $(c_x, c_y)$ denotes cell position. $\theta$ is predicted via ReLU and taken modulo $2\pi$.

• Fixed-box Assumption:

Object shapes are parameterized as $(\hat{x}_i, \hat{y}_i, \theta_i)$ and assigned a fixed size (e.g., 40×70 px for bees).

YOLO-UniOW (Universal)

• Region Embeddings:

The YOLOv10 backbone produces region embeddings $u_i \in \mathbb{R}^d$, boxes $b_i$, and objectness scores $(x, y)$0.

• Dual Head Output:
  • One-to-one: $(x, y)$1 detection boxes and objectness values.
  • One-to-many: $(x, y)$2 class scores, where $(x, y)$3 is the vocabulary plus “object” and “unknown”.

3. Region–Text Alignment and Classification

YOLO-UniOW (Universal) introduces an efficient CLIP-based alignment approach via Adaptive Decision Learning (AdaDL):

• Low-Rank Adaptation:

Frozen CLIP text encoder’s projections are injected with trainable low-rank adapters:

$(x, y)$4

This enables text embeddings $(x, y)$5 to capture image-contextual variations.

• Region–Text Similarity:

Cosine similarity is used:

$(x, y)$6

Final matching scores for classification/localization are:

$(x, y)$7

• Inference Efficiency:

All $(x, y)$8 are precomputed and region–text alignment reduces to a single $(x, y)$9 matrix multiplication. No fusion layers are needed in the vision network.

4. Handling Unknowns: Wildcard Learning

Open-world detection requires explicit handling of unknown object categories:

• Special Wildcards:
  • $\theta$0: Represents generic “object” through self-supervised tuning on pretraining boxes.
  • $\theta$1: Represents “unknown,” learned via pseudo-labeling and fine-tuning.
• Pseudo-labeling for Unknowns:

Candidate unknown boxes are generated by applying detection with $\theta$2, selecting boxes by score and IoU with known classes ($\theta$3).

• Unknown Filtering:

During inference, unknown-labeled boxes are suppressed if IoU with any confident known-class box exceeds $\theta$4.

• Vocabulary Expansion:

Once $\theta$5 is trained, new classes can be added to $\theta$6 without exemplar replay, enabling dynamic adaptation.

5. Loss Functions and Training Procedures

YOLO-UniOW (YUDO)

• Composite Loss:

$\theta$7

with $\theta$8 (MSE on $\theta$9), $$\hat{x}_i = \sigma(t_{x,i}) + c_x,\quad \hat{y}_i = \sigma(t_{y,i}) + c_y$$0 (cosine distance: $$\hat{x}_i = \sigma(t_{x,i}) + c_x,\quad \hat{y}_i = \sigma(t_{y,i}) + c_y$$1), and $$\hat{x}_i = \sigma(t_{x,i}) + c_x,\quad \hat{y}_i = \sigma(t_{y,i}) + c_y$$2 (BCE per YOLOv7). Weights: $$\hat{x}_i = \sigma(t_{x,i}) + c_x,\quad \hat{y}_i = \sigma(t_{y,i}) + c_y$$3.

• Training Data:

Honeybee Segmentation & Tracking datasets; 13,908 training, 1,392 validation images (512×512). SGD with EMA, one-cycle lr schedule.

YOLO-UniOW (Universal)

• Pretraining:

On Objects365 + GoldG, with backbone/necks frozen and LoRA (rank $$\hat{x}_i = \sigma(t_{x,i}) + c_x,\quad \hat{y}_i = \sigma(t_{y,i}) + c_y$$4) in CLIP encoder; batch size 128, lr $$\hat{x}_i = \sigma(t_{x,i}) + c_x,\quad \hat{y}_i = \sigma(t_{y,i}) + c_y$$5.

• Fine-tuning:

Includes pseudo-labeling for “unknown," class and object BCE, and a contrastive loss:

$$\hat{x}_i = \sigma(t_{x,i}) + c_x,\quad \hat{y}_i = \sigma(t_{y,i}) + c_y$$6

• Unified Loss:

Incorporates box regression ($$\hat{x}_i = \sigma(t_{x,i}) + c_x,\quad \hat{y}_i = \sigma(t_{y,i}) + c_y$$7: GIoU), objectness, classification, and contrastive objectives.

6. Directed IoU and Evaluation Metrics

YOLO-UniOW (YUDO)

• Directed IoU (DirIoU):

Enhances rotated IoU by penalizing head–tail flips:

$$\hat{x}_i = \sigma(t_{x,i}) + c_x,\quad \hat{y}_i = \sigma(t_{y,i}) + c_y$$8

$$\hat{x}_i = \sigma(t_{x,i}) + c_x,\quad \hat{y}_i = \sigma(t_{y,i}) + c_y$$9. Used for both mAP matching and NMS at threshold $(c_x, c_y)$0.

YOLO-UniOW (Universal)

• Standard IoU-based metrics:

Evaluation follows AP/AP$(c_x, c_y)$1 on LVIS, U-Recall and known class mAP on M-OWODB/S-OWODB, and nuScenes benchmarks, always reporting inference FPS and parameter count.

7. Empirical Results and Comparative Analysis

Model	Params (M)	Main AP / AP$(c_x, c_y)$2	FPS	Notable Metric
YOLO-UniOW-L (Liu et al., 2024)	29.4	34.6 / 30.0	64.8	U-Recall=82.6% M-OWODB Task4, CK mAP ≈ 20% nuScenes
YOLO-UnioW (YUDO) (Nedeljković, 2023)	~6	70.5%@30 (Bee AP@30 = 85.1%)	(not listed)	Model size: 13.1 GFLOPS, efficient single head

YOLO-UniOW (Universal) achieves LVIS AP=34.6 while running at 69.6 FPS, outperforming prior large open-vocabulary/fusion detectors such as GLIPv2-T (29.0 AP at 0.12 FPS) and Grounding DINO (33.5 AP). On M-OWODB, YOLO-UniOW-M attains U-Recall=82.6% and known Both=60.9% in incremental Task 4, surpassing OVOW. On nuScenes, U-Recall more than doubles prior state-of-the-art for unknowns.

YOLO-UniOW (YUDO) achieves mAP@30=85.1% (bee), 55.9% (abdomen), and maintains a highly compact runtime profile by removing anchors, unused heads, and omitting $(c_x, c_y)$3 regression, realizing real-time directed object detection for uniform objects.

• [YUDO: YOLO for Uniform Directed Object Detection, (Nedeljković, 2023)]
• [YOLO-UniOW: Efficient Universal Open-World Object Detection, (Liu et al., 2024)]