YOLO-UniOW: Unified Open-World Detector

• YOLO-UniOW is a unified object detection framework that simultaneously recognizes known objects and flags unknown ones using vision-language integration.
• It employs adaptive decision learning with low-rank CLIP calibration and dual detection heads to maintain high accuracy and real-time speed.
• The framework supports dynamic vocabulary expansion without incremental re-training, demonstrating state-of-the-art performance on multiple benchmarks.

YOLO-UniOW is an efficient and versatile object detection framework that unifies open-vocabulary and open-world object detection within a single architecture. It addresses the limitations of traditional closed-set and open-vocabulary methods by simultaneously recognizing known categories, detecting out-of-distribution unknowns, and supporting dynamic vocabulary expansion without incremental re-training. This model leverages innovations in adaptive decision learning and wildcard learning strategies to achieve high accuracy and real-time speed across diverse detection scenarios (Liu et al., 2024).

1. Universal Open-World Detection: Problem Formulation

Traditional object detectors, such as Faster R-CNN or YOLOv5, are constrained by a fixed training vocabulary $C_{\text{train}}$ and treat any out-of-vocabulary (OOV) object as background. Open-vocabulary detection (OVD) allows recognition of novel classes by leveraging vision–LLMs (VLMs), such as CLIP, to align text and image features via cross-modal fusion. However, OVD approaches typically predefine class names at inference and rely on computationally intensive fusion operations. Open-world object detection (OWOD) extends the paradigm by requiring the detection of unknown (unlabeled) categories and enabling incremental learning of new classes without catastrophic forgetting.

YOLO-UniOW operationalizes the Universal Open-World Detection (Uni-OWD) paradigm, defining the category space as $C = C_k \cup C_{\text{unk}}$, where $C_k$ are known categories (each assigned a text name $T_c \in V$) and $C_{\text{unk}}$ are unknown categories. The detector $D$ maps an image $I$ and vocabulary $V$ to:

• Known category localization: $$D(I,V) \rightarrow \{(b, c_k) \mid b \in B_{c_k}, c_k \in C_k\}$$
• Unknown object flagging: $$D(I, T_w) \rightarrow \{(b, \text{unknown}) \mid b \in B_{\text{unk}}\}$$
• Dynamic vocabulary expansion upon discovery, without extensive incremental re-training

2. Model Architecture and CLIP Integration

YOLO-UniOW is based on the YOLOv10 detection backbone with a PAN-style neck and features two parallel detection heads per anchor:

• One-to-One (o2o) head: predicts a single box and class per anchor
• One-to-Many (o2m) head: predicts multiple boxes per anchor and applies NMS

Each predicted region is associated with a $C = C_k \cup C_{\text{unk}}$0-dimensional region embedding $C = C_k \cup C_{\text{unk}}$1. The architecture integrates a frozen CLIP text encoder that produces embeddings $C = C_k \cup C_{\text{unk}}$2 for each class name $C = C_k \cup C_{\text{unk}}$3, including special wildcard tokens. Instead of early or mid-level fusion of image and text, the classification for each region is computed as the cosine similarity:

$C = C_k \cup C_{\text{unk}}$4

These similarity scores serve as logits over the (potentially dynamic) class vocabulary, including known classes and wildcards.

Dataflow Overview

Stage	Output	Description
YOLOv10 backbone	Feature maps	Multi-scale image features
PAN neck	Aggregated feature maps	Path aggregation for spatial context
Detection heads	Boxes, embeddings	o2o/o2m heads output boxes, objectness, and region embeddings $C = C_k \cup C_{\text{unk}}$5
CLIP text encoder	Text embeddings $C = C_k \cup C_{\text{unk}}$6	Cached for all classes, including wildcards
Cosine classifier	Scores $C = C_k \cup C_{\text{unk}}$7	Score for each region-class pairing

3. Adaptive Decision Learning

To circumvent the inference overhead associated with cross-modal fusion in OVD, YOLO-UniOW introduces Adaptive Decision Learning (ADL). ADL “calibrates” the CLIP text encoder using Low-Rank Adaptation (LoRA):

$C = C_k \cup C_{\text{unk}}$8

Here, $C = C_k \cup C_{\text{unk}}$9 is the frozen CLIP projection layer, and only the low-rank updates $C_k$0 are trained. This approach allows the CLIP text encoder to adapt its class prototypes toward those most suitable for detection, while retaining zero overhead at inference (as all $C_k$1 are precomputed).

Supervision is applied via a temperature-scaled softmax contrastive loss:

$C_k$2

where $C_k$3 is the softmax temperature.

A dual-head matching strategy incorporates both cosine similarity (semantic alignment) and box IoU (spatial alignment):

$C_k$4

with $C_k$5 and $C_k$6, using the same exponents for both detection heads.

4. Wildcard Learning for Unknown Discovery

YOLO-UniOW embeds two wildcard tokens in the CLIP vocabulary:

• $C_k$7: "object"—trains the model to recognize generic objects, not tied to specific categories.
• $C_k$8: "unknown"—learns to flag instances not aligned with any known class.

Wildcard learning uses a two-stage procedure:

1. Object pre-tuning: All training instances are labeled "object," and only the LoRA modules of the text encoder are trained.
2. Unknown fine-tuning: With all embeddings frozen except for $C_k$9, pseudo-unknown boxes are generated (using $T_c \in V$0 and known class predictions). Boxes with $T_c \in V$1 to any ground truth and $T_c \in V$2 are pseudo-labeled as unknown. $T_c \in V$3 is then fine-tuned with a binary cross-entropy loss using $T_c \in V$4 as a soft target.

After prediction, a de-duplication filter removes unknown boxes with IoU $T_c \in V$5 with any high-scoring known detection.

5. Training and Inference Protocols

• Pre-training: Conducted on Objects365 and GoldG (image–text pairs) using AdamW (lr $T_c \in V$6, weight decay 0.025), batch size 128 across 8 GPUs, LoRA rank 16, and standard YOLO augmentations.
• Open-world fine-tuning: Three epochs for $T_c \in V$7 at lr $T_c \in V$8; three epochs for $T_c \in V$9 and known class text embeddings at lr $C_{\text{unk}}$0 with weight decay zero. All other weights frozen, batch size 16 per GPU.
• Inference: All $C_{\text{unk}}$1 embeddings are cached; detection is thresholded at score $C_{\text{unk}}$2 (confident if $C_{\text{unk}}$3). Unknown boxes are deduplicated as described. For the one-to-one detection head, no NMS is necessary; inference speed reaches up to 119.3 FPS.

6. Experimental Results and Ablation Studies

On the LVIS minival dataset (1,203 classes), YOLO-UniOW-L achieves 34.6 AP (including 30.0 AP on rare categories), outperforming YOLO-Worldv2-L by +1.6 AP and with +7.7 AP$C_{\text{unk}}$4 gain, at an inference speed of 69.6 FPS. State-of-the-art performance on M-OWODB, S-OWODB, and nuScenes datasets is demonstrated, including an 82.6 U-Recall in Task 1 of M-OWODB (+16.7 over OVOW*) and substantial improvements in other tasks.

Ablations indicate:

• Omitting VL-PAN fusion does not reduce performance; the Adaptive Decision Learning strategy yields +4 AP improvements.
• LoRA calibration of the text encoder produces the largest AP gains, especially for rare categories.
• Zero-shot "object" detection outperforms oracle baselines regarding unknown discovery; full wildcard learning further boosts U-Recall to approximately 80%.

7. Limitations and Future Directions

YOLO-UniOW's unknown recall in highly cluttered environments (nu-OWODB) is in the 40–45% range, indicating sensitivity to small or camouflaged objects. Pseudo-labeling with $C_{\text{unk}}$5 may introduce noise from over-generalization. Future enhancements include better pseudo-label thresholding using spatial reasoning, joint end-to-end fine-tuning of the image encoder, continual online discovery and vocabulary management, and generalization to instance and panoptic segmentation (Liu et al., 2024).

---

YOLO-UniOW establishes a unified, computationally efficient open-world detection methodology that integrates adaptive text-visual alignment and robust unknown detection, setting new accuracy and versatility benchmarks for the field.