YOLOE-26: Open-Vocabulary Extension

• YOLOE-26 is the open-vocabulary extension of the YOLO26 family, unifying tasks like object detection, segmentation, and classification in real-time.
• It employs a dual-head design with dense and end-to-end inference paths, eliminating NMS and DFL to reduce latency and computational load.
• Advanced training methods such as MuSGD, Progressive Loss, and STAL enhance generalization and small object handling for robust performance.

YOLOE-26 is the open-vocabulary extension of the Ultralytics YOLO26 object detection family, designed to support unified, real-time, end-to-end inference across closed-set and open-vocabulary object detection, instance segmentation, pose estimation, classification, and oriented bounding box (OBB) detection. YOLOE-26 incorporates prompt-based and prompt-free open-vocabulary capabilities while retaining the efficient, deployment-friendly characteristics of the underlying YOLO26 architecture, including dual-head NMS-free inference and DFL (Distribution Focal Loss)-free regression. The architecture and training pipeline facilitate robust generalization to novel categories via multi-modal prompting or built-in embeddings, advancing the accuracy-latency Pareto front for real-time vision applications (Jocher et al., 2 Jun 2026).

1. Architectural Foundations and Dual-Head Design

YOLOE-26 is architecturally grounded in the YOLO26 backbone and neck, which combine a convolutional feature extractor (CSP/ConvNeXt-style) with a PANet-like neck to produce detection features at multiple spatial resolutions (strides 8, 16, 32). The detection head is a dual-path, anchor-free design comprising two parallel "Task-Aligned" (TAL) heads:

• One-to-Many ("Dense") Head: Each spatial location yields multiple predictions; training uses top-$k$ assignment ($k=10$ typically) and classical NMS at inference, yielding rich supervision.
• One-to-One ("End-to-End") Head: Each instance is mapped to at most one predicted box via a two-stage top-$k$ matching procedure (topk$_1$=7, topk$_2$=1), enabling fully NMS-free, constant-time decoding at inference.

Inference by default uses the one-to-one path for deployment universality, with the dense path retained for maximal mAP when NMS latency is tolerable. This duality ensures synergy between supervised learning (dense) and efficient inference (e2e), and underpins open-vocabulary extension (Jocher et al., 2 Jun 2026).

2. Distribution Focal Loss Removal and Regression Strategies

YOLOE-26 discards DFL, which in prior YOLO variants modeled each box side as a discrete probability distribution over $K$ bins, necessitating softmax and expectation operations:

$d = \sum_{i=0}^{K-1} i \cdot \mathrm{softmax}(z)_i$

Removing DFL reduces the head parameter count (by up to 20% in small models) and removes the bounding range constraint, replacing it with direct regression of four real-valued offsets optimized by $\ell_1$ loss ("reg_max=1"). This DFL-free approach improves quantization, accelerates inference (eliminating costly exponentials/integrals), and is synergistic with edge deployment, as INT8 NPUs often lack efficient support for softmax/integral operations (Jocher et al., 2 Jun 2026).

3. Open-Vocabulary Prompting and Classification

YOLOE-26 generalizes to novel categories via unified prompting modes:

• Text-prompted (TP): A MobileCLIP2 text encoder embeds $K$ prompts as $W \in \mathbb{R}^{K\times C}$.
• Visual-prompted (VP): The SAVPE module converts reference image crops into prompt-embeddings of identical shape.
• Prompt-free (PF): The LRPCHead employs a fixed internal vocabulary with learned embeddings, requiring no prompts at inference.

A batch-normed (BN) contrastive classification head is used at each detection level:

$k=10$0

where $k=10$1 denotes inner product over channels, and $k=10$2 is passed through a sigmoid for per-prompt confidence. Training objectives include a regional contrastive classification loss and a standard regression loss on bounding boxes. Three-stage training comprises (1) TP stage on grounded corpora, (2) VP/PF fine-tuning with prompt-specific heads, and (3) decoupled segmentation training (Jocher et al., 2 Jun 2026).

4. Training Advances: MuSGD, Progressive Loss, and STAL

The YOLOE-26 training pipeline employs a coordinated suite of advances to stabilize optimization and improve small object handling:

• MuSGD ("Muon-inspired" SGD): For weight matrices, momentum is orthogonalized via Newton–Schulz polar iterations before the update step.

$k=10$3

$k=10$4

• Progressive Loss: A curriculum-style epoch-varying re-weighting of losses, shifting from the one-to-many head to the end-to-end detection head as training progresses. The weighting at epoch $k=10$5 is

$k=10$6

with a linear schedule $k=10$7 from 0.8 to 0.1.

• Small-Target-Aware Label Assignment (STAL): Guarantees positive assignment for objects smaller than the network’s minimum stride by replacing undersized box dimensions with a reference size in the assignment mask, ensuring that tiny objects are not starved of positive gradients (Jocher et al., 2 Jun 2026).

5. Multi-Task and Model Scaling

YOLOE-26 supports unified inference for detection, instance segmentation, pose estimation, classification, OBB, and open-vocabulary detection. Task-specific heads leverage shared backbone and neck features:

• Instance segmentation: Combines shared prototype tensors with per-instance coefficients.
• Pose estimation: Employs residual-based normalizing flow (RLE loss) for joint localization with uncertainty modeling.
• Oriented bounding boxes: Predicts direct angles and incorporates an auxiliary angle loss for square-like shapes.

Model scaling follows the standard YOLO width ($k=10$8) and depth ($k=10$9) multipliers, with five discrete sizes: n, s, m, l, x. For example, YOLO26x comprises 55.7M parameters and achieves 57.5 mAP on COCO at 11.8 ms T4 TensorRT FP16 latency; YOLO26n has 2.4M parameters, yielding 40.9 mAP at 1.7 ms (Jocher et al., 2 Jun 2026).

6. Performance, Deployment, and Practical Considerations

On LVIS minival, YOLOE-26x reaches 40.6 AP under text prompting, compared to 34.4 AP for DetCLIP-T. The end-to-end (one-to-one) head incurs $k$00.8 mAP drop versus the dense path with NMS. The inference path is portable across ONNX, TorchScript, TensorRT, CoreML, TFLite, OpenVINO, NCNN, and ExecuTorch, among other backends. The absence of DFL and NMS enables both minimized parameter count and reduced latency, supporting edge and mobile deployment scenarios. NMS-free decoding can fall back to the dense head when backends lack top-$k$1 primitive support (Jocher et al., 2 Jun 2026).

The unified multi-task and open-vocabulary pipeline (official code: https://github.com/ultralytics/ultralytics) allows for deployment flexibility, broad real-world applicability, and supports both prompt-driven and prompt-free generalization.

7. Context and Significance

YOLOE-26 demonstrates that dual-head, DFL-free, NMS-free object detection architectures augmented with curriculum-style supervision scheduling (Progressive Loss), MuSGD, and robust small-object assignment (STAL) can be efficiently extended to open-vocabulary vision tasks. By delivering prompt-based and prompt-free inference within a single model family, and providing deployment universality, YOLOE-26 advances the practical state-of-the-art for real-time, unified computer vision pipelines in both closed-set and open-vocabulary detection settings (Jocher et al., 2 Jun 2026).