DINO-X Pro: Unified Object-Centric Vision

• DINO-X Pro is a unified object-centric vision model defined by its integration of an enhanced Transformer architecture, multimodal prompt tuning, and large-scale pre-training for open-world detection.
• It employs multiple perception heads—including detection, segmentation, keypoint estimation, and language tasks—to deliver state-of-the-art zero-shot performance on varied benchmarks.
• The model leverages the extensive Grounding-100M dataset to achieve robust rare-category recognition and efficiently handle long-tailed object distributions in diverse domains.

DINO-X Pro is a unified object-centric vision model developed by IDEA Research, designed for open-world object detection and understanding. It extends the Transformer-based encoder–decoder architecture of Grounding DINO 1.5 with significant enhancements in both architectural components and training data scale. DINO-X Pro pursues a foundational object-level representation enabling detection, segmentation, pose estimation, captioning, and question answering in open-vocabulary and long-tailed scenarios. Its performance establishes new state-of-the-art (SOTA) results on key zero-shot object detection and segmentation benchmarks, particularly excelling in rare-category detection (Ren et al., 21 Nov 2024).

1. Transformer-Based Model Architecture

DINO-X Pro inherits a Transformer-based encoder–decoder backbone with a pre-trained Vision Transformer (ViT) as the visual feature extractor. Multi-scale image features are fused early via deformable attention, supporting robust object grounding across scales. Object queries are generated by a language-guided query selection module and processed through multiple Transformer decoder layers.

Architectural modifications include:

• CLIP-based text encoder: Replaces the original BERT text encoder to improve multimodal alignment and accelerate convergence.
• T-Rex2 visual prompt encoder: Incorporates support for both box and point prompts, leveraging sine-cosine positional embeddings and multi-scale deformable cross-attention.
• Customized prompt slot: Allows for domain- or function-specific vocabularies through prompt-tuning §2.1.1, eliminating the need for full-model retraining.
• Multi-head outputs: In addition to the box head (using L1 and G-IoU losses with contrastive classification), three specialized heads are attached:
  • Mask Head: Mask2Former-style pixel embeddings and per-query dot product for segmentation.
  • Keypoint Heads: ED-Pose-style decoders for human (17 keypoints) and hand (21 keypoints) pose estimation.
  • Language Head: An autoregressive module based on RoIAlign features and task tokens supports object-level captioning, recognition, OCR, and region-based QA.

2. Prompting Mechanisms and Flexibility

DINO-X Pro provides several input prompt modalities that control object grounding behavior:

• Text Prompting: User-provided noun lists or descriptive sentences are embedded by the CLIP encoder and deeply fused with visual features.
• Visual Prompting: Boxes or points interactively drawn around objects are encoded into positional signals via T-Rex2 and serve as decoder queries.
• Customized Prompting: Learned prompt embeddings, fine-tuned for specific long-tailed or specialized domains (e.g., industrial/medical detection) via prompt-tuning techniques.
• Universal Object Prompt: A special learned prompt enables prompt-free mode, where the model detects all objects in an image without user input.

This system allows open-world and domain-adaptive operation, with universal prompt facilitating fully prompt-free open-vocabulary detection.

3. Large-Scale Pre-Training with Grounding-100M

DINO-X Pro is pre-trained on Grounding-100M, a large-scale dataset exceeding 100 million web-mined images with high-quality grounding annotations. The dataset includes:

• Visual prompt pre-training samples from T-Rex2 and additional industrial contexts.
• Pseudo-mask annotations for approximately 30% of the data, generated via SAM/SAM2, supporting mask learning.
• Ten million samples labeled for recognition, captioning, OCR, and QA tasks to enhance the Language Head.

This dataset builds a foundational object-level visual representation, supporting robust open-vocabulary transfer and handling of long-tailed distributions. As a result, the model can generalize to unseen categories and low-frequency classes inherent in open-world settings.

4. Integrated Multi-Task Perception Heads

DINO-X Pro, after grounding pre-training, supports multi-task operation via additional perception heads with the encoder frozen. The model accommodates:

• Box Head: Zero-shot, open-vocabulary object bounding box detection.
• Mask Head: Single-pass, per-query mask prediction for segmentation.
• Keypoint Heads: Decoding human and hand keypoints from detection queries.
• Language Head: An autoregressive decoder for captioning, recognition, OCR, and visual question answering (region-QA).

This design enables simultaneous support for an array of object-centric understanding tasks using a shared backbone.

5. Zero-Shot and Long-Tailed Performance Benchmarks

DINO-X Pro demonstrates state-of-the-art zero-shot detection and segmentation performance on standardized public benchmarks. No COCO/LVIS images or labels are used during Stage 1 pre-training. Average Precision (AP) is computed as:

$\mathrm{AP} = \int_{0}^{1} p(r) \, \mathrm{d}r$

where $p(r)$ is precision at recall $r$.

Object Detection (Box AP)

Dataset	Overall AP	Rare-Class AP	Δ vs. Previous SOTA
COCO-val	56.0	N/A	+1.7 (G-DINO 1.5 Pro)
LVIS-minival	59.8	63.3	+2.0/+5.8 (G-DINO 1.6)
LVIS-val	52.4	56.5	+1.1/+5.0 (G-DINO 1.6)

Segmentation (Mask AP)

Dataset	Mask AP
COCO	37.9
LVIS-minival	43.8
LVIS-val	38.5

Against previous models:

• On rare-class LVIS-minival, DINO-X Pro improves by +7.2 AP (vs. G-DINO 1.5 Pro) and +5.8 AP (vs. G-DINO 1.6 Pro).
• On rare-class LVIS-val, gains are +11.9 AP (vs. G-DINO 1.5 Pro) and +5.0 AP (vs. G-DINO 1.6 Pro).
• It maintains superiority over prior open-set detectors such as GLIPv2, GLIP, and OWL-VIT on zero-shot COCO and LVIS metrics.

6. Open-World Object Detection Capabilities and Implications

The combination of large-scale grounding pre-training, modality-flexible prompting (including universal/prompt-free operation), and integrated multi-task heads enables DINO-X Pro to excel in both common and rare-class zero-shot detection without finetuning on target task data. In particular, its rare-category performance is especially favorable in long-tailed recognition, which suggests it is robust for scenarios with significant class imbalance or emergent categories.

The model’s modular, object-centric approach and scale-associated pre-training offer a unified solution for a range of open-world object understanding tasks. A plausible implication is that similar training regimes and architectures may be extensible to additional object-level vision tasks and underexplored visual domains (Ren et al., 21 Nov 2024).