Grounding DINO 1.5: Advance the "Edge" of Open-Set Object Detection (2405.10300v2)

Abstract: This paper introduces Grounding DINO 1.5, a suite of advanced open-set object detection models developed by IDEA Research, which aims to advance the "Edge" of open-set object detection. The suite encompasses two models: Grounding DINO 1.5 Pro, a high-performance model designed for stronger generalization capability across a wide range of scenarios, and Grounding DINO 1.5 Edge, an efficient model optimized for faster speed demanded in many applications requiring edge deployment. The Grounding DINO 1.5 Pro model advances its predecessor by scaling up the model architecture, integrating an enhanced vision backbone, and expanding the training dataset to over 20 million images with grounding annotations, thereby achieving a richer semantic understanding. The Grounding DINO 1.5 Edge model, while designed for efficiency with reduced feature scales, maintains robust detection capabilities by being trained on the same comprehensive dataset. Empirical results demonstrate the effectiveness of Grounding DINO 1.5, with the Grounding DINO 1.5 Pro model attaining a 54.3 AP on the COCO detection benchmark and a 55.7 AP on the LVIS-minival zero-shot transfer benchmark, setting new records for open-set object detection. Furthermore, the Grounding DINO 1.5 Edge model, when optimized with TensorRT, achieves a speed of 75.2 FPS while attaining a zero-shot performance of 36.2 AP on the LVIS-minival benchmark, making it more suitable for edge computing scenarios. Model examples and demos with API will be released at https://github.com/IDEA-Research/Grounding-DINO-1.5-API

• The paper demonstrates that Grounding DINO 1.5 Pro sets new accuracy records (54.3 AP on COCO and 55.7 AP on LVIS-minival) through a larger vision backbone and deep early fusion.
• The paper shows that Grounding DINO 1.5 Edge achieves real-time detection at 75.2 FPS while maintaining robust zero-shot performance on edge devices.
• The paper emphasizes scalability and generalization by training on over 20 million images and optimizing deployment for diverse and unpredictable environments.

Grounding DINO 1.5: Advancements in Open-Set Object Detection

What is Grounding DINO 1.5?

Grounding DINO 1.5 is a powerful suite of models designed by IDEA Research for open-set object detection. It includes two main models: Grounding DINO 1.5 Pro and Grounding DINO 1.5 Edge. The Pro model focuses on improving detection performance, while the Edge model optimizes for faster inference speeds, making it suitable for use in real-time applications on edge devices.

Notable Results

• Grounding DINO 1.5 Pro
  • Achieved a 54.3 AP (Average Precision) on the COCO zero-shot transfer benchmark.
  • Set new records with a 55.7 AP on the LVIS-minival zero-shot benchmark.
• Grounding DINO 1.5 Edge
  • Remarkably, it reached a speed of 75.2 FPS with a zero-shot performance of 36.2 AP on the LVIS-minival benchmark.
  • Optimized for edge devices, demonstrating its utility in real-time applications.

Key Improvements

Grounding DINO 1.5 Pro

• Larger Vision Backbone: Uses the pre-trained ViT-L architecture for enhanced model performance.
• Deep Early Fusion: Integrates language and image features more deeply before the decoding phase, leading to higher detection recall and better bounding box precision accuracy.
• Larger Dataset: Trained on over 20 million images with grounding annotations, significantly enriching semantic understanding.

Grounding DINO 1.5 Edge

• Efficient Feature Enhancer: Limits fusion to high-level image features, reducing computational demands while maintaining robust detection capabilities.
• Optimized for Edge Deployment: Uses EfficientViT-L1 as the backbone and achieves over 10 FPS inference speed on an NVIDIA Orin NX platform.

Practical Implications

1. Detection Accuracy: The Grounding DINO 1.5 Pro model sets new benchmarks in detection accuracy, making it valuable for applications requiring high precision.
2. Real-time Applicability: The Grounding DINO 1.5 Edge model’s ability to maintain high detection performance while running efficiently on edge devices has significant implications for autonomous systems, medical imaging, and more.
3. Scalability: Training on a large, diverse dataset means these models can generalize better to unseen categories, a critical factor for applications operating in varied and unpredictable environments.

Future Directions

1. Enhanced Edge Deployment: Further research could focus on optimizing the models for even faster inference speeds and less resource-intensive deployments.
2. Broader Dataset Integration: Incorporating even more diverse datasets could further improve the model's ability to generalize across different scenarios.
3. Reducing Hallucinations: Continuing to refine early fusion techniques and sampling strategies could address the challenges of model hallucinations, enhancing the robustness and reliability of detections.

Conclusion

Grounding DINO 1.5 marks a significant step forward in the field of open-set object detection, with notable achievements both in terms of detection accuracy and real-time application feasibility. As AI continues to evolve, these advancements pave the way for more sophisticated and reliable computer vision systems that can operate seamlessly across a wide range of applications and environments.