Dense Distinct Query for End-to-End Object Detection (2303.12776v2)

Abstract: One-to-one label assignment in object detection has successfully obviated the need for non-maximum suppression (NMS) as postprocessing and makes the pipeline end-to-end. However, it triggers a new dilemma as the widely used sparse queries cannot guarantee a high recall, while dense queries inevitably bring more similar queries and encounter optimization difficulties. As both sparse and dense queries are problematic, then what are the expected queries in end-to-end object detection? This paper shows that the solution should be Dense Distinct Queries (DDQ). Concretely, we first lay dense queries like traditional detectors and then select distinct ones for one-to-one assignments. DDQ blends the advantages of traditional and recent end-to-end detectors and significantly improves the performance of various detectors including FCN, R-CNN, and DETRs. Most impressively, DDQ-DETR achieves 52.1 AP on MS-COCO dataset within 12 epochs using a ResNet-50 backbone, outperforming all existing detectors in the same setting. DDQ also shares the benefit of end-to-end detectors in crowded scenes and achieves 93.8 AP on CrowdHuman. We hope DDQ can inspire researchers to consider the complementarity between traditional methods and end-to-end detectors. The source code can be found at \url{https://github.com/jshilong/DDQ}.

• The paper introduces the DDQ method to balance query density and distinctiveness in end-to-end object detection.
• It demonstrates improved performance with 52.1 AP on COCO and enhanced recall in crowded scenes using DDQ-DETR.
• The method employs pyramid shuffle, class-agnostic NMS, and auxiliary loss to stabilize and optimize training.

Analysis of Dense Distinct Queries for End-to-End Object Detection

This paper presents a nuanced approach to object detection by introducing the concept of Dense Distinct Queries (DDQ) for end-to-end object detection models. The authors aim to address the limitations of sparse and dense queries in one-to-one label assignments by proposing a novel query formulation that balances the need for both density and distinctiveness.

Context and Methodology

Object detection models strive to detect and delineate objects in images accurately. Traditional methods rely on dense queries, leading to redundant predictions that necessitate non-maximum suppression (NMS). End-to-end detectors, such as DETR, attempt to simplify this pipeline by using sparse queries and one-to-one label assignments, eliminating NMS but introducing new challenges, such as low recall due to sparse queries and optimization difficulties.

The paper proposes Dense Distinct Queries (DDQ) as a solution. DDQ integrates dense query distribution typical of traditional detectors with the distinctiveness required for effective optimization in end-to-end models. This approach effectively enhances recall while maintaining query distinctiveness, addressing the inherent challenges faced by both traditional and more recent models.

Key Results

The introduction of DDQ improved the performance across multiple detector architectures, including FCN, R-CNN, and DETRs:

• DDQ-DETR achieved 52.1 AP on the MS-COCO dataset within 12 epochs using a ResNet-50 backbone, outperforming existing models.
• On the CrowdHuman dataset, DDQ improved object detection performance in crowded scenes, achieving 93.8 AP and significantly enhancing recall (98.7).
• The paper also demonstrates the robustness of the DDQ approach by validating it across distinct model architectures and datasets.

Technical Evaluation

The authors employ several refinements to implement DDQ effectively:

• Pyramid Shuffle Operation for FCN structures to ensure cross-level interaction among dense queries, improving training stability and performance.
• The application of class-agnostic NMS across both training and inference to ensure distinctiveness in selected queries, setting it apart from traditional NMS.
• Inclusion of auxiliary loss for dense queries to harness the potential of the filtered queries and stabilize training.

Implications and Future Work

The results suggest that DDQ effectively blends the benefits of traditional grid-based query systems with the novel end-to-end paradigms established by models like DETR. The concept of maintaining both density and distinctiveness in queries could inspire future explorations in object detection and broader AI application domains. Future work might further explore optimizing the balance between query density and training efficiency or extend the DDQ framework to other kinds of tasks, such as segmentation or tracking.

Conclusion

The proposed Dense Distinct Queries (DDQ) offer a promising direction for advancing object detection models by reconciling the conflicting needs of query density and optimization distinctiveness. This paper not only enhances understanding of query dynamics in object detection but also sets the stage for developing even more efficient and effective generative models in the computer vision domain.