Accelerating DETR Convergence via Semantic-Aligned Matching (2203.06883v1)

Abstract: The recently developed DEtection TRansformer (DETR) establishes a new object detection paradigm by eliminating a series of hand-crafted components. However, DETR suffers from extremely slow convergence, which increases the training cost significantly. We observe that the slow convergence is largely attributed to the complication in matching object queries with target features in different feature embedding spaces. This paper presents SAM-DETR, a Semantic-Aligned-Matching DETR that greatly accelerates DETR's convergence without sacrificing its accuracy. SAM-DETR addresses the convergence issue from two perspectives. First, it projects object queries into the same embedding space as encoded image features, where the matching can be accomplished efficiently with aligned semantics. Second, it explicitly searches salient points with the most discriminative features for semantic-aligned matching, which further speeds up the convergence and boosts detection accuracy as well. Being like a plug and play, SAM-DETR complements existing convergence solutions well yet only introduces slight computational overhead. Extensive experiments show that the proposed SAM-DETR achieves superior convergence as well as competitive detection accuracy. The implementation codes are available at https://github.com/ZhangGongjie/SAM-DETR.

Accelerating DETR Convergence via Semantic-Aligned Matching

The paper "Accelerating DETR Convergence via Semantic-Aligned Matching" tackles a significant issue inherent in the DEtection TRansformer (DETR) framework: its slow convergence rate during training, which substantially increases computational costs and limits its applicability. Despite DETR breaking away from conventional object detection models by eliminating various hand-crafted components, the convergence dilemma presents a clear technical hurdle. The authors introduce SAM-DETR (Semantic-Aligned-Matching DETR) as an effective solution to expedite DETR's convergence without adverse effects on accuracy.

Core Contributions and Methodology

SAM-DETR enhances DETR's framework through the introduction of a Semantic-Aligned Matching module, strategically placed ahead of DETR's cross-attention mechanism. This module ensures that object queries and encoded image features are projected into the same semantic space. Consequently, this alignment addresses the fundamental matching difficulties that traditionally slow DETR's learning process. The Semantic-Aligned Matching module operates by resampling object queries directly from encoded image features, informed by initially learnable reference boxes for potential object locations. This imposes a strong prior, prompting these queries to gravitate towards semantically relevant regions, thus facilitating efficient convergence.

Moreover, SAM-DETR introduces a novel mechanism of explicitly searching for salient points with discriminative features. These salient points are pivotal in improving object detection accuracy and expediting model training. The search process is seamlessly integrated with DETR's multi-head attention, complementing its capacity to focus on distinct regions within an image.

Numerical Results and Performance

Empirical evaluations underscore the efficacy of SAM-DETR. When integrated with SMCA-DETR, a notable convergence-boosting solution, SAM-DETR achieves performance on par with Faster R-CNN models, both in terms of speed of convergence and detection accuracy, within a 12-epoch training scheme. Specifically, SAM-DETR demonstrates an increase of 10.8% in average precision (AP) over baseline DETR under identical configurations and a competitive alignment with the stronger SMCA-DETR. This is particularly striking given DETR's original inefficiency compared to other methods like Faster R-CNN when trained over shorter cycles.

Theoretical and Practical Implications

Theoretically, the introduction of SAM provides a previously missing interpretative lens through which the cross-attention mechanism can be viewed: as a process of matching and feature distillation. This reframing not only elucidates the convergence challenge but offers a tangible resolution. Practically, SAM-DETR's plug-and-play nature allows for integration with existing convergence solutions, bridging distinct methodologies and pushing DETR's applicability into scenarios requiring rapid detection cycles. This integration is achieved with minimal computational overhead, ensuring SAM-DETR's deployment remains feasible in resource-constrained settings.

Future Directions

The promising results pose future exploration into multi-scale feature integration to enhance performance on smaller objects, a known limitation of the DETR model family. Further refinement of the Semantic-Aligned Matching module could also innovate towards adaptive resampling strategies, potentially allowing models to adjust to various object sizes and complexities dynamically. The method's compatibility with other architectures beyond SMCA-DETR could also be explored, broadening its applicability and the robustness of object detection transformers further.

In conclusion, the authors present a robust advancement in the field of object detection through SAM-DETR, addressing pressing computational challenges while maintaining competitive accuracy standards. This methodological innovation not only advances DETR's practical deployment but also enriches object detection transformation discourse with new theoretical insights.