Fast Convergence of DETR with Spatially Modulated Co-Attention (2101.07448v1)

Abstract: The recently proposed Detection Transformer (DETR) model successfully applies Transformer to objects detection and achieves comparable performance with two-stage object detection frameworks, such as Faster-RCNN. However, DETR suffers from its slow convergence. Training DETR \cite{carion2020end} from scratch needs 500 epochs to achieve a high accuracy. To accelerate its convergence, we propose a simple yet effective scheme for improving the DETR framework, namely Spatially Modulated Co-Attention (SMCA) mechanism. The core idea of SMCA is to conduct regression-aware co-attention in DETR by constraining co-attention responses to be high near initially estimated bounding box locations. Our proposed SMCA increases DETR's convergence speed by replacing the original co-attention mechanism in the decoder while keeping other operations in DETR unchanged. Furthermore, by integrating multi-head and scale-selection attention designs into SMCA, our fully-fledged SMCA can achieve better performance compared to DETR with a dilated convolution-based backbone (45.6 mAP at 108 epochs vs. 43.3 mAP at 500 epochs). We perform extensive ablation studies on COCO dataset to validate the effectiveness of the proposed SMCA.

Essay on "Fast Convergence of DETR with Spatially Modulated Co-Attention"

This paper introduces an enhancement to the Detection Transformer (DETR) framework, focusing on improving its convergence speed. The authors propose a novel module, the Spatially Modulated Co-Attention (SMCA), which replaces the original co-attention mechanism within DETR's decoder. This substitution facilitates faster convergence while guaranteeing accuracy, primarily through a regression-aware approach that spatially modulates the attention responses.

The core principle of SMCA is centered around dynamically predicted Gaussian-like weight maps that are used to enhance co-attention features. These weight maps are derived from initial estimates of the bounding box location for each object query and serve to emphasize informational clustering around these predictions, thereby confining the search area spatially. As a result, SMCA accelerates DETR's convergence significantly, as it does not require extended epochs to adequately refine co-attention areas.

A notable numerical outcome from this research is that SMCA achieves a performance of 45.6 mAP at only 108 epochs, compared to DETR's 43.3 mAP at 500 epochs. This is accomplished through augmentations such as multi-head attention and multi-scale feature encodings, which contribute to more precise processing with fewer iterations. The integration of multi-scale features, drawn from different convolution layers, and the scale-selection attention mechanism further enhance SMCA's competency in handling objects across varying sizes and scales.

The implications of this research are manifold. Practically, the optimized convergence reduces computation costs and expedites training cycles, making DETR-based object detection more accessible for application development and iterative research. Theoretically, this work reveals potential in manipulating spatial priors to refine attention mechanisms in vision transformers. Furthermore, it underscores the utility of combining global information handling within self-attention layers with tailored, local optimization, potentially influencing future directions in AI development that seek efficient learning with limited computational resources.

As a future direction, it is conceivable that SMCA's principles could be adapted beyond object detection to other fields where attention mechanisms play a pivotal role. The integration of global and local data interpretations could be particularly beneficial in areas requiring fast and robust learning, such as video analysis and natural language processing. As research advances, exploring adaptive mechanisms for co-attention that leverage both spatial and semantic features will likely emerge as a critical theme in enhancing transformer models' efficiency and versatility.