AlignDet: Aligning Pre-training and Fine-tuning in Object Detection (2307.11077v2)

Abstract: The paradigm of large-scale pre-training followed by downstream fine-tuning has been widely employed in various object detection algorithms. In this paper, we reveal discrepancies in data, model, and task between the pre-training and fine-tuning procedure in existing practices, which implicitly limit the detector's performance, generalization ability, and convergence speed. To this end, we propose AlignDet, a unified pre-training framework that can be adapted to various existing detectors to alleviate the discrepancies. AlignDet decouples the pre-training process into two stages, i.e., image-domain and box-domain pre-training. The image-domain pre-training optimizes the detection backbone to capture holistic visual abstraction, and box-domain pre-training learns instance-level semantics and task-aware concepts to initialize the parts out of the backbone. By incorporating the self-supervised pre-trained backbones, we can pre-train all modules for various detectors in an unsupervised paradigm. As depicted in Figure 1, extensive experiments demonstrate that AlignDet can achieve significant improvements across diverse protocols, such as detection algorithm, model backbone, data setting, and training schedule. For example, AlignDet improves FCOS by 5.3 mAP, RetinaNet by 2.1 mAP, Faster R-CNN by 3.3 mAP, and DETR by 2.3 mAP under fewer epochs.

• The paper presents a two-stage pre-training process that decouples image-domain and box-domain phases for better alignment with detection tasks.
• It mitigates data, model, and task discrepancies by pre-training all modules, ensuring the model learns both classification and regression tasks.
• Experiments demonstrate significant performance improvements, including a boost of 5.3 mAP for FCOS, highlighting its practical efficiency.

Insightful Overview of "AlignDet: Aligning Pre-training and Fine-tuning in Object Detection"

This essay provides an in-depth analysis of the paper titled "AlignDet: Aligning Pre-training and Fine-tuning in Object Detection" by Ming Li, Jie Wu, Xionghui Wang, Chen Chen, Jie Qin, Xuefeng Xiao, Rui Wang, Min Zheng, and Xin Pan. The paper introduces AlignDet, a novel pre-training framework designed to address and alleviate the inherent discrepancies between pre-training and fine-tuning stages in object detection algorithms.

Core Contributions

The authors identify three major discrepancies in the current pre-training paradigms for object detection: data, model, and task discrepancies. To mitigate these issues, they propose a two-stage pre-training approach: image-domain pre-training and box-domain pre-training. This paradigm shift ensures that the pre-trained model is better aligned with the requirements of the downstream object detection tasks.

1. Data Discrepancy: The pre-training phase typically uses object-centric datasets like ImageNet, which differ significantly from the multi-object contexts found in detection datasets such as COCO. AlignDet introduces a multi-object pre-training phase to bridge this gap.
2. Model Discrepancy: Traditional approaches often focus on partial model components during the pre-training phase, leaving critical modules such as the regression head and RPN with random initialization. AlignDet ensures comprehensive pre-training across all model components.
3. Task Discrepancy: Existing pre-training relies heavily on classification tasks, neglecting the spatial and positional information crucial for object detection. AlignDet incorporates both classification and regression tasks during pre-training, ensuring the model learns relevant tasks for object detection explicitly.

Methodology

AlignDet's methodology focuses on decoupling the pre-training process into two distinct stages:

1. Image-domain Pre-training: This stage optimizes the detection backbone to capture high-level semantic abstractions using either supervised or self-supervised learning on large-scale image classification datasets. This step aligns the backbone's capabilities with general visual feature extraction.
2. Box-domain Pre-training: The second stage focuses on learning instance-level semantics and task-aware concepts. It leverages unsupervised proposals generated via selective search and augments them to generate multiple views. By employing a combination of contrastive learning and coordinate-based regression losses, this stage pre-trains all modules within the detection architecture, thereby ensuring better alignment with the final detection task.

Experimental Results

The paper provides extensive empirical evidence to validate the effectiveness of AlignDet. The framework demonstrates robust performance improvements across diverse object detection models, backbones, and data configurations. For example, the application of AlignDet results in a performance boost of 5.3 mAP for FCOS, 2.1 mAP for RetinaNet, 3.3 mAP for Faster R-CNN, and 2.3 mAP for DETR, all under fewer training epochs. Such improvements underscore the framework's ability to enhance convergence speed and generalization ability effectively.

Implications

The results indicate substantial potential for practical and theoretical advancements in object detection. On the practical front, AlignDet's ability to enhance performance with fewer epochs and across various models and backbones makes it an attractive option for real-world applications where computational efficiency and accuracy are paramount. Theoretically, AlignDet contributes to a more nuanced understanding of the alignment between pre-training and fine-tuning stages, challenging the traditional paradigms and prompting a reevaluation of how pre-training should be conducted to maximize downstream performance.

Future Speculations

The framework's success opens several avenues for future research in AI and computer vision. One potential direction is exploring the application of AlignDet to other vision tasks and models, such as segmentation and pose estimation. Additionally, the principles underlying AlignDet could be extended to address discrepancies in other types of machine learning tasks beyond vision, potentially leading to advancements in natural language processing and multi-modal learning.

Conclusion

AlignDet represents a significant advancement in the field of object detection, providing a comprehensive solution to the discrepancies that have traditionally limited the effectiveness of pre-training. Through its innovative two-stage pre-training process, AlignDet ensures that models are better aligned with the requirements of downstream detection tasks, leading to substantial improvements in performance and efficiency. By addressing the core discrepancies in data, models, and tasks, AlignDet stands out as a pivotal contribution to the ongoing evolution of object detection methodologies.