Center-Aware Detection with Swin-based Co-DETR Framework for Cervical Cytology

Abstract: Automated analysis of Pap smear images is critical for cervical cancer screening but remains challenging due to dense cell distribution and complex morphology. In this paper, we present our winning solution for the RIVA Cervical Cytology Challenge, achieving 1st place in Track B and 2nd place in Track A. Our approach leverages a powerful baseline, integrating the Co-DINO framework with a Swin-Large backbone for robust multi-scale feature extraction. To address the dataset's unique fixed-size bounding box annotations, we formulate the detection task as a center-point prediction problem. Tailoring our approach to this formulation, we introduce a center-preserving data augmentation strategy and an analytical geometric box optimization to effectively absorb localization jitter. Finally, we apply track-specific loss tuning to adapt the loss weights for each task. Experiments demonstrate that our targeted optimizations improve detection performance, providing an effective pipeline for cytology image analysis. Our code is available at https://github.com/YanKong0408/Center-DETR.

• The paper reformulates detection as a center-point localization task to align with fixed-size bounding box constraints in cervical cytology.
• It leverages a Swin-Large transformer and center-preserving augmentation to improve cell localization and reduce noisy supervision.
• Analytical geometric box optimization and track-specific loss reweighting yield significant mAP gains over traditional methods.

Center-Aware Detection with Swin-based Co-DETR for Cervical Cytology: A Technical Review

Introduction and Problem Context

Automated analysis of Pap smear images underpins scalable, reproducible cervical cancer screening yet remains constrained by the complexity of dense cellular arrangements, small object sizes, and morphological variability. The RIVA Cervical Cytology Challenge provides a benchmark dataset with the added intricacy of fixed-size bounding boxes, imposing unique constraints on detection frameworks. The paper proposes a cell localization and classification pipeline based on a center-aware adaptation of the Co-DINO framework with a Swin-Large Transformer backbone, achieving top rankings in both the localization-only and joint detection-classification challenge tracks.

Center-Aware Co-DINO with Swin-Large: Framework and Innovations

The core detection architecture couples Co-DINO, a recent DETR variant with collaborative hybrid assignments, and a hierarchical feature extractor based on the Swin-Large backbone. The Swin Transformer is selected over pure ViT architectures for its localized attention mechanisms, which are better suited for resolving tightly-packed, morphologically subtle cell instances common in Pap cytology.

Figure 1: Overview of the proposed center-aware Co-DINO framework.

The pipeline predicts cell centers and, for Track A, class probabilities across the eight Bethesda categories plus background. Rather than traditional bounding box regression, the detection task is reformulated as a center-point classification problem—an adjustment motivated by the dataset’s rigid $100 \times 100$ pixel annotation constraint. This center-aware approach directly aligns the detection head’s responsibility (center localization) with the fixed evaluation geometry of the challenge.

Methodological Contributions

Center-Preserving Data Augmentation

Typical random crop augmentations risk truncating informative cell structure, impeding downstream morphological analysis vital for accurate categorization. The proposed center-preserving augmentation discards objects whose centers fall outside randomized crops, ensuring all retained training cells preserve meaningful diagnostic morphology. The effect is a substantial reduction in noisy supervision, leading to more reliable convergence.

Analytical Geometric Bounding Box Optimization

With the evaluation metric penalizing even minor center-point shifts due to the fixed annotation size, the authors derive an analytical solution for box dimension post-processing. By setting predicted box sizes to $101.5 \times 101.5$ pixels—slightly larger than the $100 \times 100$ ground-truth boxes—the approach creates a spatial buffer (of $0.75$ pixels per side) that statistically maximizes the expected IoU in the presence of localization jitter. This increases metric robustness without architectural modification and is shown to consistently yield performance gains over a range of detectors.

Track-Specific Loss Reweighting

Given the orthogonal priorities of the challenge tracks (classification-centric vs. localization-centric), the overall loss is tuned on a per-track basis. For Track A, classification loss is prioritized; for Track B, L1 loss on the center forecast is emphasized. The main loss is further supplemented by auxiliary losses contributed by multiple dense detection heads during training, including Faster R-CNN, ATSS, RetinaNet, and FCOS, providing improved gradients for difficult training regimes. During inference, only the main Co-DINO head is active.

Experimental Results

The proposed architecture demonstrates improved mAP in both challenge tracks relative to canonical detectors (YOLO, RetinaNet, CenterNet) and transformer-based baselines (Co-DINO with ViT). Swin-Large consistently yields better localization, particularly for the detection-only track, with mAP rising from 0.604 (Co-DINO-ViT) to 0.609 (Co-DINO-Swin) in Track B. Ablation studies confirm cumulative gains from each innovation: center-preserving crops, analytical geometric box optimization, and track-specific loss interplay.

A notable result is the universal performance boost brought by analytical geometric box resizing, as reflected in increased mAP across all evaluated baselines, underscoring the method’s generality and extendibility for detection tasks with rigid annotation geometries.

Theoretical and Practical Implications

This work emphasizes that in medical detection, the interplay between annotation protocol and detector architecture can create pronounced domain-specific bottlenecks. Reformulating detection as a center-point localization task, rather than direct box regression, both aligns with constrained supervision and leverages the strengths of modern transformer-based decoding regimes. The auxiliary-branch training schema favors rapid convergence and improved generalization, particularly in dense observation environments found in medical imaging.

The analytical post-processing approach for bounding box resizing offers a model-agnostic technique applicable in other scenarios involving fixed-size or rigid evaluation geometries. This is of practical value for medical imaging pipelines where the annotation protocol is driven by clinical requirements or legacy standards.

Future Directions

Potential avenues of extension include deeper integration of cytological priors (e.g., explicit modeling of nucleus-cytoplasm relationships) to further improve classification, as well as leveraging unsupervised or semi-supervised approaches to accommodate the limited manually annotated data typical in clinical datasets. Another promising direction is the formal quantification of uncertainty in center predictions to inform active learning or downstream cell-tracking applications.

Conclusion

The paper presents a high-performing, center-aware transformer detection framework for cervical cytology, demonstrating the effectiveness of task-driven architectural and post-processing adaptations in the context of medically motivated annotation constraints. Future research will likely focus on richer class discriminators and further domain-informed optimization strategies.