RIVA Cervical Cytology Challenge

• The RIVA Cervical Cytology Challenge is a benchmark task for automated Pap smear analysis, focusing on dense cell distributions, fixed-size nucleus annotations, and severe class imbalance.
• It comprises two tracks—one for simultaneous localization and multi-class Bethesda classification, and another for localization only—prompting diverse adaptations in detection architectures.
• Solutions leverage center-preserving augmentations, imbalance mitigation techniques, and ensemble strategies to optimize performance metrics like mAP50–95 in challenging dense cytology imagery.

Searching arXiv for papers on the RIVA Cervical Cytology Challenge and related cervical cytology detection/classification methods. The RIVA Cervical Cytology Challenge is a benchmark task for automated analysis of Pap smear imagery centered on the detection and, in one track, classification of cervical cells under Bethesda-style diagnostic categories. In the challenge formulations described in recent work, the central technical difficulties are dense cell distributions, severe class imbalance, frequent nuclear overlap, and the use of fixed-size nucleus-centered annotations rather than morphology-adaptive boxes (Kogan et al., 24 Mar 2026, Kong et al., 2 Apr 2026, Amster et al., 15 Apr 2026). The challenge has become a focal point for comparing modern object detectors, center-based formulations, transfer learning strategies, imbalance mitigation methods, and ensemble systems for cervical cytology, with submissions spanning YOLO variants, DETR-family transformers, U-Net heatmap regression, and multi-stage post-processing pipelines (Kogan et al., 24 Mar 2026, Kong et al., 2 Apr 2026, Amster et al., 15 Apr 2026).

1. Challenge scope and task formulation

The RIVA challenge is presented in at least two distinct tracks in the cited literature. One formulation defines Track A as simultaneous cell localization and classification into eight Bethesda categories, and Track B as cell localization only (Kong et al., 2 Apr 2026). Another formulation describes Track A (Classification) as assigning a Bethesda category to each cell crop and Track B (Detection) as localizing all Bethesda-relevant cells in a Pap-smear patch via axis-aligned bounding boxes (Amster et al., 15 Apr 2026). These descriptions are not identical in wording, but both establish a two-track structure separating multi-class recognition from pure localization.

Across the reported challenge settings, the dataset consists of 959 Pap-smear fields or patches of size $1024 \times 1024$ pixels, acquired at 40× magnification in one account, and split into 828 training and 131 validation images in another (Kogan et al., 24 Mar 2026, Amster et al., 15 Apr 2026). One paper states that these 959 images come from 115 patients and that each cell is annotated by a bounding box and one of eight Bethesda classes, comprising five lesion and three non-lesion categories (Kogan et al., 24 Mar 2026). Another paper emphasizes that the ground-truth boxes are all 100×100 pixel square boxes centered on each cell or on the cell nucleus, with no rotation (Kong et al., 2 Apr 2026, Amster et al., 15 Apr 2026). This fixed-box annotation convention materially shapes model design, because the task can be reformulated as center prediction rather than unconstrained box regression (Kong et al., 2 Apr 2026).

The evaluation protocol also varies slightly across reports. One challenge-oriented paper uses mAP@50–95 under the official RIVA protocol, with IoU thresholds from 0.50 to 0.95 in increments of 0.05 and a confidence threshold of 0.001 (Kogan et al., 24 Mar 2026). Another emphasizes mAP at standard IoU thresholds (primarily 0.50) for its Track A and Track B results (Kong et al., 2 Apr 2026). A third explicitly defines $\mathrm{mAP}_{50\text{–}95}$ as the mean of $AP@t$ over $\mathcal{T}=\{0.50,0.55,\dots,0.95\}$ (Amster et al., 15 Apr 2026). This suggests that leaderboard interpretation must be conditioned on the exact track and metric variant used in each report.

2. Dataset properties and annotation regime

A defining property of the RIVA dataset, as described in the literature, is extreme long-tailed imbalance. One multi-class detection paper reports the following class counts: SCC: 1 586; HSIL: 1 835; ASCH: 416; LSIL: 3 048; ASCUS: 356; NILM: 9 457; INFL: 8 190; ENDO: 1 270 (Kogan et al., 24 Mar 2026). The same source explicitly states that this yields a severe long-tailed imbalance, with NILM/INFL $\gg$ ASCUS/ASCH (Kogan et al., 24 Mar 2026). That imbalance is reflected in downstream classwise performance: on the validation set at IoU $\ge 0.5$, NILM reaches AP = 0.676 and INFL reaches AP = 0.433, whereas ASCUS has AP $\approx 0.005$ and ASCH has AP $\approx 0.031$ (Kogan et al., 24 Mar 2026).

The annotation policy of uniform 100×100 px boxes centered on nuclei appears repeatedly (Kong et al., 2 Apr 2026, Amster et al., 15 Apr 2026). In one winning solution, this property is treated as sufficiently distinctive to justify reframing the task from full-box prediction to center-point prediction followed by deterministic square-box reconstruction (Kong et al., 2 Apr 2026). In another solution, it motivates bounding-box rescaling at training time to two scales, 20×20 and 50×50, while preserving box centers and remapping predicted widths and heights back to 100×100 px at inference (Amster et al., 15 Apr 2026). These approaches differ architecturally, but both are direct responses to the same annotation geometry.

The reported category inventories are not fully uniform across papers. One source lists eight categories including NILM, ASC-US, LSIL, HSIL, ASC-H, AGC, carcinoma in situ, invasive carcinoma (Kong et al., 2 Apr 2026). Another gives the Track B category context as NILM, ASC-US, LSIL, ASC-H, HSIL, SCC, AGC (Amster et al., 15 Apr 2026). The multi-class YOLO-based paper instead reports SCC, HSIL, ASCH, LSIL, ASCUS, NILM, INFL, ENDO (Kogan et al., 24 Mar 2026). This suggests some variation in label taxonomies or reporting granularity across challenge tracks and participating systems.

3. Baseline architectures and detection paradigms

The challenge has elicited two dominant modeling paradigms: one-stage CNN detectors, especially YOLO-family models, and transformer-based detectors, particularly Co-DINO/Co-DETR variants.

A representative one-stage system adopts YOLOv11m as its base architecture, specifically the medium “m” variant implemented via the Ultralytics framework, with no custom layers beyond the standard YOLOv11m backbone-neck-head chain (Kogan et al., 24 Mar 2026). The stated motivation is a favorable accuracy vs. efficiency trade-off in dense-object scenarios (Kogan et al., 24 Mar 2026). A separate Track B pipeline uses two independent YOLOv8n detectors, denoted YOLOv8n$_{20\times20}$ and YOLOv8n$_{50\times50}$, trained at 1024×1024 resolution with batch size 16 for 150 epochs (Amster et al., 15 Apr 2026). In that system, YOLO acts as a high-recall proposal generator later tempered by post-processing and a binary classifier (Amster et al., 15 Apr 2026).

On the transformer side, the strongest reported RIVA solution in the supplied material builds on Co-DINO (Collaborative DETR) with a Swin-Large backbone (Kong et al., 2 Apr 2026). The backbone uses non-overlapping 4×4 patch embedding with embedding dimension $\mathrm{mAP}_{50\text{–}95}$0, four transformer stages with depths [2,2,18,2] and attention heads [6,12,24,48], and shifted-window attention with 7×7 windows (Kong et al., 2 Apr 2026). Feature maps from stages $\mathrm{mAP}_{50\text{–}95}$1 to $\mathrm{mAP}_{50\text{–}95}$2 are fused in an FPN-style neck before transformer encoding, while the main decoder performs one-to-one bipartite matching (Hungarian) between predictions and ground truths (Kong et al., 2 Apr 2026). The framework also includes training-only auxiliary branches: Faster R-CNN, ATSS, RetinaNet, and FCOS heads, each providing one-to-many supervision (Kong et al., 2 Apr 2026).

The DETR-based solution’s center-point formulation is especially characteristic. Instead of predicting full box coordinates, it predicts a hypothesis set of center coordinates $\mathrm{mAP}_{50\text{–}95}$3 and class probability vectors $\mathrm{mAP}_{50\text{–}95}$4, with fixed square reconstruction performed in post-processing (Kong et al., 2 Apr 2026). This design is tightly coupled to the fixed-size annotation regime and contrasts with conventional variable-box regression in generic detectors.

4. Imbalance mitigation, transfer learning, and augmentation

The severe class skew in RIVA has made imbalance mitigation a primary research theme. A YOLOv11m-based approach systematically evaluates loss reweighting, data resampling, and transfer learning as separate strategies before combining them in an ensemble (Kogan et al., 24 Mar 2026).

For loss reweighting, the classification loss is scaled by class-frequency-derived weights. With $\mathrm{mAP}_{50\text{–}95}$5, the class weight is defined as

$\mathrm{mAP}_{50\text{–}95}$6

and the reweighted classification loss becomes

$\mathrm{mAP}_{50\text{–}95}$7

This directly increases emphasis on underrepresented classes (Kogan et al., 24 Mar 2026).

For data resampling, the same work defines per-class sampling weights as

$\mathrm{mAP}_{50\text{–}95}$8

where $\mathrm{mAP}_{50\text{–}95}$9 is the total object count for class $AP@t$0. Each training image receives the arithmetic mean of the $AP@t$1 values over the classes present in that image, and images are sampled with probability proportional to that weight (Kogan et al., 24 Mar 2026). The square root is explicitly used to smooth extremes while still boosting rare classes (Kogan et al., 24 Mar 2026).

The transfer learning protocol in that system pretrains YOLOv11m on SIPaKMeD, described as 966 patches, 4 049 isolated cell images, freezes the first 10 backbone layers during fine-tuning on RIVA, and fine-tunes the remaining layers on all eight classes of the RIVA training set (Kogan et al., 24 Mar 2026). This is consistent with earlier cervical cytology literature in which transfer learning from natural-image or cytology-specific sources is central. For example, DeepPap pre-trains CaffeNet (BVLC reference model) on ILSVRC2012 ImageNet, transfers conv1–conv5 and pool1–pool5, and fine-tunes on cervical cell patches without prior segmentation (Zhang et al., 2018). While DeepPap is not a RIVA challenge method, it provides an antecedent for the reuse of pretrained feature hierarchies in cervical cytology (Zhang et al., 2018).

Augmentation strategies in RIVA solutions are similarly task-specific. The YOLOv11m ensemble uses automatic orientation correction, horizontal & vertical flips, and additive noise, with each real image expanded into three augmented versions (Kogan et al., 24 Mar 2026). The Co-DINO/Swin solution proposes a center-preserving augmentation pipeline in which random scale, rotation, and translation are applied while analytically updating ground-truth center coordinates and discarding transformed centers that leave the crop window (Kong et al., 2 Apr 2026). This is motivated by the claim that standard random cropping may cut off nuclei and break morphological cues (Kong et al., 2 Apr 2026). A plausible implication is that, for fixed-center annotations, preserving center validity during augmentation is more important than aggressive appearance perturbation.

5. Ensembling, post-processing, and geometric optimization

A major line of work in RIVA treats raw detector outputs as inputs to a second optimization layer comprising fusion, geometric correction, suppression, and refinement.

The YOLOv11m multi-class system constructs a three-model ensemble from the loss-reweighted, transfer-learned, and weighted-dataloader variants, then combines their predictions using Weighted Boxes Fusion (WBF) (Kogan et al., 24 Mar 2026). Boxes are clustered when their pairwise IoU $AP@t$2. The fused coordinate is a score-weighted average,

$AP@t$3

and fused confidence is

$AP@t$4

The model weights $AP@t$5 are set to the final-test-set mAP$AP@t$6 of the component models: 0.108, 0.106, 0.114, and a very low filtering threshold of 0.001 is used to maximize recall on rare lesions (Kogan et al., 24 Mar 2026).

The Co-DINO/Swin solution instead exploits the annotation geometry analytically. Given center jitter $AP@t$7 and $AP@t$8, it derives the IoU between a predicted square of side $AP@t$9 and the ground-truth square of side 100, then chooses

$\mathcal{T}=\{0.50,0.55,\dots,0.95\}$0

With typical jitter $\mathcal{T}=\{0.50,0.55,\dots,0.95\}$1–$\mathcal{T}=\{0.50,0.55,\dots,0.95\}$2 px, the empirical optimum is reported as $\mathcal{T}=\{0.50,0.55,\dots,0.95\}$3 px, and every center prediction is converted to a 101.5×101.5 box at inference (Kong et al., 2 Apr 2026). This is a notable example of post hoc geometric calibration improving mAP without changing network structure.

A distinct Track B framework introduces a multi-stage optimization pipeline combining YOLO detectors, a U-Net heatmap regressor, overlap-based ensembling, density-aware thresholding, and a binary refinement classifier (Amster et al., 15 Apr 2026). The U-Net uses a ResNet34 pretrained on ImageNet backbone and predicts a single-channel heatmap $\mathcal{T}=\{0.50,0.55,\dots,0.95\}$4, trained against Gaussian center maps with

$\mathcal{T}=\{0.50,0.55,\dots,0.95\}$5

(Amster et al., 15 Apr 2026). During inference, multi-scale averaging is performed at scales $\mathcal{T}=\{0.50,0.55,\dots,0.95\}$6, followed by local-maximum suppression with 25×25 px max-pooling and threshold 0.2 (Amster et al., 15 Apr 2026).

That pipeline’s post-processing has three explicit stages (Amster et al., 15 Apr 2026):

Stage	Operation	Key parameter
1	NMS	IoU $\mathcal{T}=\{0.50,0.55,\dots,0.95\}$7 removed
2	Spatial density filtering	confidence $\mathcal{T}=\{0.50,0.55,\dots,0.95\}$8 if grid cell has $\mathcal{T}=\{0.50,0.55,\dots,0.95\}$9 detections; otherwise $\gg$0
3	Binary refinement classifier	keep only predictions with $\gg$1

The final classifier is EfficientNet-B0 fine-tuned for “cell” vs “garbage” using hard-negative mining, with weighted binary cross-entropy

$\gg$2

(Amster et al., 15 Apr 2026). This architecture is explicitly optimized for leaderboard mAP rather than end-to-end elegance.

6. Reported performance and comparative results

The reported performance landscape differs substantially by track.

In the YOLOv11m multi-class detection setting, preliminary test set mAP$\gg$3 is reported as 0.160 for loss reweighting, 0.152 for transfer learning, 0.127 for weighted dataloader, and 0.201 for the ensemble (Kogan et al., 24 Mar 2026). On the final test set, the same models obtain 0.108, 0.106, and 0.114, respectively, while the ensemble reaches 0.147, described as a 29 % relative improvement vs. best single model (Kogan et al., 24 Mar 2026). The authors interpret the component models as offering complementary behaviors: minority-class focus from loss reweighting, stabilization from transfer learning, and recall boost from resampling (Kogan et al., 24 Mar 2026).

In the Co-DINO/Swin challenge report, Track A validation mAP values are given as YOLOv3: 0.120; RetinaNet: 0.137; CenterNet: 0.063; Co-Deformable DETR: 0.207; Co-DINO-ViT: 0.238; Co-DINO-Swin (ours): 0.237 (Kong et al., 2 Apr 2026). For Track B, the corresponding values are YOLOv3: 0.460; RetinaNet: 0.507; CenterNet: 0.519; Co-Deformable DETR: 0.586; Co-DINO-ViT: 0.604; Co-DINO-Swin (ours): 0.609 (Kong et al., 2 Apr 2026). The paper states that this achieved 1st place in Track B and 2nd place in Track A (Kong et al., 2 Apr 2026). Its ablation study on Track B reports 0.609 for the Co-DINO-Swin baseline, 0.611 with center-preserving cropping, 0.634 with box optimization, and 0.635 after track-specific loss tuning (Kong et al., 2 Apr 2026).

The multi-stage Track B pipeline reports the following internal validation results (Amster et al., 15 Apr 2026):

Model	Recall	Precision	mAP50–95
YOLOv8n 20×20	0.9896	0.0801	0.5966
YOLOv8n 50×50	0.9937	0.0665	0.6059
YOLO Ensemble	0.9870	0.1200	0.6153
U-Net	0.7983	0.6940	0.5494
Final Pipeline (all steps)	0.9485	0.1823	0.6232

The same work reports an official challenge score (Track B) of mAP50–95 = 0.5909, corresponding to 2nd place on the leaderboard (Amster et al., 15 Apr 2026).

Taken together, these results show that Track B localization-only scores are substantially higher than Track A multi-class scores in the supplied literature (Kong et al., 2 Apr 2026, Amster et al., 15 Apr 2026). This suggests that class discrimination, rather than mere center localization, remains the harder subproblem under RIVA conditions.

7. Relation to broader cervical cytology research

The RIVA challenge sits within a longer trajectory of cervical cytology automation, but it shifts emphasis from isolated-cell classification toward dense-scene detection under crowding and fixed-box annotation constraints.

Earlier work such as DeepPap addresses cervical cell classification without prior segmentation, using ConvNets on nucleus-centered patches with extensive rotation and translation augmentation (Zhang et al., 2018). On the Herlev benchmark, DeepPap reports Acc = 98.3%, AUC = 0.998, and strong robustness to centroid translation error (Zhang et al., 2018). This line of work demonstrates that segmentation-free deep classification can be highly effective in curated single-cell settings, but it does not directly solve the dense multi-cell localization problem foregrounded by RIVA.

Another single-cell classification direction combines pretrained CNN feature extraction with PCA and Grey Wolf Optimizer feature selection, followed by an RBF-SVM classifier (Basak et al., 2021). Reported accuracies include 98.32% on Herlev and 97.87% on SIPaKMeD (Basak et al., 2021). Such results underscore that isolated-cell benchmarks can reach near-saturation with relatively classical pipelines once object localization is assumed.

By contrast, object-detection work on cervical cytology has increasingly emphasized contextual reasoning. The paper “Exploring Contextual Relationships for Cervical Abnormal Cell Detection” introduces the RoI-relationship attention module (RRAM) and global RoI attention module (GRAM) within a Double-Head Faster R-CNN + FPN framework (Liang et al., 2022). On the CCD validation set, RRAM alone improves AP from 30.8 to 32.0, GRAM alone to 31.8, and the cascade RRAM→GRAM to 32.3 (Liang et al., 2022). The authors argue that abnormality assessment benefits from comparing a cell with its neighbors and the whole image, mirroring cytopathologist practice (Liang et al., 2022). This context-centric perspective is highly relevant to RIVA, where dense clustering and subtle inter-class morphology are recurrent themes.

A common misconception is that cervical cytology automation is principally a classification problem. The challenge papers indicate otherwise: in RIVA, the dominant bottlenecks include overlap handling, rare-class detection, fixed annotation geometry, and operating under very low confidence thresholds to preserve recall (Kogan et al., 24 Mar 2026, Kong et al., 2 Apr 2026, Amster et al., 15 Apr 2026). Another misconception is that better generic detectors are sufficient without task adaptation. The strongest RIVA solutions instead rely on challenge-specific modifications such as center-preserving augmentation, analytical box-size optimization, class-frequency-aware loss reweighting, weighted dataloaders, WBF with tuned IoU, and density-aware post-processing (Kogan et al., 24 Mar 2026, Kong et al., 2 Apr 2026, Amster et al., 15 Apr 2026).

Future directions are stated explicitly in the challenge literature. One source points to class-specific augmentations or multi-stage detection/classification pipelines as responses to the near-zero AP of ASCUS/ASCH (Kogan et al., 24 Mar 2026). Another recommends adaptive box shapes or segmentation-based masks, transformer-based detectors for long-range context and occlusion handling, and dataset expansion to include more rare classes and stains (Amster et al., 15 Apr 2026). These proposals indicate that the RIVA challenge is not only a benchmark but also a vehicle for reframing cervical cytology from isolated-cell recognition toward integrated detection, contextual reasoning, and geometric calibration in crowded clinical imagery.