---
title: RIVA Cervical Cytology Challenge
url: https://www.emergentmind.com/topics/riva-cervical-cytology-challenge
type: topic
---

# RIVA Cervical Cytology Challenge

Searching arXiv for recent papers on the RIVA Cervical Cytology Challenge and related cervical cytology detection/classification methods.
The **RIVA Cervical Cytology Challenge** is a benchmark task for automated analysis of Pap smear imagery centered on the detection and, in one track, classification of cervical cells under Bethesda-style diagnostic categories. In the challenge formulations described in recent work, the central technical difficulties are dense cell distributions, severe class imbalance, frequent nuclear overlap, and the use of fixed-size nucleus-centered annotations rather than morphology-adaptive boxes [2603.23742][2604.02090][2604.13939]. The challenge has become a focal point for comparing modern object detectors, center-based formulations, transfer learning strategies, imbalance mitigation methods, and ensemble systems for cervical cytology, with submissions spanning YOLO variants, DETR-family transformers, U-Net heatmap regression, and multi-stage post-processing pipelines [2603.23742][2604.02090][2604.13939].

## 1. Challenge scope and task formulation

The RIVA challenge is presented in at least two distinct tracks in the cited literature. One formulation defines **Track A** as simultaneous cell localization and classification into eight Bethesda categories, and **Track B** as cell localization only [2604.02090]. Another formulation describes **Track A (Classification)** as assigning a Bethesda category to each cell crop and **Track B (Detection)** as localizing all Bethesda-relevant cells in a Pap-smear patch via axis-aligned bounding boxes [2604.13939]. These descriptions are not identical in wording, but both establish a two-track structure separating multi-class recognition from pure localization.

Across the reported challenge settings, the dataset consists of **959 Pap-smear fields or patches of size \(1024 \times 1024\) pixels**, acquired at **40× magnification** in one account, and split into **828 training and 131 validation images** in another [2603.23742][2604.13939]. One paper states that these 959 images come from **115 patients** and that each cell is annotated by a bounding box and one of **eight Bethesda classes**, comprising **five lesion** and **three non-lesion** categories [2603.23742]. Another paper emphasizes that the ground-truth boxes are all **100×100 pixel square boxes centered on each cell** or on the **cell nucleus**, with no rotation [2604.02090][2604.13939]. This fixed-box annotation convention materially shapes model design, because the task can be reformulated as center prediction rather than unconstrained box regression [2604.02090].

The evaluation protocol also varies slightly across reports. One challenge-oriented paper uses **mAP@50–95 under the official RIVA protocol**, with IoU thresholds from **0.50 to 0.95** in increments of **0.05** and a **confidence threshold of 0.001** [2603.23742]. Another emphasizes **mAP at standard IoU thresholds (primarily 0.50)** for its Track A and Track B results [2604.02090]. A third explicitly defines **\(\mathrm{mAP}_{50\text{–}95}\)** as the mean of \(AP@t\) over \(\mathcal{T}=\{0.50,0.55,\dots,0.95\}\) [2604.13939]. This suggests that leaderboard interpretation must be conditioned on the exact track and metric variant used in each report.

## 2. Dataset properties and annotation regime

A defining property of the RIVA dataset, as described in the literature, is **extreme long-tailed imbalance**. One multi-class detection paper reports the following class counts: **SCC: 1 586; HSIL: 1 835; ASCH: 416; LSIL: 3 048; ASCUS: 356; NILM: 9 457; INFL: 8 190; ENDO: 1 270** [2603.23742]. The same source explicitly states that this yields a severe long-tailed imbalance, with **NILM/INFL \(\gg\) ASCUS/ASCH** [2603.23742]. That imbalance is reflected in downstream classwise performance: on the validation set at **IoU \(\ge 0.5\)**, **NILM** reaches **AP = 0.676** and **INFL** reaches **AP = 0.433**, whereas **ASCUS** has **AP \(\approx 0.005\)** and **ASCH** has **AP \(\approx 0.031\)** [2603.23742].

The annotation policy of **uniform 100×100 px boxes centered on nuclei** appears repeatedly [2604.02090][2604.13939]. In one winning solution, this property is treated as sufficiently distinctive to justify reframing the task from full-box prediction to **center-point prediction** followed by deterministic square-box reconstruction [2604.02090]. In another solution, it motivates **bounding-box rescaling at training time to two scales, 20×20 and 50×50**, while preserving box centers and remapping predicted widths and heights back to **100×100 px** at inference [2604.13939]. These approaches differ architecturally, but both are direct responses to the same annotation geometry.

The reported category inventories are not fully uniform across papers. One source lists eight categories including **NILM, ASC-US, LSIL, HSIL, ASC-H, AGC, carcinoma in situ, invasive carcinoma** [2604.02090]. Another gives the Track B category context as **NILM, ASC-US, LSIL, ASC-H, HSIL, SCC, AGC** [2604.13939]. The multi-class YOLO-based paper instead reports **SCC, HSIL, ASCH, LSIL, ASCUS, NILM, INFL, ENDO** [2603.23742]. This suggests some variation in label taxonomies or reporting granularity across challenge tracks and participating systems.

## 3. Baseline architectures and detection paradigms

The challenge has elicited two dominant modeling paradigms: **one-stage CNN detectors**, especially YOLO-family models, and **transformer-based detectors**, particularly Co-DINO/Co-DETR variants.

A representative one-stage system adopts **YOLOv11m** as its base architecture, specifically the medium “m” variant implemented via the **Ultralytics** framework, with **no custom layers beyond the standard YOLOv11m backbone-neck-head chain** [2603.23742]. The stated motivation is a favorable **accuracy vs. efficiency trade-off in dense-object scenarios** [2603.23742]. A separate Track B pipeline uses **two independent YOLOv8n detectors**, denoted **YOLOv8n\(_{20\times20}\)** and **YOLOv8n\(_{50\times50}\)**, trained at **1024×1024** resolution with **batch size 16** for **150 epochs** [2604.13939]. In that system, YOLO acts as a high-recall proposal generator later tempered by post-processing and a binary classifier [2604.13939].

On the transformer side, the strongest reported RIVA solution in the supplied material builds on **Co-DINO (Collaborative DETR)** with a **Swin-Large** backbone [2604.02090]. The backbone uses non-overlapping **4×4 patch embedding** with embedding dimension **\(d=192\)**, four transformer stages with depths **[2,2,18,2]** and attention heads **[6,12,24,48]**, and **shifted-window attention** with **7×7** windows [2604.02090]. Feature maps from stages \(C_2\) to \(C_4\) are fused in an **FPN-style neck** before transformer encoding, while the main decoder performs **one-to-one bipartite matching (Hungarian)** between predictions and ground truths [2604.02090]. The framework also includes training-only auxiliary branches: **Faster R-CNN**, **ATSS**, **RetinaNet**, and **FCOS** heads, each providing one-to-many supervision [2604.02090].

The DETR-based solution’s **center-point formulation** is especially characteristic. Instead of predicting full box coordinates, it predicts a hypothesis set of center coordinates \((\hat c_{x_i}, \hat c_{y_i})\) and class probability vectors \(\hat{\mathbf p}_i \in \mathbb{R}^9\), with fixed square reconstruction performed in post-processing [2604.02090]. This design is tightly coupled to the fixed-size annotation regime and contrasts with conventional variable-box regression in generic detectors.

## 4. Imbalance mitigation, transfer learning, and augmentation

The severe class skew in RIVA has made **imbalance mitigation** a primary research theme. A YOLOv11m-based approach systematically evaluates **loss reweighting**, **data resampling**, and **transfer learning** as separate strategies before combining them in an ensemble [2603.23742].

For **loss reweighting**, the classification loss is scaled by class-frequency-derived weights. With \(p_k = (\text{count of class }k)/(\text{total object count})\), the class weight is defined as
\[
w_k = -\log(p_k),
\]
and the reweighted classification loss becomes
\[
L_{cls} = \sum_k w_k \cdot BCE(predict_k, target_k).
\]
This directly increases emphasis on underrepresented classes [2603.23742].

For **data resampling**, the same work defines per-class sampling weights as
\[
w_k = \sqrt{\sum_i c_i / c_k},
\]
where \(c_k\) is the total object count for class \(k\). Each training image receives the arithmetic mean of the \(w_k\) values over the classes present in that image, and images are sampled with probability proportional to that weight [2603.23742]. The square root is explicitly used to **smooth extremes** while still boosting rare classes [2603.23742].

The **transfer learning** protocol in that system pretrains YOLOv11m on **SIPaKMeD**, described as **966 patches, 4 049 isolated cell images**, freezes the **first 10 backbone layers** during fine-tuning on RIVA, and fine-tunes the remaining layers on all eight classes of the RIVA training set [2603.23742]. This is consistent with earlier cervical cytology literature in which transfer learning from natural-image or cytology-specific sources is central. For example, **DeepPap** pre-trains **CaffeNet (BVLC reference model)** on **ILSVRC2012 ImageNet**, transfers **conv1–conv5 and pool1–pool5**, and fine-tunes on cervical cell patches without prior segmentation [1801.08616]. While DeepPap is not a RIVA challenge method, it provides an antecedent for the reuse of pretrained feature hierarchies in cervical cytology [1801.08616].

Augmentation strategies in RIVA solutions are similarly task-specific. The YOLOv11m ensemble uses **automatic orientation correction**, **horizontal & vertical flips**, and **additive noise**, with each real image expanded into **three augmented versions** [2603.23742]. The Co-DINO/Swin solution proposes a **center-preserving augmentation** pipeline in which random scale, rotation, and translation are applied while analytically updating ground-truth center coordinates and discarding transformed centers that leave the crop window [2604.02090]. This is motivated by the claim that standard random cropping may cut off nuclei and break morphological cues [2604.02090]. A plausible implication is that, for fixed-center annotations, preserving center validity during augmentation is more important than aggressive appearance perturbation.

## 5. Ensembling, post-processing, and geometric optimization

A major line of work in RIVA treats raw detector outputs as inputs to a second optimization layer comprising **fusion**, **geometric correction**, **suppression**, and **refinement**.

The YOLOv11m multi-class system constructs a three-model ensemble from the loss-reweighted, transfer-learned, and weighted-dataloader variants, then combines their predictions using **Weighted Boxes Fusion (WBF)** [2603.23742]. Boxes are clustered when their pairwise **IoU \(\ge 0.7\)**. The fused coordinate is a score-weighted average,
\[
\bar b=\frac{\sum_m\sum_{i\in C_m} w_m s_i b_i}{\sum_m\sum_{i\in C_m} w_m s_i},
\]
and fused confidence is
\[
(\sum_{m,i} w_m s_i)/(\sum_m |C_m|).
\]
The model weights \(w_m\) are set to the final-test-set **mAP\(_{50\text{–}95}\)** of the component models: **0.108, 0.106, 0.114**, and a very low filtering threshold of **0.001** is used to maximize recall on rare lesions [2603.23742].

The Co-DINO/Swin solution instead exploits the annotation geometry analytically. Given center jitter \(\Delta_x = |\hat c_x - c_x^{gt}|\) and \(\Delta_y = |\hat c_y - c_y^{gt}|\), it derives the IoU between a predicted square of side \(S\) and the ground-truth square of side 100, then chooses
\[
S^* = \arg\max_{S>0} \mathbb{E}_{\varepsilon}[IoU(\Delta_x,\Delta_y;S)].
\]
With typical jitter \(\Delta \sim 1\)–\(1.5\) px, the empirical optimum is reported as **\(S^* \approx 101.5\) px**, and every center prediction is converted to a **101.5×101.5** box at inference [2604.02090]. This is a notable example of post hoc geometric calibration improving mAP without changing network structure.

A distinct Track B framework introduces a **multi-stage optimization pipeline** combining YOLO detectors, a U-Net heatmap regressor, overlap-based ensembling, density-aware thresholding, and a binary refinement classifier [2604.13939]. The U-Net uses a **ResNet34 pretrained on ImageNet** backbone and predicts a single-channel heatmap \(\hat H \in [0,1]^{1024\times1024}\), trained against Gaussian center maps with
\[
L_{seg} = \frac{1}{N}\sum_{i=1}^N (H_i - \hat H_i)^2
\]
[2604.13939]. During inference, multi-scale averaging is performed at scales **\(\{0.8,1.0,1.2\}\)**, followed by local-maximum suppression with **25×25 px** max-pooling and threshold **0.2** [2604.13939].

That pipeline’s post-processing has three explicit stages [2604.13939]:

| Stage | Operation | Key parameter |
|---|---|---|
| 1 | NMS | IoU \(> 0.75\) removed |
| 2 | Spatial density filtering | confidence \(\ge 0.1\) if grid cell has \(\ge 30\) detections; otherwise \(\ge 0.001\) |
| 3 | Binary refinement classifier | keep only predictions with \(p \ge 0.05\) |

The final classifier is **EfficientNet-B0** fine-tuned for **“cell” vs “garbage”** using hard-negative mining, with weighted binary cross-entropy
\[
L_{BCE} = -\alpha y\log p - (1-\alpha)(1-y)\log(1-p)
\]
[2604.13939]. This architecture is explicitly optimized for leaderboard mAP rather than end-to-end elegance.

## 6. Reported performance and comparative results

The reported performance landscape differs substantially by track.

In the YOLOv11m multi-class detection setting, **preliminary test set** mAP\(_{50\text{–}95}\) is reported as **0.160** for loss reweighting, **0.152** for transfer learning, **0.127** for weighted dataloader, and **0.201** for the ensemble [2603.23742]. On the **final test set**, the same models obtain **0.108**, **0.106**, and **0.114**, respectively, while the ensemble reaches **0.147**, described as a **29 % relative improvement vs. best single model** [2603.23742]. The authors interpret the component models as offering complementary behaviors: minority-class focus from loss reweighting, stabilization from transfer learning, and recall boost from resampling [2603.23742].

In the Co-DINO/Swin challenge report, **Track A** validation mAP values are given as **YOLOv3: 0.120; RetinaNet: 0.137; CenterNet: 0.063; Co-Deformable DETR: 0.207; Co-DINO-ViT: 0.238; Co-DINO-Swin (ours): 0.237** [2604.02090]. For **Track B**, the corresponding values are **YOLOv3: 0.460; RetinaNet: 0.507; CenterNet: 0.519; Co-Deformable DETR: 0.586; Co-DINO-ViT: 0.604; Co-DINO-Swin (ours): 0.609** [2604.02090]. The paper states that this achieved **1st place in Track B and 2nd place in Track A** [2604.02090]. Its ablation study on Track B reports **0.609** for the Co-DINO-Swin baseline, **0.611** with center-preserving cropping, **0.634** with box optimization, and **0.635** after track-specific loss tuning [2604.02090].

The multi-stage Track B pipeline reports the following internal validation results [2604.13939]:

| Model | Recall | Precision | mAP50–95 |
|---|---:|---:|---:|
| YOLOv8n 20×20 | 0.9896 | 0.0801 | 0.5966 |
| YOLOv8n 50×50 | 0.9937 | 0.0665 | 0.6059 |
| YOLO Ensemble | 0.9870 | 0.1200 | 0.6153 |
| U-Net | 0.7983 | 0.6940 | 0.5494 |
| Final Pipeline (all steps) | 0.9485 | 0.1823 | 0.6232 |

The same work reports an **official challenge score (Track B)** of **mAP50–95 = 0.5909**, corresponding to **2nd place on the leaderboard** [2604.13939].

Taken together, these results show that **Track B localization-only scores are substantially higher than Track A multi-class scores** in the supplied literature [2604.02090][2604.13939]. This suggests that class discrimination, rather than mere center localization, remains the harder subproblem under RIVA conditions.

## 7. Relation to broader cervical cytology research

The RIVA challenge sits within a longer trajectory of cervical cytology automation, but it shifts emphasis from isolated-cell classification toward dense-scene detection under crowding and fixed-box annotation constraints.

Earlier work such as **DeepPap** addresses cervical cell classification **without prior segmentation**, using ConvNets on nucleus-centered patches with extensive rotation and translation augmentation [1801.08616]. On the **Herlev** benchmark, DeepPap reports **Acc = 98.3%**, **AUC = 0.998**, and strong robustness to centroid translation error [1801.08616]. This line of work demonstrates that segmentation-free deep classification can be highly effective in curated single-cell settings, but it does not directly solve the dense multi-cell localization problem foregrounded by RIVA.

Another single-cell classification direction combines pretrained CNN feature extraction with **PCA** and **Grey Wolf Optimizer** feature selection, followed by an **RBF-SVM** classifier [2106.04919]. Reported accuracies include **98.32%** on Herlev and **97.87%** on **SIPaKMeD** [2106.04919]. Such results underscore that isolated-cell benchmarks can reach near-saturation with relatively classical pipelines once object localization is assumed.

By contrast, object-detection work on cervical cytology has increasingly emphasized **contextual reasoning**. The paper **“Exploring Contextual Relationships for Cervical Abnormal Cell Detection”** introduces the **RoI-relationship attention module (RRAM)** and **global RoI attention module (GRAM)** within a **Double-Head Faster R-CNN + FPN** framework [2207.04693]. On the CCD validation set, **RRAM alone** improves AP from **30.8 to 32.0**, **GRAM alone** to **31.8**, and the cascade **RRAM→GRAM** to **32.3** [2207.04693]. The authors argue that abnormality assessment benefits from comparing a cell with its neighbors and the whole image, mirroring cytopathologist practice [2207.04693]. This context-centric perspective is highly relevant to RIVA, where dense clustering and subtle inter-class morphology are recurrent themes.

A common misconception is that cervical cytology automation is principally a classification problem. The challenge papers indicate otherwise: in RIVA, the dominant bottlenecks include **overlap handling**, **rare-class detection**, **fixed annotation geometry**, and **operating under very low confidence thresholds to preserve recall** [2603.23742][2604.02090][2604.13939]. Another misconception is that better generic detectors are sufficient without task adaptation. The strongest RIVA solutions instead rely on challenge-specific modifications such as **center-preserving augmentation**, **analytical box-size optimization**, **class-frequency-aware loss reweighting**, **weighted dataloaders**, **WBF with tuned IoU**, and **density-aware post-processing** [2603.23742][2604.02090][2604.13939].

Future directions are stated explicitly in the challenge literature. One source points to **class-specific augmentations** or **multi-stage detection/classification pipelines** as responses to the near-zero AP of **ASCUS/ASCH** [2603.23742]. Another recommends **adaptive box shapes or segmentation-based masks**, **transformer-based detectors for long-range context and occlusion handling**, and dataset expansion to include **more rare classes** and stains [2604.13939]. These proposals indicate that the RIVA challenge is not only a benchmark but also a vehicle for reframing cervical cytology from isolated-cell recognition toward integrated detection, contextual reasoning, and geometric calibration in crowded clinical imagery.

Source: https://www.emergentmind.com/topics/riva-cervical-cytology-challenge