Herlev Pap Smear Dataset Overview

• The Herlev Pap Smear Dataset is a benchmark of 917 single-cell cervical images labeled into 7 classes, supporting both binary and multi-class classification tasks.
• It includes detailed annotations such as nucleus and cytoplasm masks, enabling segmentation, morphology-aware learning, and explainability studies.
• Researchers employ a variety of methods—from deep CNNs to transformer models—and diverse validation protocols to benchmark diagnostic screening and severity grading.

Searching arXiv for recent and foundational papers on the Herlev Pap Smear Dataset to ground the article. The Herlev Pap Smear Dataset is a publicly available benchmark of cervical cytology images that has become a standard testbed for automated Pap smear analysis, including binary normal-versus-abnormal discrimination, fine-grained multi-class classification, nuclei segmentation, morphology-aware learning, and explainability studies. Across the literature summarized here, the dataset is consistently described as containing 917 single-cell images organized into seven classes spanning normal epithelial categories and abnormal stages from dysplasia to carcinoma in situ (Pirovano et al., 2019, Zhang et al., 2018, Rahaman et al., 2021). Its methodological importance derives from the coexistence of image-level labels, clinically meaningful severity structure, and, in several usages, nucleus and cytoplasm masks, which together support work on classification, segmentation, attribution, and morphology-guided modeling (Pirovano et al., 2019, Lin et al., 2018, Zhao et al., 2018).

1. Dataset composition and label taxonomy

The dataset is repeatedly characterized as a collection of 917 single-cell Pap smear images labeled into seven categories (Pirovano et al., 2019, Rahaman et al., 2021, Zhang et al., 2018). The class names are reported with minor wording variation across papers, but the common taxonomy is stable: three normal epithelial classes and four abnormal classes representing increasing atypia or malignancy-related change.

In one formulation, the seven classes are normal columnar, normal intermediate, normal superficial, light dysplastic, moderate dysplastic, severe dysplastic, and carcinoma in situ (Pirovano et al., 2019). Other papers state the same structure as superficial squamous epithelial, intermediate squamous epithelial, columnar epithelial, mild dysplasia, moderate dysplasia, severe dysplasia, and carcinoma-in-situ (Yilmaz et al., 2020), or as Normal squamous, Intermediate squamous, Columnar, Mild dysplasia, Moderate dysplasia, Severe dysplasia, and Carcinoma in situ with explicit counts 74, 70, 98, 182, 146, 197, and 150, respectively (Rahaman et al., 2021). The binary grouping used throughout the literature is equally consistent: the first three classes are treated as normal or benign, and the last four as abnormal or malignant, yielding 242 normal/benign and 675 abnormal/malignant images (Rahaman et al., 2021, Zhang et al., 2018, Deo et al., 2023).

A further property that distinguishes the dataset is its compatibility with ordinal reformulations. One study merges the three normal subclasses into a single normal category to construct a five-class severity problem: {normal, light dysplastic, moderate dysplastic, severe dysplastic, carcinoma in situ}, with ordinal scores 1–5 for regression-constrained learning (Pirovano et al., 2019). This is clinically motivated because the abnormal categories are ordered by severity, with carcinoma in situ indicating highest severity and hinting at the presence of actual cancer (Pirovano et al., 2019).

This structure has made the dataset useful for both binary screening-style tasks and more difficult fine-grained classification. A plausible implication is that Herlev occupies an intermediate position between simple diagnostic screening benchmarks and morphology-intensive severity grading benchmarks.

2. Imaging characteristics, annotations, and dataset-specific assets

The images are described as single-cell Pap smear samples throughout the cited work (Pirovano et al., 2019, Zhang et al., 2018, Rahaman et al., 2021). Several acquisition-related details appear in the literature, though not every paper reports them. One source states that specimens were prepared via conventional Pap smear with Pap staining, captured by a digital camera on a microscope (Herlev University Hospital), with pixel resolution 0.201 μm per pixel (Zhang et al., 2018). Another similarly reports collection at Herlev University Hospital using a digital camera and microscope, again with conventional Pap smear preparation and Pap staining and the same 0.201 μm per pixel resolution (Lin et al., 2018).

Reported image size ranges vary by paper because some give native dimensions while others describe only approximate extents. One study states that images are between 50 and 400 pixels wide (Pirovano et al., 2019). Another reports BMP format with image sizes ranging from 77×43 to 360×577 pixels (Deo et al., 2023). Papers that do not provide acquisition metadata often explicitly note that staining protocol, magnification, resolution, microscope vendor, original resolution, or file format are not reported in their text (Albzour et al., 25 Aug 2025, Rahaman et al., 2021).

Annotation resources are an important differentiator. In explainability-oriented work, the dataset is said to include annotation masks delineating nucleus and cytoplasm, which are used for quantitative attribution evaluation rather than for classifier training (Pirovano et al., 2019). In segmentation work, the original images are described as manually segmented into 4 labels: background, cytoplasm, nuclei, and unknown regions, with unknown treated as background (Zhao et al., 2018). DeepPap also reports a ground-truth nucleus mask, used to obtain the nucleus centroid for patch extraction while deliberately avoiding segmentation in the classification pipeline (Zhang et al., 2018). Morphology-aware classification uses the ground-truth binary segmentation masks for the nucleus and the cytoplasm as additional input channels (Lin et al., 2018).

These complementary uses show that Herlev supports several distinct supervision regimes: image-level classification, region-based explainability, and pixel-level segmentation. This suggests that the dataset’s continuing relevance is tied not only to class labels but also to its support for morphology-centric experimentation.

3. Canonical task formulations built on Herlev

The literature uses the dataset in at least four recurring task formulations.

First, the most common formulation is binary normal vs abnormal classification, where the three normal epithelial types are merged and the four dysplastic or carcinoma-related classes are merged (Pirovano et al., 2019, Zhang et al., 2018, Deo et al., 2023). This mapping reflects the core screening question and is used in classical machine learning, CNN, transformer, and feature-fusion studies (Yilmaz et al., 2020, Deo et al., 2023, Rahaman et al., 2021).

Second, the full 7-class classification problem preserves all original labels and is widely used as a fine-grained recognition benchmark (Rahaman et al., 2021, Lin et al., 2018, Basak et al., 2021). In this setting, performance is generally lower than in the binary task, reflecting morphological overlap, stain variability, and class imbalance (Rahaman et al., 2021, Lin et al., 2018).

Third, one study defines a 5-class severity classification by merging the three normal subclasses into a single class and preserving the ordinal abnormal categories (Pirovano et al., 2019). The associated loss combines softmax cross-entropy with an ordinal regression term:

$$\mathcal{L}(x) = \mathcal{CE}(p; y_x) + \left(y_x - \sum_{i=0}^{4} (i+1)\,p_i \right)^2.$$

Here, $p$ denotes softmax probabilities over the five severity classes and $y_x \in \{1,2,3,4,5\}$ is the ground-truth severity score (Pirovano et al., 2019). The stated motivation is to penalize predictions that place probability mass far from the true ordinal position, thereby reducing extreme confusions and improving consistency with the medical notion of ordered severity (Pirovano et al., 2019).

Fourth, the dataset is used for nuclei segmentation. In that setting, one paper trains a model to predict background, cytoplasm, normal nuclei, and abnormal nuclei, then forms the final nucleus mask by taking the union of the two nucleus channels (Zhao et al., 2018). Another paper uses segmentation as an upstream preprocessing stage for downstream binary classification and evaluates whether segmented images improve performance relative to non-segmented images (Albzour et al., 25 Aug 2025).

Because the same images support binary diagnosis, multi-class grading, ordinal severity modeling, segmentation, and explanation, Herlev has become a compact benchmark for comparing modeling assumptions rather than merely a source of raw data.

4. Experimental protocols and preprocessing conventions

Experimental protocols vary considerably across papers, and this heterogeneity is central to interpreting reported numbers. Several studies use five-fold cross-validation (Zhang et al., 2018, Albzour et al., 25 Aug 2025), while another uses 4 random folds and presents AUC boxplots across 4 random folds (Pirovano et al., 2019). A transformer-based study instead uses a 90%/10% split, giving 825 train and 92 test images (Deo et al., 2023). A feature-fusion study uses 60% training, 20% validation, and 20% testing per class, with augmented training totals reported separately (Rahaman et al., 2021). Some segmentation work does not state an explicit split or cross-validation protocol at all and reports performance for all images from Herlev dataset (Zhao et al., 2018).

Preprocessing choices are likewise inconsistent. Some papers report very limited preprocessing. The regression-constraint study does not describe any stain normalization, color-space transforms, segmentation for training, resizing to the CNN input size, normalization, or data augmentation settings, using masks only for attribution evaluation and choosing a white image as the baseline for Integrated Gradients (Pirovano et al., 2019). A classical ML/CNN comparison resizes images to 64×64×3, uses a test split of 0.15, and a validation split of 0.15 within training, with no data augmentation reported (Yilmaz et al., 2020). CerviFormer resizes images to 72×72×3, applies Layer normalization, and uses horizontal flipping plus random zoom with height = 0.2 and width = 0.2 (Deo et al., 2023). DeepCervix rescales images to 224×224×3, applies backbone-specific Keras preprocess_input, and uses extensive offline and in-place augmentation, including affine transformations, CLAHE, gamma contrast, edge detection, grayscale conversion, hue/saturation changes, brightness adaptations, and flips (Rahaman et al., 2021).

Patch-based methods introduce additional protocol details. DeepPap extracts 128×128 nucleus-centered patches, upsamples them to 256×256×3, then takes 227×227 random crops, using asymmetric rotation and translation augmentation to balance the normal and abnormal classes (Zhang et al., 2018). The morphology-aware CNN similarly extracts 128×128 patches centered on the ground-truth nucleus centroid, upsamples RGB to 256×256×3, masks to 256×256×2, concatenates them into a 256×256×5 tensor, and then uses architecture-specific crops of 227×227 or 224×224 (Lin et al., 2018).

Segmentation-first workflows impose yet another preprocessing regime. Albzour and Lam resize all images to 128×128 pixels; for segmentation, images are converted to grayscale and normalized to [0,1], with adaptive thresholding, Gaussian blurring, and morphological transformations to refine masks, and segmented masks post-processed to [0,255] before classification (Albzour et al., 25 Aug 2025).

These differences mean that Herlev results are not directly comparable unless the split protocol, task definition, and preprocessing pipeline are aligned. This is not a flaw of the dataset itself, but it is a recurring source of apparent discrepancy in the literature.

5. Classification benchmarks and methodological trajectories

Herlev has been a benchmark for several methodological trajectories in cervical cytology classification.

A segmentation-free deep CNN baseline is represented by DeepPap, which fine-tunes an ImageNet-pretrained BVLC CaffeNet (AlexNet-style) on nucleus-centered patches and aggregates 1000 sub-patch predictions per cell:

$$\hat{y} = \frac{1}{N}\sum_{i=1}^{N} p_i, \quad N=1000.$$

Under five-fold cross-validation, DeepPap reports Sensitivity = 98.2% ± 1.2, Specificity = 98.3% ± 0.9, Accuracy = 98.3% ± 0.7, H-mean = 98.3% ± 0.3, F-measure = 98.8% ± 0.5, and AUC ≈ 0.998 for binary classification on Herlev (Zhang et al., 2018). It also reports a seven-class overall error of 1.6% after changing the final layer from 2 to 7 outputs (Zhang et al., 2018).

Morphology-aware CNNs extend this idea by injecting mask information directly. In a patient-level five-fold cross-validation study, GoogleNet-5C—using RGB plus nucleus and cytoplasm masks—achieves AUC 0.984 ± 0.012, Accuracy 94.5 ± 2.8%, Sensitivity 97.4 ± 2.7%, and Specificity 90.4 ± 3.1% for the binary task, and 64.5 ± 4.2% for the 7-class task (Lin et al., 2018). The reported effect of adding morphology is generally positive, especially for GoogleNet, AlexNet, and ResNet (Lin et al., 2018).

Feature-fusion approaches push the 7-class setting further. DeepCervix, using VGG16, VGG19, ResNet50, and XceptionNet as backbones and concatenating four 1024-dimensional feature vectors into a 4096-dimensional representation, reports Accuracy 98.91% for binary benign-vs-malignant classification and 90.32% for 7-class classification in the main Herlev results table, while noting an abstract value of 98.32% for the binary result (Rahaman et al., 2021). The same paper explicitly states that ResNet50 is the strongest individual backbone on Herlev and that XceptionNet is the weakest, particularly in the 7-class task (Rahaman et al., 2021).

A related hybrid feature-selection pipeline uses deep features extracted from VGG-16, ResNet-50, Inception v3, and DenseNet-121, compresses them with PCA retaining 99% variance, then applies Grey Wolf Optimizer feature selection before an SVM with RBF kernel (Basak et al., 2021). On Herlev, the best configuration—ResNet-50+VGG-16—achieves Accuracy 98.32%, Precision 98.66%, Recall 97.65%, and F1-score 98.12% in the 7-class problem (Basak et al., 2021).

Transformer-based work has also used Herlev. CerviFormer, operating on 72×72×3 inputs tokenized into 1296 image tokens with a 256-token latent array, reports 94.57% accuracy for the 2-state task, along with Sensitivity = 87.50%, Specificity = 97.06%, Cohen’s kappa = 0.85, and class-wise metrics of Normal: Precision 0.88, Recall 0.91, F1 0.89 and Abnormal: Precision 0.97, Recall 0.96, F1 0.96 (Deo et al., 2023).

By contrast, a simpler CNN and traditional ML comparison reports lower performance. Using the dataset’s 20 precomputed, labeled morphological features, the best classical models—kNN and XGBoost—reach 85% accuracy, while a custom 4-layer CNN on 64×64×3 inputs reaches 93% test accuracy, 93% recall, 96% precision, 89% specificity, and 95% F1 after 50 epochs (Yilmaz et al., 2020).

Across these studies, Herlev has functioned both as a benchmark for high-performing binary discrimination and as a more difficult fine-grained classification problem where architecture choice, feature fusion, and class structure matter substantially.

6. Segmentation, explainability, and morphology-centric analysis

Beyond classification, Herlev has been used to study whether models rely on clinically meaningful cellular structures.

In explainability work, Integrated Gradients is used to attribute model output to pixels. The discrete approximation given is

$$\mathrm{Am}(x_i; F, x') = (x_i - x'_i)\,\frac{1}{m}\sum_{k=0}^{m} \frac{\partial F\!\left(x' + \frac{k}{m}(x - x')\right)}{\partial x_i},$$

with a white image baseline chosen because of the white background of Pap smear slides (Pirovano et al., 2019). Using nucleus and cytoplasm masks, regional attribution fractions are computed as

$$At_{N} = \frac{\sum_{i \in \mathcal{N}} \mathrm{Am}(x_i)}{\sum_i \mathrm{Am}(x_i)}, \qquad At_{C} = \frac{\sum_{i \in \mathcal{C}} \mathrm{Am}(x_i)}{\sum_i \mathrm{Am}(x_i)}.$$

Qualitative results show activated pixels predominantly in the nucleus, and quantitatively the nucleus attribution fraction increases with severity, with carcinoma in situ contributing about twice as much as in normal cases (Pirovano et al., 2019). The paper explicitly links this to Bethesda-guideline nuclear features such as nucleus size, nuclear irregularity, chromatin density, and nucleus-to-cytoplasm ratio (Pirovano et al., 2019).

Segmentation-specific work treats Herlev as a pixel-wise benchmark. The Deformable Multipath Ensemble Model (D-MEM) uses a U-shaped convolutional network with dense blocks, deformable convolution, and a three-network ensemble composed of Dense-UNet, D-Con, and D-Exp, aggregated by majority voting (Zhao et al., 2018). It reports ZSI (Dice) = 0.933 ± 0.14, Precision = 0.946 ± 0.06, Recall = 0.984 ± 0.00, and inference time < 0.1 seconds per image on Herlev (Zhao et al., 2018). The evaluation metric is defined as

$$\mathrm{ZSI} = \mathrm{DSC} = \frac{2\,|X \cap Y|}{|X| + |Y|}.$$

The same work compares against UNet, Dense-UNet, and earlier methods, reporting UNet ZSI 0.869, Dense-UNet ZSI 0.910, and D-MEM ZSI 0.933 (Zhao et al., 2018).

A later segmentation-plus-classification study uses a pretrained U-Net to generate masks because the Herlev dataset does not come with segmentation masks in that paper’s workflow (Albzour et al., 25 Aug 2025). Classification is then performed with a custom CNN on raw versus segmented images. The reported averages across folds are 81.02% accuracy, 80.44% precision, 81.02% recall, 78.25% F1-score for non-segmented images and 80.80% accuracy, 80.85% precision, 80.80% recall, 79.55% F1-score for segmented images, corresponding to a +0.41% precision and +1.30% F1-score gain for segmented inputs, with accuracy and recall essentially unchanged (−0.22%) (Albzour et al., 25 Aug 2025).

Taken together, these results indicate that Herlev supports both explicit segmentation research and post hoc validation of feature use. This suggests that the dataset has been especially valuable for testing whether cytology models attend to nucleus-centric morphology rather than background structure.

7. Limitations, reproducibility issues, and interpretive cautions

Several limitations recur across studies, and they shape how Herlev results should be interpreted. The dataset is repeatedly described as small, with only 917 images, and several papers note class imbalance or uneven per-class counts (Pirovano et al., 2019, Rahaman et al., 2021, Albzour et al., 25 Aug 2025, Basak et al., 2021). One study emphasizes that abnormal images (675) outnumber normals (242), while another notes that deeper residual models tend to perform better on Herlev because it is smaller and highly imbalanced (Rahaman et al., 2021, Lin et al., 2018).

Generalization is another recurring concern. Albzour and Lam state that Herlev comprises isolated single cells captured under controlled lab conditions, unlike clinical workflows involving multi-cell clusters, complex overlapping morphology, and whole-slide images with larger field-of-view variation and staining heterogeneity (Albzour et al., 25 Aug 2025). The same paper argues that, without reported imaging metadata, domain shift to real-world screening is likely (Albzour et al., 25 Aug 2025). Similar cautions appear elsewhere, including remarks that performance may be sensitive to lab-specific staining and acquisition differences because stain normalization and domain adaptation are often absent (Pirovano et al., 2019).

Reproducibility is uneven. Some works provide detailed hyperparameters, such as DeepPap’s learning rates, batch size, momentum, weight decay, dropout, and test-time aggregation scheme (Zhang et al., 2018), or DeepCervix’s learning-rate schedule, batch sizes, augmentation operators, and runtime environment (Rahaman et al., 2021). Other papers omit optimizer choices, learning rates, seeds, hardware, exact split definitions, or segmentation metrics (Pirovano et al., 2019, Albzour et al., 25 Aug 2025, Zhao et al., 2018). In a few cases, internal inconsistencies are explicitly noted: DeepCervix reports 98.32% binary accuracy in the abstract but 98.91% in the main results table (Rahaman et al., 2021), and the explainability paper reports 94% binary accuracy in the abstract while the conclusion mentions 96.7% binary accuracy with F1 = 0.95 on another run (Pirovano et al., 2019).

There are also interpretive cautions about attribution and segmentation. Integrated Gradients is described as post hoc and correlational; highlighting the nucleus does not prove causation, and the choice of a white image baseline may affect attribution magnitudes (Pirovano et al., 2019). Morphology-aware classifiers that use ground-truth masks achieve strong results, but deployment would require automated segmentation, whose error propagation is not quantified in that work (Lin et al., 2018). DeepPap avoids segmentation but assumes access to a nucleus center, obtained there from ground truth, though robustness to ±5–10 pixel center error is demonstrated (Zhang et al., 2018).

A common misconception is that Herlev yields a single canonical accuracy figure. In fact, reported performance depends strongly on task definition, split protocol, augmentation, use of masks, and whether the problem is binary, 5-class ordinal, or 7-class fine-grained classification. Another misconception is that segmentation is always necessary. The literature contains both strong segmentation-free systems—such as DeepPap and DeepCervix—and studies in which segmentation offers only marginal classification gains (Zhang et al., 2018, Rahaman et al., 2021, Albzour et al., 25 Aug 2025).

In aggregate, the Herlev Pap Smear Dataset remains influential because it condenses several clinically meaningful problems into a compact benchmark: binary screening, severity grading, morphology-aware representation learning, and nuclei segmentation. Its small size, imbalance, and controlled single-cell setting limit direct clinical extrapolation, but those same constraints have made it a durable reference point for methodological comparison across the cervical cytology literature.