---
title: Herlev Pap Smear Dataset Overview
url: https://www.emergentmind.com/topics/herlev-pap-smear-dataset
type: topic
---

# Herlev Pap Smear Dataset Overview

Searching arXiv for recent and foundational papers on the Herlev Pap Smear Dataset to ground the article.
The **Herlev Pap Smear Dataset** is a publicly available benchmark of cervical cytology images that has become a standard testbed for automated Pap smear analysis, including binary normal-versus-abnormal discrimination, fine-grained multi-class classification, nuclei segmentation, morphology-aware learning, and explainability studies. Across the literature summarized here, the dataset is consistently described as containing **917 single-cell images** organized into **seven classes** spanning normal epithelial categories and abnormal stages from dysplasia to carcinoma in situ [1908.02650], [1801.08616], [2102.12191]. Its methodological importance derives from the coexistence of image-level labels, clinically meaningful severity structure, and, in several usages, nucleus and cytoplasm masks, which together support work on classification, segmentation, attribution, and morphology-guided modeling [1908.02650], [1810.06058], [1812.00527].

## 1. Dataset composition and label taxonomy

The dataset is repeatedly characterized as a collection of **917 single-cell Pap smear images** labeled into **seven categories** [1908.02650], [2102.12191], [1801.08616]. The class names are reported with minor wording variation across papers, but the common taxonomy is stable: three normal epithelial classes and four abnormal classes representing increasing atypia or malignancy-related change.

In one formulation, the seven classes are **normal columnar, normal intermediate, normal superficial, light dysplastic, moderate dysplastic, severe dysplastic, and carcinoma in situ** [1908.02650]. Other papers state the same structure as **superficial squamous epithelial, intermediate squamous epithelial, columnar epithelial, mild dysplasia, moderate dysplasia, severe dysplasia, and carcinoma-in-situ** [2009.06366], or as **Normal squamous, Intermediate squamous, Columnar, Mild dysplasia, Moderate dysplasia, Severe dysplasia, and Carcinoma in situ** with explicit counts **74, 70, 98, 182, 146, 197, and 150**, respectively [2102.12191]. The binary grouping used throughout the literature is equally consistent: the first three classes are treated as **normal** or **benign**, and the last four as **abnormal** or **malignant**, yielding **242 normal/benign** and **675 abnormal/malignant** images [2102.12191], [1801.08616], [2303.10222].

A further property that distinguishes the dataset is its compatibility with ordinal reformulations. One study merges the three normal subclasses into a single **normal** category to construct a five-class severity problem: **{normal, light dysplastic, moderate dysplastic, severe dysplastic, carcinoma in situ}**, with ordinal scores **1–5** for regression-constrained learning [1908.02650]. This is clinically motivated because the abnormal categories are ordered by severity, with **carcinoma in situ indicating highest severity and hinting at the presence of actual cancer** [1908.02650].

This structure has made the dataset useful for both binary screening-style tasks and more difficult fine-grained classification. A plausible implication is that Herlev occupies an intermediate position between simple diagnostic screening benchmarks and morphology-intensive severity grading benchmarks.

## 2. Imaging characteristics, annotations, and dataset-specific assets

The images are described as **single-cell** Pap smear samples throughout the cited work [1908.02650], [1801.08616], [2102.12191]. Several acquisition-related details appear in the literature, though not every paper reports them. One source states that specimens were prepared via **conventional Pap smear with Pap staining**, captured by **a digital camera on a microscope (Herlev University Hospital)**, with **pixel resolution 0.201 μm per pixel** [1801.08616]. Another similarly reports collection at **Herlev University Hospital using a digital camera and microscope**, again with **conventional Pap smear preparation and Pap staining** and the same **0.201 μm per pixel** resolution [1810.06058].

Reported image size ranges vary by paper because some give native dimensions while others describe only approximate extents. One study states that images are **between 50 and 400 pixels wide** [1908.02650]. Another reports **BMP** format with image sizes ranging **from 77×43 to 360×577 pixels** [2303.10222]. Papers that do not provide acquisition metadata often explicitly note that staining protocol, magnification, resolution, microscope vendor, original resolution, or file format are not reported in their text [2508.17728], [2102.12191].

Annotation resources are an important differentiator. In explainability-oriented work, the dataset is said to include **annotation masks delineating nucleus and cytoplasm**, which are used for **quantitative attribution evaluation** rather than for classifier training [1908.02650]. In segmentation work, the original images are described as **manually segmented into 4 labels: background, cytoplasm, nuclei, and unknown regions**, with **unknown treated as background** [1812.00527]. DeepPap also reports a **ground-truth nucleus mask**, used to obtain the nucleus centroid for patch extraction while deliberately avoiding segmentation in the classification pipeline [1801.08616]. Morphology-aware classification uses the **ground-truth binary segmentation masks for the nucleus and the cytoplasm** as additional input channels [1810.06058].

These complementary uses show that Herlev supports several distinct supervision regimes: image-level classification, region-based explainability, and pixel-level segmentation. This suggests that the dataset’s continuing relevance is tied not only to class labels but also to its support for morphology-centric experimentation.

## 3. Canonical task formulations built on Herlev

The literature uses the dataset in at least four recurring task formulations.

First, the most common formulation is **binary normal vs abnormal classification**, where the three normal epithelial types are merged and the four dysplastic or carcinoma-related classes are merged [1908.02650], [1801.08616], [2303.10222]. This mapping reflects the core screening question and is used in classical machine learning, CNN, transformer, and feature-fusion studies [2009.06366], [2303.10222], [2102.12191].

Second, the full **7-class classification** problem preserves all original labels and is widely used as a fine-grained recognition benchmark [2102.12191], [1810.06058], [2106.04919]. In this setting, performance is generally lower than in the binary task, reflecting morphological overlap, stain variability, and class imbalance [2102.12191], [1810.06058].

Third, one study defines a **5-class severity classification** by merging the three normal subclasses into a single class and preserving the ordinal abnormal categories [1908.02650]. The associated loss combines softmax cross-entropy with an ordinal regression term:
$$
\mathcal{L}(x) = \mathcal{CE}(p; y_x) + \left(y_x - \sum_{i=0}^{4} (i+1)\,p_i \right)^2.
$$
Here, $p$ denotes softmax probabilities over the five severity classes and $y_x \in \{1,2,3,4,5\}$ is the ground-truth severity score [1908.02650]. The stated motivation is to penalize predictions that place probability mass far from the true ordinal position, thereby reducing extreme confusions and improving consistency with the medical notion of ordered severity [1908.02650].

Fourth, the dataset is used for **nuclei segmentation**. In that setting, one paper trains a model to predict **background, cytoplasm, normal nuclei, and abnormal nuclei**, then forms the final nucleus mask by taking the union of the two nucleus channels [1812.00527]. Another paper uses segmentation as an upstream preprocessing stage for downstream binary classification and evaluates whether segmented images improve performance relative to non-segmented images [2508.17728].

Because the same images support binary diagnosis, multi-class grading, ordinal severity modeling, segmentation, and explanation, Herlev has become a compact benchmark for comparing modeling assumptions rather than merely a source of raw data.

## 4. Experimental protocols and preprocessing conventions

Experimental protocols vary considerably across papers, and this heterogeneity is central to interpreting reported numbers. Several studies use **five-fold cross-validation** [1801.08616], [2508.17728], while another uses **4 random folds** and presents **AUC boxplots across 4 random folds** [1908.02650]. A transformer-based study instead uses a **90%/10% split**, giving **825 train and 92 test images** [2303.10222]. A feature-fusion study uses **60% training, 20% validation, and 20% testing** per class, with augmented training totals reported separately [2102.12191]. Some segmentation work does not state an explicit split or cross-validation protocol at all and reports performance **for all images from Herlev dataset** [1812.00527].

Preprocessing choices are likewise inconsistent. Some papers report very limited preprocessing. The regression-constraint study does **not describe any stain normalization, color-space transforms, segmentation for training, resizing to the CNN input size, normalization, or data augmentation settings**, using masks only for attribution evaluation and choosing **a white image** as the baseline for Integrated Gradients [1908.02650]. A classical ML/CNN comparison resizes images to **64×64×3**, uses a **test split of 0.15**, and a **validation split of 0.15** within training, with **no data augmentation reported** [2009.06366]. CerviFormer resizes images to **72×72×3**, applies **Layer normalization**, and uses **horizontal flipping** plus **random zoom with height = 0.2 and width = 0.2** [2303.10222]. DeepCervix rescales images to **224×224×3**, applies backbone-specific **Keras preprocess_input**, and uses extensive offline and in-place augmentation, including affine transformations, CLAHE, gamma contrast, edge detection, grayscale conversion, hue/saturation changes, brightness adaptations, and flips [2102.12191].

Patch-based methods introduce additional protocol details. DeepPap extracts **128×128** nucleus-centered patches, upsamples them to **256×256×3**, then takes **227×227** random crops, using asymmetric rotation and translation augmentation to balance the normal and abnormal classes [1801.08616]. The morphology-aware CNN similarly extracts **128×128** patches centered on the ground-truth nucleus centroid, upsamples RGB to **256×256×3**, masks to **256×256×2**, concatenates them into a **256×256×5** tensor, and then uses architecture-specific crops of **227×227** or **224×224** [1810.06058].

Segmentation-first workflows impose yet another preprocessing regime. Albzour and Lam resize all images to **128×128 pixels**; for segmentation, images are converted to **grayscale** and normalized to **[0,1]**, with **adaptive thresholding, Gaussian blurring, and morphological transformations** to refine masks, and segmented masks post-processed to **[0,255]** before classification [2508.17728].

These differences mean that Herlev results are not directly comparable unless the split protocol, task definition, and preprocessing pipeline are aligned. This is not a flaw of the dataset itself, but it is a recurring source of apparent discrepancy in the literature.

## 5. Classification benchmarks and methodological trajectories

Herlev has been a benchmark for several methodological trajectories in cervical cytology classification.

A segmentation-free deep CNN baseline is represented by **DeepPap**, which fine-tunes an ImageNet-pretrained **BVLC CaffeNet (AlexNet-style)** on nucleus-centered patches and aggregates **1000 sub-patch** predictions per cell:
$$
\hat{y} = \frac{1}{N}\sum_{i=1}^{N} p_i, \quad N=1000.
$$
Under **five-fold cross-validation**, DeepPap reports **Sensitivity = 98.2% ± 1.2**, **Specificity = 98.3% ± 0.9**, **Accuracy = 98.3% ± 0.7**, **H-mean = 98.3% ± 0.3**, **F-measure = 98.8% ± 0.5**, and **AUC ≈ 0.998** for binary classification on Herlev [1801.08616]. It also reports a **seven-class overall error of 1.6%** after changing the final layer from 2 to 7 outputs [1801.08616].

Morphology-aware CNNs extend this idea by injecting mask information directly. In a patient-level five-fold cross-validation study, **GoogleNet-5C**—using RGB plus nucleus and cytoplasm masks—achieves **AUC 0.984 ± 0.012**, **Accuracy 94.5 ± 2.8%**, **Sensitivity 97.4 ± 2.7%**, and **Specificity 90.4 ± 3.1%** for the binary task, and **64.5 ± 4.2%** for the 7-class task [1810.06058]. The reported effect of adding morphology is generally positive, especially for GoogleNet, AlexNet, and ResNet [1810.06058].

Feature-fusion approaches push the 7-class setting further. **DeepCervix**, using **VGG16, VGG19, ResNet50, and XceptionNet** as backbones and concatenating four **1024-dimensional** feature vectors into a **4096-dimensional** representation, reports **Accuracy 98.91%** for binary benign-vs-malignant classification and **90.32%** for 7-class classification in the main Herlev results table, while noting an **abstract value of 98.32%** for the binary result [2102.12191]. The same paper explicitly states that **ResNet50** is the strongest individual backbone on Herlev and that **XceptionNet** is the weakest, particularly in the 7-class task [2102.12191].

A related hybrid feature-selection pipeline uses deep features extracted from **VGG-16, ResNet-50, Inception v3, and DenseNet-121**, compresses them with **PCA** retaining **99% variance**, then applies **Grey Wolf Optimizer** feature selection before an **SVM with RBF kernel** [2106.04919]. On Herlev, the best configuration—**ResNet-50+VGG-16**—achieves **Accuracy 98.32%**, **Precision 98.66%**, **Recall 97.65%**, and **F1-score 98.12%** in the **7-class** problem [2106.04919].

Transformer-based work has also used Herlev. **CerviFormer**, operating on **72×72×3** inputs tokenized into **1296 image tokens** with a **256-token latent array**, reports **94.57% accuracy** for the **2-state** task, along with **Sensitivity = 87.50%**, **Specificity = 97.06%**, **Cohen’s kappa = 0.85**, and class-wise metrics of **Normal: Precision 0.88, Recall 0.91, F1 0.89** and **Abnormal: Precision 0.97, Recall 0.96, F1 0.96** [2303.10222].

By contrast, a simpler CNN and traditional ML comparison reports lower performance. Using the dataset’s **20 precomputed, labeled morphological features**, the best classical models—**kNN** and **XGBoost**—reach **85% accuracy**, while a custom **4-layer CNN** on **64×64×3** inputs reaches **93% test accuracy**, **93% recall**, **96% precision**, **89% specificity**, and **95% F1** after **50 epochs** [2009.06366].

Across these studies, Herlev has functioned both as a benchmark for high-performing binary discrimination and as a more difficult fine-grained classification problem where architecture choice, feature fusion, and class structure matter substantially.

## 6. Segmentation, explainability, and morphology-centric analysis

Beyond classification, Herlev has been used to study whether models rely on clinically meaningful cellular structures.

In explainability work, Integrated Gradients is used to attribute model output to pixels. The discrete approximation given is
$$
\mathrm{Am}(x_i; F, x') = (x_i - x'_i)\,\frac{1}{m}\sum_{k=0}^{m} \frac{\partial F\!\left(x' + \frac{k}{m}(x - x')\right)}{\partial x_i},
$$
with a **white image baseline** chosen because of the white background of Pap smear slides [1908.02650]. Using nucleus and cytoplasm masks, regional attribution fractions are computed as
$$
At_{N} = \frac{\sum_{i \in \mathcal{N}} \mathrm{Am}(x_i)}{\sum_i \mathrm{Am}(x_i)}, \qquad
At_{C} = \frac{\sum_{i \in \mathcal{C}} \mathrm{Am}(x_i)}{\sum_i \mathrm{Am}(x_i)}.
$$
Qualitative results show activated pixels **predominantly in the nucleus**, and quantitatively the **nucleus attribution fraction increases with severity**, with **carcinoma in situ contributing about twice as much as in normal cases** [1908.02650]. The paper explicitly links this to Bethesda-guideline nuclear features such as **nucleus size, nuclear irregularity, chromatin density, and nucleus-to-cytoplasm ratio** [1908.02650].

Segmentation-specific work treats Herlev as a pixel-wise benchmark. The **Deformable Multipath Ensemble Model (D-MEM)** uses a **U-shaped convolutional network** with **dense blocks**, **deformable convolution**, and a **three-network ensemble** composed of **Dense-UNet**, **D-Con**, and **D-Exp**, aggregated by **majority voting** [1812.00527]. It reports **ZSI (Dice) = 0.933 ± 0.14**, **Precision = 0.946 ± 0.06**, **Recall = 0.984 ± 0.00**, and **inference time < 0.1 seconds per image** on Herlev [1812.00527]. The evaluation metric is defined as
$$
\mathrm{ZSI} = \mathrm{DSC} = \frac{2\,|X \cap Y|}{|X| + |Y|}.
$$
The same work compares against UNet, Dense-UNet, and earlier methods, reporting **UNet ZSI 0.869**, **Dense-UNet ZSI 0.910**, and **D-MEM ZSI 0.933** [1812.00527].

A later segmentation-plus-classification study uses a **pretrained U-Net** to generate masks because **the Herlev dataset does not come with segmentation masks** in that paper’s workflow [2508.17728]. Classification is then performed with a custom CNN on raw versus segmented images. The reported averages across folds are **81.02% accuracy, 80.44% precision, 81.02% recall, 78.25% F1-score** for non-segmented images and **80.80% accuracy, 80.85% precision, 80.80% recall, 79.55% F1-score** for segmented images, corresponding to a **+0.41% precision** and **+1.30% F1-score** gain for segmented inputs, with **accuracy and recall essentially unchanged (−0.22%)** [2508.17728].

Taken together, these results indicate that Herlev supports both explicit segmentation research and post hoc validation of feature use. This suggests that the dataset has been especially valuable for testing whether cytology models attend to nucleus-centric morphology rather than background structure.

## 7. Limitations, reproducibility issues, and interpretive cautions

Several limitations recur across studies, and they shape how Herlev results should be interpreted. The dataset is repeatedly described as **small**, with only **917 images**, and several papers note **class imbalance** or uneven per-class counts [1908.02650], [2102.12191], [2508.17728], [2106.04919]. One study emphasizes that **abnormal images (675) outnumber normals (242)**, while another notes that deeper residual models tend to perform better on Herlev because it is **smaller and highly imbalanced** [2102.12191], [1810.06058].

Generalization is another recurring concern. Albzour and Lam state that Herlev comprises **isolated single cells captured under controlled lab conditions**, unlike clinical workflows involving **multi-cell clusters, complex overlapping morphology, and whole-slide images with larger field-of-view variation and staining heterogeneity** [2508.17728]. The same paper argues that, **without reported imaging metadata**, domain shift to real-world screening is likely [2508.17728]. Similar cautions appear elsewhere, including remarks that performance may be sensitive to **lab-specific staining and acquisition differences** because **stain normalization** and **domain adaptation** are often absent [1908.02650].

Reproducibility is uneven. Some works provide detailed hyperparameters, such as DeepPap’s learning rates, batch size, momentum, weight decay, dropout, and test-time aggregation scheme [1801.08616], or DeepCervix’s learning-rate schedule, batch sizes, augmentation operators, and runtime environment [2102.12191]. Other papers omit optimizer choices, learning rates, seeds, hardware, exact split definitions, or segmentation metrics [1908.02650], [2508.17728], [1812.00527]. In a few cases, internal inconsistencies are explicitly noted: DeepCervix reports **98.32%** binary accuracy in the abstract but **98.91%** in the main results table [2102.12191], and the explainability paper reports **94%** binary accuracy in the abstract while the conclusion mentions **96.7%** binary accuracy with **F1 = 0.95** on another run [1908.02650].

There are also interpretive cautions about attribution and segmentation. Integrated Gradients is described as **post hoc and correlational**; highlighting the nucleus does **not prove causation**, and the choice of a **white image baseline** may affect attribution magnitudes [1908.02650]. Morphology-aware classifiers that use ground-truth masks achieve strong results, but deployment would require **automated segmentation**, whose error propagation is not quantified in that work [1810.06058]. DeepPap avoids segmentation but assumes access to a **nucleus center**, obtained there from ground truth, though robustness to **±5–10 pixel** center error is demonstrated [1801.08616].

A common misconception is that Herlev yields a single canonical accuracy figure. In fact, reported performance depends strongly on task definition, split protocol, augmentation, use of masks, and whether the problem is binary, 5-class ordinal, or 7-class fine-grained classification. Another misconception is that segmentation is always necessary. The literature contains both strong segmentation-free systems—such as DeepPap and DeepCervix—and studies in which segmentation offers only **marginal** classification gains [1801.08616], [2102.12191], [2508.17728].

In aggregate, the Herlev Pap Smear Dataset remains influential because it condenses several clinically meaningful problems into a compact benchmark: binary screening, severity grading, morphology-aware representation learning, and nuclei segmentation. Its small size, imbalance, and controlled single-cell setting limit direct clinical extrapolation, but those same constraints have made it a durable reference point for methodological comparison across the cervical cytology literature.

Source: https://www.emergentmind.com/topics/herlev-pap-smear-dataset