---
title: 'Fitzpatrick17k–C: Clean Benchmark for Dermatology'
url: https://www.emergentmind.com/topics/fitzpatrick17k-c-cleaned-dataset
type: topic
---

# Fitzpatrick17k–C: Clean Benchmark for Dermatology

Fitzpatrick17k–C, frequently referred to as CleanPatrick, is a rigorously curated benchmark dataset for image data cleaning, built upon the original Fitzpatrick17k dermatology corpus. Designed to enable systematic comparison of data cleaning algorithms in the presence of realistic, naturally occurring noise, Fitzpatrick17k–C provides high-fidelity, expert-validated issue annotations for off-topic content, near-duplicates, and label errors. In both scope and methodology, Fitzpatrick17k–C addresses key deficiencies in synthetic or narrowly scoped cleaning benchmarks and is an essential resource for data-centric artificial intelligence in medical imaging [2505.11034], [2401.14497].

## 1. Origin and Motivation

The original Fitzpatrick17k dataset comprises 16,577 clinical photographs of cutaneous disease from DermNet and Atlas Dermatológico, labeled into 114 diagnostic classes via weak supervision (atlas captions) and annotated with Fitzpatrick skin types (I–VI). Despite its wide adoption for skin lesion analysis, Fitzpatrick17k is characterized by substantial contamination: naturally occurring data quality issues span off-topic images, label misassignments, and duplicate representations, affecting ~16–30% of the data. Notably, Groh et al. did not provide a true held-out test split, compounding the risk of data leakage and evaluation bias [2505.11034], [2401.14497].

Fitzpatrick17k–C was constructed as the first exhaustively annotated, realistically contaminated benchmark for data cleaning in the image domain, specifically addressing:

- Real-world distribution of off-topic samples, near-duplicates, and mislabels.
- The need for standardized evaluation and ranking metrics for cleaning methods across these issue types.
- Improved reproducibility and comparability for downstream model training and benchmarking.

## 2. Exhaustive Annotation and Cleaning Pipeline

### Medical Crowdsourcing and Expert Adjudication

A critical methodological innovation is the large-scale annotation campaign executed through Centaur Labs / DiagnosUs, engaging 933 screened medical crowd workers. For each image (or image pair), binary yes/no annotations were collected across three orthogonal tasks:

- Off-topic detection: “Should this image be in a skin-lesion dataset?”
- Near-duplicate detection: “Are these two images near-duplicates?”
- Label-error detection: “Is the assigned diagnosis clearly wrong?”

In total, 496,377 binary annotations were amassed, corresponding to an average of ~10 votes per sample (range 1–225). Definitions and adjudicated prevalence rates, post-aggregation and expert calibration, are summarized below:

| Issue Type     | Definition                                                     | Prevalence             |
|----------------|----------------------------------------------------------------|------------------------|
| Off-topic      | Not a human skin lesion (e.g., slides, histology, text, etc.)  | 613 / 16,577 ≈ 4%      |
| Near-duplicate | Alternate captures of a lesion (thumbnails, angle, lighting)   | 3,556 / 16,577 ≈ 21%   |
| Label error    | Clearly incorrect diagnosis given image content                | 3,666 / 16,577 ≈ 22%   |

Expert validation was performed by three board-certified dermatologists on stratified random samples (400 items per issue type), yielding Krippendorff’s $\alpha$ of 0.91 (near-duplicate), 0.60 (off-topic), and 0.42 (label errors). Agreement between crowd consensus and expert review was 96% for off-topic/near-duplicate and 67% for label errors.

### IRT-Inspired Aggregation

Final labels were computed using a generative model adapted from item-response theory (GLAD):

\[
P\bigl(y_{a,i}=1\mid c_a,b_i\bigr) = \sigma\bigl(c_a\,b_i\bigr)
\]

where $y_{a,i}$ is the binary label from annotator $a$ for item $i$, $c_a$ encodes annotator ability, $b_i$ item difficulty/orientation ($c_a \sim N(0,1)$, $b_i \sim N(0, \sigma_b^2)$, $\sigma_b = 10^3$), and $\sigma(\cdot)$ is the sigmoid function. Posterior inference was performed via variational mean-field optimization (Adam, lr=0.1, 10,000 steps, Pyro framework). Final positive class probability $\bar p_i$ was computed as:

\[
\bar p_i = \frac{1}{M}\sum_{m=1}^M \mathbb{I}\left[b_i^{(m)} > 0\right]
\]

where $M=1{,}000$ posterior samples. Binary labels were assigned by thresholding $\bar p_i$ at expert-calibrated values $t_{OT}$, $t_{ND}$, $t_{LE}$ for off-topic, near-duplicate, and label-error respectively [2505.11034].

## 3. Dataset Structure, Splits, and Statistics

Fitzpatrick17k–C retains all 16,577 de-duplicated images as the starting point for its annotation framework; however, the alternate pipeline in [2401.14497] applies aggressive duplicate/outlier elimination for a downstream subset with 11,394 images and standardized 70:10:20 train:val:test splits by diagnosis label. CleanPatrick provides the following metadata per image:

- "image_path" or "image" (PIL object/array)
- "diagnosis" (class label in $\{1,\dots,114\}$)
- "skin_type" (I–VI)
- "off_topic_score", "off_topic" (probability, binary flag)
- "near_duplicate_score", "near_duplicate"
- "label_error_score", "label_error"
- "duplicate_group_id" (near-duplicate cluster membership)
- Raw crowd voting table (optional)

No new train/val/test split is imposed by CleanPatrick itself; the original Fitzpatrick17k’s splits (~80% train, ~10% val, ~10% test) can be adopted. The distributions of diagnostic classes and Fitzpatrick skin types are approximately preserved by stratified sampling, e.g., in one split $[19.81, 19.26, 22.64, 25.84, 28.78, 24.68]\%$ of test images across skin types I–VI [2401.14497].

## 4. Benchmarking Methodology and Performance

CleanPatrick formalizes data cleaning as a ranking problem. For each contamination type, a method $f$ assigns a real-valued score $s(x)$ (or $s(x,x')$ for pairs), and practitioners audit the top-$k$ items.

Key evaluation metrics:

- Precision@k ($P@k$): $(1/k) \sum_{i=1}^k \mathbb{I}[\hat{y}_{(i)}=1]$
- Recall@k ($R@k$): $(1/P^+) \sum_{i=1}^k \mathbb{I}[\hat{y}_{(i)}=1]$
- Average Precision (AP): $\mathrm{AP} = \sum_{k=1}^N P(k) \, \Delta r(k)$, $\Delta r(k)=\frac{\mathbb{I}[\hat y_{(k)}=1]}{P^+}$
- AUROC: area under the true vs. false positive curve

Selected baseline results [2505.11034]:

| Issue Type          | Prevalence | Best Method (AUROC, AP/Prec@100)                  |
|---------------------|------------|--------------------------------------------------|
| Off-topic           | 3.7%       | IForest AUROC=0.77, AP=0.16; SelfClean P@100=0.52|
| Near-duplicate      | 21.4%      | SelfClean AUROC=0.92, AP=0.88; pHash/SSIM ≈0.50  |
| Label error         | 22.1%      | SelfClean AUROC=0.57, AP=0.27; NoiseRank ≈0.48   |

Self-supervised representation-based approaches excel at near-duplicate detection; classical methods (e.g., IForest) achieve competitive off-topic detection under constrained annotation budgets. Label-error detection remains challenging in fine-grained medical tasks.

Notably, post-cleaning performance drops dramatically: For VGG-16 on random stratified splits, accuracy on Fitzpatrick17k–C decreases from $67.3\%$ (original, contaminated data) to $22.25\%\pm0.60\%$ after cleaning ($\Delta Acc =-45.05$pp), confirming severe benchmark inflation due to prior contamination. All skin-tone subgroups show 5–30pp accuracy drops [2401.14497].

## 5. Cleaning Methodology and Best Practices

The cleaning methodology applied in [2401.14497] employs metric-learning (fastdup, cleanvision) for initial duplicate discovery, cosine similarity clustering via union-find, and rigorous manual/algorithmic conflict resolution on label and skin type. Clusters with homogeneous labels retain only the highest-resolution representative; those with label conflicts are entirely removed. Erroneous or outlier images are identified via nearest-neighbor similarity thresholds and removed. A stratified split into train:val:test sets ensures label/site balance and eliminates data leakage.

Best practices derived from this process include:

- Use self-supervised metrics for duplicate detection and verify candidates above similarity thresholds (e.g., $\tau\geq0.95$).
- Apply union-find for cluster merging and strict removal of label-inconsistent clusters.
- Remove non-dermatological and structurally outlier images before model evaluation.
- Release mapping metadata, code, and standardized splits for reproducibility.
- Quantify the cleaning effect using $\Delta Acc$ (cleaned versus original test accuracy).

## 6. Access, Usage, and Community Impact

Fitzpatrick17k–C is hosted on Hugging Face Datasets, facilitating standardized programmatic access:

```python
# pip install datasets transformers pillow
from datasets import load_dataset
ds = load_dataset("Digital-Dermatology/CleanPatrick")
print(ds)  # shows splits (usually 'train' with all images)
sample = ds["train"][0]
print(sample.keys())
# Access e.g. sample["image"], sample["diagnosis"], sample["off_topic"], sample["duplicate_group_id"]
```

All images are distributed as PIL.Image objects or file paths, with scores and binary flags per contamination type, and near-duplicate clusters indexed by `duplicate_group_id`. Release of this resource enables transparent experimental protocols and high-fidelity benchmarking for data-centric workflows in dermatology AI and can inform broader cleaning efforts in other image domains [2505.11034].

A plausible implication is that systematic cleaning of other large-scale medical datasets could reveal similar inflation in reported benchmarks and necessitate recalibration of progress in medical AI model evaluation. The rigorous, expert-grounded annotation and aggregation framework employed here establishes a standard for future work in data-centric machine learning and medical imaging corpus curation.

Source: https://www.emergentmind.com/topics/fitzpatrick17k-c-cleaned-dataset