ImageNet-F: A Privacy-Enhanced ILSVRC Variant

• ImageNet-F is a face-obfuscated version of ILSVRC that applies blurring and overlaying techniques to enhance privacy while preserving dataset utility.
• It employs a two-stage semi-automatic annotation pipeline using automated face detection and human refinement to process about 17% of images with faces.
• Empirical results show minimal accuracy drops (≤1.0%) with competitive transfer performance across various deep learning models and tasks.

Searching arXiv for relevant papers on ImageNet-F and related ImageNet benchmark variants. ImageNet-F is a convenient shorthand for the face-obfuscated ImageNet variant enabled by “A Study of Face Obfuscation in ImageNet” (Yang et al., 2021). In that work, the name is not presented as an official pre-existing benchmark label; rather, it refers to an ImageNet / ILSVRC variant in which incidental human faces are annotated by bounding boxes and then obfuscated, while images with no faces remain unchanged. The resource is therefore a privacy-enhanced version of ILSVRC built to preserve the benchmark and pretraining value of ImageNet while addressing privacy concerns created by visible faces in a widely distributed dataset.

1. Definition, scope, and naming

In this setting, ImageNet-F consists of the original ILSVRC images, face localizations, and face-obfuscated image variants generated from those localizations. The obfuscation methods studied in the main benchmark are blurring and overlaying. The benchmark modifies only the face regions of images that contain at least one detected or annotated face; images with no faces remain unchanged (Yang et al., 2021).

The paper reports that 1,431,093 images are annotated in ILSVRC, 243,198 images contain at least one face, and there are 562,626 total faces. Thus, about 17% of all images are affected by face obfuscation. The authors also note substantial category variation: 106 categories have faces in more than 50% of images, and 216 categories have faces in more than 25% of images. Examples of face-heavy categories include volleyball and military uniform.

Quantity	Value
Annotated ILSVRC images	1,431,093
Images containing at least one face	243,198
Total faces	562,626

The paper explicitly frames this resource as “a privacy-enhanced version of ILSVRC” and as “face-obfuscated versions of ILSVRC,” rather than as a formally named benchmark called ImageNet-F. A common misconception is therefore that ImageNet-F is an official in-paper benchmark title. The more precise description is that ImageNet-F is an externally convenient shorthand for the face-obfuscated ImageNet variant released through the project repository: https://github.com/princetonvisualai/imagenet-face-obfuscation.

2. Annotation pipeline and dataset construction

The construction process is a two-stage semi-automatic annotation pipeline. First, the authors run Amazon Rekognition face detection on all ILSVRC images. They selected Rekognition after comparing vendors and finding it strongest especially for small faces and multiple faces per image. Second, they refine those detections with human workers on Amazon Mechanical Turk, who adjust incorrect boxes, delete false positives, and add missed faces (Yang et al., 2021).

The detector alone produced both false positives and false negatives. Typical false positives were animal faces detected as human faces, whereas false negatives occurred under poor lighting or heavy occlusion. Workers were also instructed to annotate NSFW areas, although the released data contains very few such annotations relative to faces.

The face-box definition matters because obfuscation is applied directly to those boxes. Workers were instructed that a face box should cover at least the mouth, nose, eyes, forehead, and cheeks. This operational definition fixes the geometry of what is blurred or overlaid.

Quality control used a structured MTurk procedure. Each HIT contained 50 images, including 3 gold-standard images with verified ground truth. Workers began with 2 lives; mistakes on gold images cost a life, and losing both lives forced a restart of the HIT. The paper describes this as a “game-like” setup that improved quality.

Reported annotation quality was assessed on validation images from 20 categories deliberately chosen to stress the detector: 10 mammal categories likely to induce false positives and 10 categories with many faces likely to induce false negatives. Each category contributed 50 validation images, inspected manually by two graduate students. After human refinement, the average errors per 50 images were 1.25 false positives and 0.95 false negatives. Because these categories were intentionally difficult, the paper states that overall dataset error is expected to be lower.

The dataset statistics also characterize face density. Across face-containing images, 90.1% have fewer than 5 faces, but some images contain up to 100 faces, which is the cap of Amazon Rekognition, especially in sports or crowd scenes. The paper works on ILSVRC training and validation data and reports validation accuracy for evaluation; it does not discuss a separate public ILSVRC test-set construction.

3. Obfuscation mechanisms and formal definition

Two primary obfuscation methods define ImageNet-F. The first is face overlaying, in which the face box is covered by a constant patch using the average color of ILSVRC training data, $(0.485, 0.456, 0.406)$. The paper describes this as a gray shade. Appendix experiments also test red, green, and blue overlays, but the main benchmark uses the mean-color overlay (Yang et al., 2021).

The second is face blurring, implemented as a softened form of Gaussian blur designed to avoid hard box edges. Let the image be

$I \in D^{h \times w \times 3}, \quad D=[0,1].$

If there are $m$ face boxes, each box is enlarged by a margin proportional to its diagonal length $d_i$, and the union of the enlarged boxes is converted into a mask $M$. The paper then defines

$M_{\text{blurred}} = \text{Gaussian}(M, d_{\max}),$

$I_{\text{blurred}} = \text{Gaussian}(I, d_{\max}),$

where $d_{\max}$ is the maximum diagonal length among face boxes in the image. The final compositing step is

$$I_{\text{new}} = M_{\text{blurred}} \cdot I_{\text{blurred}} + (1 - M_{\text{blurred}}) \cdot I.$$

Using $M_{\text{blurred}}$ rather than a hard binary mask avoids sharp transitions between blurred and unblurred regions. This is the main formal specification introduced for the benchmark. The paper does not define a new ImageNet-F-specific training objective; standard supervised ImageNet training and standard downstream fine-tuning remain unchanged.

The comparison between the two obfuscation methods is consistent throughout the experiments: blurring is slightly better for accuracy, whereas overlaying removes more information and therefore causes somewhat larger drops. This suggests that, among the methods tested, blur is the more benchmark-friendly privacy intervention when preserving recognition utility is the dominant criterion.

4. Classification protocol and benchmark behavior

The main classification study trains models from scratch in three matched settings: original images for training and evaluation, face-blurred images for training and evaluation, and face-overlaid images for training and evaluation. The architectures evaluated are AlexNet, SqueezeNet, ShuffleNet, VGG11, VGG13, VGG16, VGG19, MobileNet, DenseNet121, DenseNet201, ResNet18, ResNet34, ResNet50, ResNet101, and ResNet152 (Yang et al., 2021).

The training setup is standard ImageNet practice: random $I \in D^{h \times w \times 3}, \quad D=[0,1].$0 crop and random horizontal flip for training, center crop for evaluation, batch size $I \in D^{h \times w \times 3}, \quad D=[0,1].$1, SGD with momentum $I \in D^{h \times w \times 3}, \quad D=[0,1].$2, weight decay $I \in D^{h \times w \times 3}, \quad D=[0,1].$3, and 90 epochs. The learning rate drops by a factor of 10 every 30 epochs. The initial learning rate is 0.01 for AlexNet, SqueezeNet, and VGG, and 0.1 for the other models. Each experiment is repeated 3 times, and the paper reports mean and standard error. Metrics are top-1 and top-5 validation accuracy.

The main empirical result is that face obfuscation produces only a slight reduction in ImageNet recognition accuracy. Averaged over the 15 main models, original training and evaluation gives 70.0 top-1 and 89.1 top-5; blurred gives 69.4 top-1 and 88.6 top-5; and overlaid gives 69.1 top-1 and 88.4 top-5. The average drops are therefore 0.7 top-1 and 0.4 top-5 for blurring, and 0.9 top-1 and 0.7 top-5 for overlaying. The abstract summarizes the overall degradation as $I \in D^{h \times w \times 3}, \quad D=[0,1].$4 (Yang et al., 2021).

Representative model-level results show the same pattern. For AlexNet, top-1 / top-5 accuracy changes from 56.0 / 78.8 on original images to 55.8 / 78.6 on blurred images and 55.5 / 78.2 on overlaid images. For ResNet50, the corresponding numbers are 75.5 / 92.5, 75.0 / 92.4, and 74.9 / 92.2. For ResNet152, they are 77.9 / 93.9, 77.3 / 93.7, and 77.0 / 93.3. For DenseNet201, they are 77.0 / 93.5, 76.6 / 93.2, and 76.1 / 92.9. The losses are consistently small, with overlaying somewhat worse than blurring.

The appendices also analyze mismatch protocols. When models pretrained on original ImageNet are evaluated on obfuscated validation images, the average top-5 accuracy decreases from 90.7 on original evaluation to 89.9 with blur and 89.6 with mean-color overlay; the average drop for blurred evaluation is denoted $I \in D^{h \times w \times 3}, \quad D=[0,1].$5. Conversely, when models are trained on blurred images and evaluated on original validation images, the average accuracy remains close: 70.0 $I \in D^{h \times w \times 3}, \quad D=[0,1].$6 69.1 in top-1 and 89.1 $I \in D^{h \times w \times 3}, \quad D=[0,1].$7 88.4 in top-5. This suggests that training on privacy-preserved data remains usable even when deployment images are unobfuscated.

5. Category-level effects and transfer learning

The paper analyzes why the global accuracy loss is small while some classes exhibit larger changes. One factor is the fraction of image area that is blurred. The category-wise correlation between accuracy drop and blurred fraction is $I \in D^{h \times w \times 3}, \quad D=[0,1].$8 for top-1 and $I \in D^{h \times w \times 3}, \quad D=[0,1].$9 for top-5, with very small $m$0-values, $m$1 and $m$2, respectively (Yang et al., 2021). Larger obfuscated area therefore tends to correlate with larger loss.

A stronger predictor is the overlap between the face region and the object of interest. Using ILSVRC localization bounding boxes, the authors compute the fraction of object area covered by faces and find correlations of $m$3 for top-1 and $m$4 for top-5. The most affected classes are those in which the target object is physically worn on or near the face. For mask, 24.84% of object area is covered by faces and the top-5 drop is 8.71%. For harmonica, 29.09% is covered and the top-5 drop is 8.93%. For snorkel, 30.51% is covered and the top-5 drop is 6.00%.

The paper also cautions that some large top-1 changes are artifacts of confusion among highly similar classes. A striking example is eskimo dog, whose top-1 accuracy changes from 50.8 to 38.0, while siberian husky changes from 46.3 to 63.2. Yet the top-5 changes are much smaller: 95.5 $m$5 95.1 for eskimo dog and 97.0 $m$6 97.2 for siberian husky. The authors therefore argue that top-5 is the more reliable metric for this analysis.

The downstream transfer results are central to the benchmark’s utility claims. The paper compares no pretraining, pretraining on original ImageNet, pretraining on blurred ImageNet, and pretraining on overlaid ImageNet, then fine-tunes on four tasks (Yang et al., 2021). On scene recognition on SUN, the differences are negligible: for ResNet34, 57.8 ± 0.0 with original pretraining, 57.9 ± 0.1 with blurred pretraining, and 57.8 ± 0.1 with overlaid pretraining; for ResNet18, 55.0 ± 0.2, 55.0 ± 0.1, and 55.1 ± 0.1. On object detection on PASCAL VOC, using Faster R-CNN with a ResNet50 backbone, mAP is 79.40 ± 0.31 for original pretraining, 79.29 ± 0.22 for blurred pretraining, and 79.39 ± 0.02 for overlaid pretraining. For the person class AP, the scores are 84.40 ± 0.14, 84.80 ± 0.50, and 84.47 ± 0.05.

On face attribute classification on CelebA, which is the most face-centric transfer test, results vary by architecture but remain competitive under obfuscated pretraining. For VGG11, mAP is 56.0 ± 0.7 with original pretraining, 57.4 ± 0.6 with blurred pretraining, and 58.1 ± 0.9 with overlaid pretraining. For VGG13, the corresponding numbers are 58.4 ± 0.6, 59.0 ± 0.5, and 58.2 ± 0.4. For AlexNet, the scores are 55.5 ± 0.7, 50.7 ± 0.8, and 52.5 ± 0.4. The paper’s summary claim is that pretraining on obfuscated ImageNet remains competitive even for a face-understanding task. On CIFAR-10, although the supplied excerpt notes OCR corruption in the table, the interpretation is explicit: obfuscated pretraining performs closely to original pretraining and does not hurt transfer.

6. Privacy model, limitations, and position within related ImageNet research

The privacy claim associated with ImageNet-F is deliberately limited. The paper states that face obfuscation provides “some privacy protection” and can enhance privacy, but it does not provide formal privacy guarantees (Yang et al., 2021). Identity may still be inferred by humans or machine models from cues outside the face, including height, clothing, and context. The benchmark therefore represents privacy enhancement rather than anonymity.

The authors also justify their use of simple local transformations. They choose blurring and overlaying because stronger guarantees are not available in this setting, sophisticated adversarial anonymization still lacks guarantees, and more aggressive privacy interventions may reduce dataset utility. Two ethical caveats are emphasized. First, face obfuscation can create a false sense of privacy because it is imperfect. Second, commercial face detection may exhibit racial or gender disparities, so relying on automatic detection alone could protect some groups less well; manual crowdsourcing refinement is intended to partially mitigate that risk. The paper further notes that faces are only one privacy attribute. Other sensitive information, including names, addresses, license plates, and other personal content, may remain visible.

Within the broader ImageNet literature, ImageNet-F should be distinguished from other ImageNet variants. “Do ImageNet Classifiers Generalize to ImageNet?” introduces ImageNetV2, a newly collected test set meant to replicate the original ImageNet validation distribution, and explicitly does not define ImageNet-F (Recht et al., 2019). That work is relevant as background on natural distribution shift, but it studies recollection and recuration rather than privacy-preserving face obfuscation. The distinction matters because ImageNet-F changes local face regions within ILSVRC, whereas ImageNetV2 changes the image sample itself.

A second contextual issue concerns evaluation procedures built on ImageNet-pretrained feature spaces. “The Role of ImageNet Classes in Fréchet Inception Distance” argues that FID computed in the standard Inception-V3 feature space is strongly driven by ImageNet class semantics and can reward alignment with ImageNet classification structure rather than human-perceived fidelity (Kynkäänniemi et al., 2022). This does not redefine ImageNet-F, whose primary experiments use top-1, top-5, and downstream task metrics, but it suggests caution for any auxiliary evaluation protocol around ImageNet-F that uses ImageNet-pretrained embeddings as supposedly generic perceptual features. A plausible implication is that ImageNet-based feature metrics may encode ImageNet semantic biases even when the nominal task is not ImageNet classification.

Taken together, these points locate ImageNet-F as a privacy-aware ILSVRC variant rather than a new formal task family. Its central contribution is to show that ImageNet can be made more privacy-aware by local face obfuscation while preserving most of its utility as both a classification benchmark and a pretraining corpus, provided that its limited privacy guarantees and benchmark-specific evaluation assumptions are kept explicit.