PatchMap: Spatial Benchmark for Patch Attacks

• PatchMap is a spatial benchmark designed to evaluate adversarial patch placements using exhaustive location sweeps on the ImageNet-1K dataset.
• It decouples patch content from selection and placement, rigorously measuring untargeted misclassification rates and confidence drops across varied patch sizes and locations.
• The benchmark offers reproducible, large-scale evaluations that provide actionable insights for developing location-aware defenses and advanced patch-placement strategies.

PatchMap is a spatially exhaustive benchmark for adversarial patch placement that pastes fixed, universal patches at every stride-2 location on ImageNet-1K validation images and records classification outcomes and confidences, thereby producing dense per-image, per-location vulnerability maps (Kimhi et al., 3 Aug 2025). In PatchMap v1.0, patch content is held fixed—ten 50×50 RGB universal adversarial patches from ImageNet-Patch, each optimized for a distinct target class—while the benchmark studies the selection of a 2D location and a scale on 224×224 images. Its primary reporting emphasizes untargeted misclassification rather than targeted success, even though the source patches are targeted (Kimhi et al., 3 Aug 2025).

1. Problem setting and conceptual separation

Adversarial patch attacks place a localized, printable pattern onto an image so that a downstream classifier’s prediction is corrupted. Such patches can be physically realized as stickers or posters and are constrained to compact spatial support; in PatchMap, the supported textures are square. PatchMap explicitly decouples two orthogonal design choices: patch content $p$, meaning texture, color, and shape, and patch selection and placement, meaning which fixed patch is used and where it is pasted (Kimhi et al., 3 Aug 2025).

This decoupling is central to the benchmark’s scope. In v1.0, no re-optimization is performed, patches are applied without rotation, and color is unconstrained but fixed per patch. The benchmark task is: given an image $x$, select a 2D location $l$ and a scale $s$ from a predefined set and paste the fixed patch $p$. The square side lengths are $50$, $25$, and $10$ pixels on $224\times224$ images, corresponding to areas of approximately $4.98\%$, $x$0, and $x$1 of the image, respectively. A common misconception is that PatchMap’s headline numbers are targeted-attack results; in fact, the primary benchmark metric is untargeted attack success, namely whether the patched image causes a misclassification relative to the clean-correct label (Kimhi et al., 3 Aug 2025).

2. Benchmark construction and data representation

PatchMap v1.0 is built on the 50,000-image ImageNet-1K validation split, resized to $x$2. It uses ten fixed universal patches from ImageNet-Patch, three square sizes, and a stride-2 grid of feasible patch centers over the image plane. The grid has $x$3 positions. This yields approximately

$x$4

potential placements (Kimhi et al., 3 Aug 2025).

The released benchmark is exhaustive over locations but optimized for tractable execution. By batching inference and discarding out-of-frame placements, PatchMap v1.0 actually runs approximately $x$5 forward passes. The abstract reports “over $x$6”; the released v1.0 dataset contains $x$7M+ predictions and confidences. The attacked model in v1.0 is a standard torchvision ResNet-50 pretrained on ImageNet, with no adversarial fine-tuning. For each $x$8 triple, PatchMap stores a NumPy array of shape $x$9: slice $l$0 holds predicted class indices as int16, and slice $l$1 holds softmax confidences as float32. The release comprises approximately $l$2M files and approximately $l$3 GB compressed. All benchmark evaluations can be reproduced on a single machine with at least $l$4 GB RAM, and the code regenerates figures with approximately $l$5 GPU-hours on one V100. A 6.5B-entry v2.0 with multiple backbones is designated as the full bench (Kimhi et al., 3 Aug 2025).

These design choices make PatchMap both exhaustive and usable. The exhaustive spatial sweep exposes location sensitivity that is invisible in single-placement or small-sample evaluations, while the stored predictions and confidences allow recomputation of downstream statistics without rerunning the entire inference pipeline.

3. Formalization, operators, and evaluation metrics

PatchMap formalizes patch application with an overwrite operator. Let $l$6 be an image, $l$7 the patch texture, and $l$8 a patch center. Let $l$9 be a binary mask that embeds a size-$s$0 square centered at $s$1, and let $s$2 be $s$3 zero-padded to image size and placed at $s$4. The patching operator is

$s$5

No alpha blending is used; the patch directly replaces the covered pixels (Kimhi et al., 3 Aug 2025).

Let $s$6 be the classifier’s softmax probabilities over $s$7 classes, with predicted class

$s$8

Given ground-truth label $s$9, the clean-correct set consists of images with $p$0. PatchMap defines untargeted success at location $p$1 by

$p$2

and targeted success toward target $p$3 by

$p$4

Top-1 accuracy drop for a set $p$5 is

$p$6

and confidence drop for the ground-truth class is

$p$7

Over a population of images, patches, and sizes, PatchMap defines the per-location susceptibility map

$p$8

and the confidence-drop map

$p$9

Its released tensors allow computing both quantities as averages over clean-correct images. The location-wise attack-success heat-map on clean-correct images is denoted $50$0. PatchMap also reports worst-location confidence drop per image,

$50$1

and spatial robustness via quantiles $50$2, which measures the fraction of images fooled on more than a $50$3-portion of locations. Metrics are reported with 95% bootstrap confidence intervals using 1,000 resamples. Because softmax confidences are stored, calibration shifts such as ECE and Brier can also be computed. A transfer matrix $50$4 is proposed for cross-model replay of scored locations, and v2.0 is intended to enable such analysis across architectures including ViT and ConvNeXt (Kimhi et al., 3 Aug 2025).

4. Empirical findings on spatial vulnerability

PatchMap’s principal empirical result is that vulnerability to adversarial patches is strongly location-dependent. Dense maps reveal stable spatial hot-spots where small patches cause confident misclassifications and large confidence drops. These hot-spots align strongly with object regions, as inferred by segmentation, and persist across images and patch sizes (Kimhi et al., 3 Aug 2025).

The benchmark also shows that very small patches remain potent. For the patch “Plate,” mean optimal ASR—the fraction of images for which some location causes misclassification—is $50$5 at $50$6, $50$7 at $50$8, and $50$9 at $25$0. For “Electric Guitar,” the corresponding values are $25$1, $25$2, and $25$3. Confidence collapse is similarly pronounced: average worst-location confidence drops on the ground-truth class for “Plate” are $25$4, $25$5, and $25$6 across the same sizes, while “Electric Guitar” yields $25$7, $25$8, and $25$9 (Kimhi et al., 3 Aug 2025).

Spatial robustness curves further refine the picture. Some patches, such as “Plate” at $10$0, achieve strong misclassification rates even across the worst approximately $10$1 of locations, whereas others, such as “Guitar” at $10$2, achieve higher peak effects on the top approximately $10$3 of locations. PatchMap also reports that higher segmentation object-confidence correlates with larger classifier confidence drop $10$4 at those locations, with strong correlation for scores above $10$5, especially for smaller patches. This suggests that the benchmark is capturing structure tied to object-centric evidence rather than arbitrary spatial artifacts (Kimhi et al., 3 Aug 2025).

5. Segmentation-guided placement heuristic

PatchMap uses its dense spatial analysis to motivate a simple zero-gradient placement heuristic. Let $10$6 be per-pixel softmax from a segmentation model, with channel $10$7 denoting background. Define object-confidence by

$10$8

Given a binary patch mask $10$9, the heuristic slides $224\times224$0 over $224\times224$1 and selects the center maximizing the sum of object-confidence under the patch:

$224\times224$2

The patch is then placed at $224\times224$3, without using gradients or queries from the attacked classifier. In the reported study, segmentation cues come from DeepLab-v3+ with a ResNet-101 backbone pretrained on PASCAL VOC 2012, and the attacked classifiers are ResNet-18, ResNet-50, MobileNet-V2, EfficientNet-B1, and a Fast Adversarially Trained ResNet-50 (Kimhi et al., 3 Aug 2025).

The heuristic improves attack success consistently. For “Plate” at $224\times224$4 (approximately $224\times224$5 area), ASR on ResNet-50 is $224\times224$6 for segmentation-guided placement, versus $224\times224$7 for Random and $224\times224$8 for Fixed. On the adversarially trained ResNet-50, the values are $224\times224$9, $4.98\%$0, and $4.98\%$1. On ResNet-18 they are $4.98\%$2, $4.98\%$3, and $4.98\%$4; on MobileNet-V2, $4.98\%$5, $4.98\%$6, and $4.98\%$7; and on EfficientNet-B1, $4.98\%$8, $4.98\%$9, and $x$00. The average improvement is reported as $x$01 percentage points versus Random and $x$02 percentage points versus Fixed. Gains remain visible for smaller patches: for “Plate” at $x$03, ResNet-50 reaches $x$04 versus $x$05 versus $x$06, and for “Electric Guitar” at $x$07, ResNet-50 reaches $x$08 versus $x$09 versus $x$10. On ResNet-18 with $x$11 patches, segmentation-guided placement reaches $x$12 ASR versus $x$13 for Random and $x$14 for Fixed (Kimhi et al., 3 Aug 2025).

The reported interpretation is direct: the heuristic places the patch over segmentation-confident object regions, which typically contain pixels most diagnostic for classification. PatchMap’s dense maps and $x$15 analyses confirm that occluding these regions produces larger confidence collapses and misclassifications. The study does not compare against gradient-based or reinforcement-learning location optimizers; PatchMap is intended to function as a common benchmark for such methods rather than as a closed comparison set (Kimhi et al., 3 Aug 2025).

6. Scope, limitations, reproducibility, and research implications

PatchMap is consequential primarily because it makes the spatial dimension of adversarial patch attacks measurable. The results indicate that robust training alone does not neutralize strategic location choice: on Fast Adversarially Trained ResNet-50, segmentation-guided placement still improves ASR, for example from $x$16 to $x$17 for “Plate” at $x$18. The paper therefore argues that defenses should be location-aware. Suggested directions include training-time data augmentation that pastes universal patches over segmentation-confident regions, monitoring calibration shifts such as ECE and Brier under patching, and training spatial detectors on PatchMap hot-spot statistics to flag suspicious occlusions (Kimhi et al., 3 Aug 2025).

The benchmark also has clear limitations. Patch content is limited to ten fixed universal textures from ImageNet-Patch; no rotation or non-square shapes are considered. The evaluation domain is ImageNet-1K validation at $x$19, so object-layout biases and resolution effects may influence where hot-spots appear. Architecture coverage in the exhaustive v1.0 maps is ResNet-50-specific, although the segmentation-guided study extends to four additional classifiers and one adversarially trained model. Exhaustive spatial sweeps are compute-intensive, even if PatchMap mitigates this by releasing precomputed tensors. Finally, although the source patches originate from targeted optimization, the benchmark’s primary emphasis remains untargeted misclassification; targeted success rates are derivable but not the default report (Kimhi et al., 3 Aug 2025).

Reproducibility is a core feature of the release. PatchMap v1.0, comprising approximately 100M predictions, is hosted on HuggingFace under CC-BY-4.0, and loaders, plotting scripts, and leaderboard scripts are provided on GitHub. The file format is per-$x$20 .npz, storing predicted labels and softmax confidences in a $x$21 tensor, so users can stream batches and compute metrics without monolithic archives. The planned v2.0, described as a 6.5B-entry full bench with multiple backbones, is intended to extend PatchMap from a single-model exhaustive release into a broader substrate for location-aware defenses, adaptive attacks, and cross-architecture transfer analysis (Kimhi et al., 3 Aug 2025).