Pixle: Black-Box Pixel Rearrangement Attack

• Pixle is a sparse, black-box adversarial attack that rearranges pixels rather than adding noise, preserving the image’s original color values.
• It employs a restart-iterative search with random patch remapping to efficiently reduce model confidence with minimal pixel changes.
• Empirical results demonstrate high success rates (up to 100%) across datasets, highlighting a trade-off between query efficiency and perturbation sparsity.

Pixle is a black-box adversarial attack on image classifiers that constructs adversarial examples by rearranging a small number of pixels rather than adding arbitrary noise. In the original formulation, the attacked model is queried only through its outputs, and the attack operates under a sparse, effectively $L_0$-type perturbation model in which only a limited set of pixel locations is altered. Pixle is positioned in the adversarial-ML literature as a gradient-free, query-based method that is visually subtle because it preserves pixel values while modifying their spatial arrangement; later work has used it both as a comparative black-box baseline and as a direct threat model in domains such as Arabic handwriting recognition (Pomponi et al., 2022).

1. Definition and positioning

Pixle was introduced as “a fast and effective black-box attack based on rearranging pixels” (Pomponi et al., 2022). Its defining premise is that misclassification can often be induced without changing the set of colors present in an image: instead of synthesizing additive perturbations, the attack moves existing pixel values to new positions. The original paper explicitly frames this as a black-box attack that requires no access to model architecture, gradients, weights, or training procedure, while assuming query access to model outputs (Pomponi et al., 2022).

Within the broader taxonomy of adversarial attacks, Pixle belongs to the family of sparse image-space attacks. It is closely related to $L_0$-style perturbation models because success is driven by changing a small number of pixels rather than distributing small-amplitude noise across the full image. Relative to One-Pixel and ScratchThat, the attack space is structural rather than additive: One-Pixel changes a few pixel values, ScratchThat draws curve-like perturbations, and Pixle instead reuses the image’s own pixels by relocating them (Pomponi et al., 2022).

Later literature retains this characterization. A comparative benchmark of black-box affine attacks cites Pixle as a black-box, gradient-free adversarial baseline with visually distinct perturbations based on rearranging pixels, and an Arabic handwriting recognition study describes it as an attack that “iteratively rearranges the pixel values, maintaining natural image statistics” while manipulating specific pixels to mislead the model (Echim et al., 8 Sep 2025, Khayati et al., 18 May 2026).

2. Threat model and optimization objective

In the original formalization, the classifier is written as

$$f : x \in [0,1]^{3 \times w \times h} \rightarrow \mathbb{R}^c,$$

with predicted label

$\hat{y}(x) = \arg\max_i f_i(x).$

The attack starts from an image $x$ that is correctly classified, $\hat{y}(x)=y$, and seeks an adversarial image $\overline{x}$ that changes the prediction. For an untargeted attack, the objective is

$\arg\max_i f_i(\overline{x}) \neq y,$

while for a targeted attack toward class $\overline{y}\neq y$, the objective is

$\arg\max_i f_i(\overline{x}) = \overline{y}.$

The paper also writes the search as a constrained optimization problem over perturbed images $L_0$0, with an $L_0$1 bound and Pixle specializing to the sparse $L_0$2 case (Pomponi et al., 2022).

The loss used for the search is explicitly score-based. In the untargeted case, Pixle minimizes the probability of the correct class,

$L_0$3

and in the targeted case it maximizes the target-class probability through

$L_0$4

This makes the method a black-box attack with probability access rather than a gradient-based white-box procedure (Pomponi et al., 2022).

Subsequent papers preserve essentially the same threat model at a higher level of abstraction. The Arabic handwriting recognition study emphasizes realistic black-box conditions in which the attacker lacks access to architecture, parameters, training data, and gradients, but has query access to the model’s input-output behavior; it also stresses a perceptual constraint, namely that the perturbed handwritten characters remain structurally intact and almost imperceptible to the human eye (Khayati et al., 18 May 2026).

3. Rearrangement mechanism and search procedure

Pixle samples a small patch of adjacent pixels and maps those pixels to new positions in the image. The image is denoted

$L_0$5

and a patch is represented as

$L_0$6

where $L_0$7 is the top-left origin and $L_0$8 are patch width and height. The associated set of patch coordinates is

$L_0$9

A mapping function

$$f : x \in [0,1]^{3 \times w \times h} \rightarrow \mathbb{R}^c,$$0

assigns each source position in the patch to a destination coordinate in the image (Pomponi et al., 2022).

The core perturbation step constructs the adversarial image by overwriting destination locations with source-patch values. In the implementation described in the paper, for each $$f : x \in [0,1]^{3 \times w \times h} \rightarrow \mathbb{R}^c,$$1, if $$f : x \in [0,1]^{3 \times w \times h} \rightarrow \mathbb{R}^c,$$2, then

$$f : x \in [0,1]^{3 \times w \times h} \rightarrow \mathbb{R}^c,$$3

The paper also notes a swap variant, but the main experiments use the overwrite-style formulation because it tends to keep the $$f : x \in [0,1]^{3 \times w \times h} \rightarrow \mathbb{R}^c,$$4 change small (Pomponi et al., 2022).

Several mapping strategies are defined. A random mapping chooses destination locations uniformly at random and is the default in the main experiments. Other variants use similarity or dissimilarity in color space, either deterministically or via a distribution biased toward similar or dissimilar pixels. The paper reports that distribution-based mappings can reduce iterations and $$f : x \in [0,1]^{3 \times w \times h} \rightarrow \mathbb{R}^c,$$5 relative to deterministic similarity rules, but the main benchmark comparisons use the random mapping because of its simplicity and speed (Pomponi et al., 2022).

Two search schemes are described. The recommended version is a restart-iterative algorithm with an outer loop over restarts and an inner loop over iterations. At each iteration it samples a patch, builds a candidate adversarial image by applying the mapping, queries the model, and keeps the candidate if it reduces the true-class probability. After each restart, the best candidate from that restart becomes the new baseline. A simpler iterative algorithm without restarts is also presented, but the paper reports that the restart-iterative variant achieves substantially higher success at comparable budgets (Pomponi et al., 2022).

The attack budget is controlled indirectly through patch size, maximum restarts, maximum iterations per restart, and early stopping upon success. In the original experiments, patch side lengths of $$f : x \in [0,1]^{3 \times w \times h} \rightarrow \mathbb{R}^c,$$6, $$f : x \in [0,1]^{3 \times w \times h} \rightarrow \mathbb{R}^c,$$7, or $$f : x \in [0,1]^{3 \times w \times h} \rightarrow \mathbb{R}^c,$$8 are studied. Larger patches reduce the number of required queries but increase $$f : x \in [0,1]^{3 \times w \times h} \rightarrow \mathbb{R}^c,$$9; smaller patches achieve sparser perturbations but require more search (Pomponi et al., 2022).

4. Original empirical results

The original evaluation uses 1000 correctly classified images per dataset across CIFAR10, TinyImageNet, and ImageNet, with ResNet and VGG models. In the main comparison, Pixle is evaluated against One-Pixel and ScratchThat under a restart-iterative configuration with up to 100 restarts, 50 iterations per restart, and patch side $\hat{y}(x) = \arg\max_i f_i(x).$0 (Pomponi et al., 2022).

On CIFAR10, Pixle achieves a $\hat{y}(x) = \arg\max_i f_i(x).$1 success rate on both ResNet18 and VGG11. For ResNet18, the paper reports $\hat{y}(x) = \arg\max_i f_i(x).$2 queries on average with $\hat{y}(x) = \arg\max_i f_i(x).$3; for VGG11, it reports $\hat{y}(x) = \arg\max_i f_i(x).$4 queries with $\hat{y}(x) = \arg\max_i f_i(x).$5. These results outperform One-Pixel in success rate and query count, and outperform ScratchThat in query count while matching or exceeding its success (Pomponi et al., 2022).

On TinyImageNet, Pixle reports $\hat{y}(x) = \arg\max_i f_i(x).$6 success on ResNet50 with $\hat{y}(x) = \arg\max_i f_i(x).$7 queries and $\hat{y}(x) = \arg\max_i f_i(x).$8, and $\hat{y}(x) = \arg\max_i f_i(x).$9 success on VGG16 with $x$0 queries and $x$1. On ImageNet, it reports $x$2 success on ResNet50 with $x$3 queries and $x$4, and $x$5 success on VGG16 with $x$6 queries and $x$7 (Pomponi et al., 2022).

The same paper also evaluates targeted attacks on CIFAR10 and states that Pixle achieves near-perfect success on almost all source-target class pairs, whereas One-Pixel often fails and ScratchThat is perfect or near-perfect on only about half of the pairs (Pomponi et al., 2022).

A patch-size study clarifies the sparsity-efficiency trade-off. For CIFAR10 on ResNet18, patch side $x$8 yields $x$9 success with $\hat{y}(x)=y$0 queries and $\hat{y}(x)=y$1; patch side $\hat{y}(x)=y$2 yields $\hat{y}(x)=y$3 success with $\hat{y}(x)=y$4 queries and $\hat{y}(x)=y$5; patch side $\hat{y}(x)=y$6 yields $\hat{y}(x)=y$7 success with $\hat{y}(x)=y$8 queries and $\hat{y}(x)=y$9. This directly exhibits the design trade-off between query efficiency and perturbation sparsity (Pomponi et al., 2022).

Qualitatively, the paper highlights an ImageNet example misclassified with $\overline{x}$0, illustrating that even a one-pixel rearrangement can suffice. The paper’s interpretation is that images often already contain the pixels needed to induce misclassification; Pixle merely relocates them to a more adversarial spatial arrangement (Pomponi et al., 2022).

5. Later applications and comparative uses

Later work has used Pixle in two notably different ways: as an external baseline for black-box attack benchmarking and as a direct attack model in application-specific security studies.

A 2025 benchmark of affine and genetic black-box attacks includes Pixle as one of two external black-box adversarial methods used for comparison, alongside Square Attack. Its role is explicitly qualitative rather than systematic: the paper compares Swin Transformer V2 predictions on original Caltech-256 images and on images perturbed by ATA, AGA, Square Attack, and Pixle, but does not report full dataset-level Pixle metrics or parameter sweeps. In that qualitative comparison, the model classifies all 12 original images correctly, while adversarial success is obtained in 6 of 12 cases for ATA, 8 of 12 for AGA, 5 of 12 for Square Attack, and 4 of 12 for Pixle (Echim et al., 8 Sep 2025).

That paper’s most specific characterization of Pixle is visual. It states that “Pixle output is only an interchange of image pixels,” contrasting it with Square Attack’s square patches and stripes and with ATA/AGA’s affine transformations. The paper cites a “mars” example in which a slight change of pixels leads the model to classify the image as “galaxy,” suggesting that rearranged spatial correlations can create semantically misleading background patterns for the classifier (Echim et al., 8 Sep 2025).

A 2026 study on Arabic handwriting recognition uses Pixle more directly and finds it to be the most damaging attack in the evaluation. On IFHCDB, Pixle reduces MobileNet attack accuracy to $\overline{x}$1 with ASR $\overline{x}$2, SqueezeNet to $\overline{x}$3 with ASR $\overline{x}$4, ShuffleNet to $\overline{x}$5 with ASR $\overline{x}$6, and MnasNet to $\overline{x}$7 with ASR $\overline{x}$8. On AHCD, it yields attack accuracy $\overline{x}$9 with ASR $\arg\max_i f_i(\overline{x}) \neq y,$0 for MobileNet, $\arg\max_i f_i(\overline{x}) \neq y,$1 with ASR $\arg\max_i f_i(\overline{x}) \neq y,$2 for ShuffleNet, and $\arg\max_i f_i(\overline{x}) \neq y,$3 with ASR $\arg\max_i f_i(\overline{x}) \neq y,$4 for MnasNet; SqueezeNet on AHCD is the main exception, with attack accuracy $\arg\max_i f_i(\overline{x}) \neq y,$5 and ASR $\arg\max_i f_i(\overline{x}) \neq y,$6 (Khayati et al., 18 May 2026).

That handwriting study emphasizes a specific property of Pixle that is consistent with the original attack design: despite extremely high attack success, the perturbations preserve the structural integrity of the characters and remain almost imperceptible to the human eye. This suggests that pixel rearrangement is especially well matched to domains in which global shape is a dominant human cue but local spatial correlations remain highly predictive for ConvNet classifiers (Khayati et al., 18 May 2026).

6. Limitations, defenses, and related concepts

The original Pixle paper identifies several practical limitations. The method assumes access to output probabilities, or at least to a score such as the true-class probability, so its direct applicability to strict decision-only settings is not established. It is also stochastic and search-based, which leads to high variance in both query counts and achieved $\arg\max_i f_i(\overline{x}) \neq y,$7. Moreover, the algorithm stops at the first successful adversarial example rather than solving for a globally minimal rearrangement (Pomponi et al., 2022).

Defense evaluation is limited in the original work. The paper discusses adversarial training, defensive distillation, ensemble methods, and preprocessing defenses such as noise reduction, color squeezing, and spatial smoothing, but does not systematically test any of them against Pixle. Later papers likewise leave major robustness questions open: the affine-attack benchmark does not evaluate defended models under Pixle, and the Arabic handwriting study does not analyze explicit query budgets, perturbation norms, or perceptual metrics such as SSIM or PSNR for Pixle-generated examples (Pomponi et al., 2022, Echim et al., 8 Sep 2025, Khayati et al., 18 May 2026).

Conceptually, Pixle is significant because it broadens the class of adversarial perturbations that must be considered. The attack shows that the relevant adversarial degrees of freedom are not limited to additive noise inside an $\arg\max_i f_i(\overline{x}) \neq y,$8 or $\arg\max_i f_i(\overline{x}) \neq y,$9 ball. Spatial rearrangement of existing content can also reliably cross decision boundaries. A plausible implication is that defenses tuned only to additive perturbations may leave models exposed to sparse structural changes in the input.

A recurring source of confusion is nomenclature. In contemporary arXiv literature, “Pixle” denotes the pixel-rearrangement black-box adversarial attack (Pomponi et al., 2022). Distinct works with similar names include “PIXEL,” an activation-steering method for LLMs (Yu et al., 11 Oct 2025), and “Perceive Everything as Pixels” with the PixelWorld benchmark, which concerns unified pixel-based perception rather than adversarial image attacks (Lyu et al., 31 Jan 2025). These are separate lines of research and should not be conflated with Pixle as used in adversarial machine learning.