Art-Attack: Evolutionary Black-Box Attacks

• Art-Attack is an adversarial method that recasts image attacks as an evolutionary art problem using overlapping transparent geometric shapes under l∞ constraints.
• It employs a gradient-free, (1+1) evolutionary strategy with an adaptive mutation operator to efficiently explore a low-dimensional, shape-based search space.
• Empirical evaluations on CIFAR-10 models demonstrate that Art-Attack achieves higher success rates and query efficiency, particularly with optimized circle-based perturbations.

Art-Attack most specifically denotes a score-based black-box adversarial attack that generates targeted adversarial examples by evolving layered transparent geometric primitives rather than optimizing pixels directly. In the formulation introduced in "Art-Attack: Black-Box Adversarial Attack via Evolutionary Art" (Williams et al., 2022), the attacker can query a classifier and observe class probabilities, but has no access to gradients, architecture, or training data. The method casts adversarial example generation as an evolutionary art problem, in which a candidate perturbation is represented as a set of overlapping shapes and iteratively improved under an $l_\infty$-bounded constraint. In later literature, related uses of the phrase have appeared in contexts such as attentional face impersonation attacks, ASCII-art toxicity evasion, and benign-prompt red-teaming for text-to-image systems, but the proper noun Art-Attack refers most directly to the evolutionary-art attack on image classifiers (Williams et al., 2022).

1. Definition and problem setting

Art-Attack is formulated for image classification in the black-box setting. The targeted model is written as

$$f:\mathcal{X}\subseteq \mathbb{R}^d \to \mathbb{R}^K,$$

and the attacker is assumed to query the model with an input image and observe returned class probabilities. The objective in the targeted case is to find a perturbation $\delta$ such that the perturbed image is classified as a chosen target class $c$: $$\arg\max_{r=1,\cdots,K} f_r(x+\delta)=c,\qquad \delta\in\mathcal{T},\ x+\delta\in\mathcal{X}. \tag{1}$$ The paper also gives a corresponding optimization problem using a score-based loss, and defines the untargeted version as

$$\arg\max_{r=1,\cdots,K} f_r(x+\delta)\neq y, \qquad \delta\in\mathcal{T},\ x+\delta\in\mathcal{X}. \tag{3}$$

The attack follows an $l_\infty$-bounded threat model: $\|x_{\text{adv}-x\|_\infty \le \epsilon. \tag{4}$

A central design choice is that Art-Attack does not use substitute models or gradient estimation. Instead, it performs a gradient-free search in a structured parameter space. This distinguishes it from substitute-model attacks, which depend on transferability, and from gradient-estimation methods, which estimate gradients in a high-dimensional pixel space (Williams et al., 2022). This suggests that the method is intended as a query-efficient alternative to more conventional black-box optimization strategies.

2. Optimization objective and evolutionary-art representation

The attack uses a log-probability objective that increases target-class confidence while suppressing other classes: $\max_{\delta \in \mathbb{R}^{d} \log\big(f_c(x+\delta)\big) -\log\Big(\sum_{i\neq c} f_i(x+\delta)\Big), \qquad \|x_{\text{adv}-x\|_\infty \le \epsilon. \tag{5}$ The logarithm is used for numerical stability (Williams et al., 2022).

Its main contribution is to reformulate adversarial example generation as evolutionary art. Rather than optimizing a perturbation over all pixels, the attack evolves a single candidate solution composed of $N$ overlapping transparent shapes. Each shape is parameterized by continuous values in $$f:\mathcal{X}\subseteq \mathbb{R}^d \to \mathbb{R}^K,$$0, and the full representation is

$$f:\mathcal{X}\subseteq \mathbb{R}^d \to \mathbb{R}^K,$$1

where $$f:\mathcal{X}\subseteq \mathbb{R}^d \to \mathbb{R}^K,$$2 depends on the chosen shape family (Williams et al., 2022).

The search loop is simple and explicit. The method initializes one solution $$f:\mathcal{X}\subseteq \mathbb{R}^d \to \mathbb{R}^K,$$3 uniformly at random, renders it into an adversarial image $$f:\mathcal{X}\subseteq \mathbb{R}^d \to \mathbb{R}^K,$$4, queries the target model to obtain a fitness value, mutates $$f:\mathcal{X}\subseteq \mathbb{R}^d \to \mathbb{R}^K,$$5 to obtain a child $$f:\mathcal{X}\subseteq \mathbb{R}^d \to \mathbb{R}^K,$$6, and keeps whichever candidate has the better objective value. The process repeats until attack success or exhaustion of the query budget. The paper characterizes this as essentially a $$f:\mathcal{X}\subseteq \mathbb{R}^d \to \mathbb{R}^K,$$7-style evolutionary search, but operating in a shape-based image-generation space rather than directly in pixel space (Williams et al., 2022).

This structured representation is significant because a relatively small number of continuous parameters can induce complex image changes. A plausible implication is that Art-Attack reduces the dimensionality of black-box adversarial search while preserving enough expressivity to construct effective targeted perturbations.

3. Shape families and image construction

Art-Attack studies three families of transparent geometric primitives: circles, triangles, and rectangles / squares (Williams et al., 2022).

For circles, each primitive $$f:\mathcal{X}\subseteq \mathbb{R}^d \to \mathbb{R}^K,$$8 contains center coordinates, radius, RGB color, and transparency $$f:\mathcal{X}\subseteq \mathbb{R}^d \to \mathbb{R}^K,$$9. If the image has height $\delta$0 and width $\delta$1, the center and radius are mapped as follows:

• center $\delta$2
• radius $\delta$3, with

$\delta$4

The paper sets $\delta$5, which explicitly limits circle size (Williams et al., 2022).

For triangles, each primitive $\delta$6 contains six parameters for three vertices, three for RGB, and one for transparency. The vertices are defined as

$\delta$7

For rectangles / squares, each shape has eight parameters: four for corners, three for RGB, and one for transparency (Williams et al., 2022).

The final adversarial image is constructed by layering all shapes: $\delta$8 followed by projection onto the valid pixel box and perturbation constraint: $\delta$9

The choice of parameterization is not merely cosmetic. The paper reports that circles are the most effective shape type, especially when used in larger numbers, and attributes part of this to the explicit radius limit, which makes each perturbation patch relatively sparse and localized (Williams et al., 2022). This suggests that localized structured patches interact with classifier vulnerabilities differently from denser polygonal overlays.

4. Mutation operator and search dynamics

The mutation operator is the main mechanism by which Art-Attack explores the search space. Given a parent solution $c$0, the algorithm copies it, chooses a random index $c$1, samples the number of changed parameters as

$c$2

occasionally rolls a subsequence of shapes with probability $c$3, mutates selected parameters either by resampling from $c$4 with probability $c$5 or by adding noise from $c$6, and clips the result back to $c$7 (Williams et al., 2022).

The mutation rate is adaptive. The algorithm tracks $c$8, the number of consecutive iterations without improvement. If the objective improves, $c$9; otherwise $$\arg\max_{r=1,\cdots,K} f_r(x+\delta)=c,\qquad \delta\in\mathcal{T},\ x+\delta\in\mathcal{X}. \tag{1}$$0 increases, and the mutation rate is updated as

$$\arg\max_{r=1,\cdots,K} f_r(x+\delta)=c,\qquad \delta\in\mathcal{T},\ x+\delta\in\mathcal{X}. \tag{1}$$1

with $$\arg\max_{r=1,\cdots,K} f_r(x+\delta)=c,\qquad \delta\in\mathcal{T},\ x+\delta\in\mathcal{X}. \tag{1}$$2 and $$\arg\max_{r=1,\cdots,K} f_r(x+\delta)=c,\qquad \delta\in\mathcal{T},\ x+\delta\in\mathcal{X}. \tag{1}$$3 (Williams et al., 2022). This increases exploration under stagnation.

The paper interprets this operator as supporting both small local edits and larger randomized jumps. That interpretation aligns with the observed role of adaptive mutation in escaping local optima. A plausible implication is that the success of Art-Attack depends not only on a low-dimensional representation, but also on the ability to shift between exploitation and exploration without relying on explicit gradient information.

The overall algorithm is correspondingly compact: initialize $$\arg\max_{r=1,\cdots,K} f_r(x+\delta)=c,\qquad \delta\in\mathcal{T},\ x+\delta\in\mathcal{X}. \tag{1}$$4, evaluate the loss, repeatedly mutate to obtain $$\arg\max_{r=1,\cdots,K} f_r(x+\delta)=c,\qquad \delta\in\mathcal{T},\ x+\delta\in\mathcal{X}. \tag{1}$$5, keep the better candidate, and stop when a successful adversarial example is found or when the maximum number of iterations is reached (Williams et al., 2022).

5. Experimental evaluation and empirical findings

The primary evaluation targets three CIFAR-10 classifiers:

• VGG-16 with approximately 93% test accuracy
• Network in Network (NiN) with approximately 85%
• All-CNN with approximately 87% (Williams et al., 2022)

The protocol uses 100 correctly classified CIFAR-10 test images. For each image, the attack is run against all $$\arg\max_{r=1,\cdots,K} f_r(x+\delta)=c,\qquad \delta\in\mathcal{T},\ x+\delta\in\mathcal{X}. \tag{1}$$6 incorrect target labels. The perturbation bound is

$$\arg\max_{r=1,\cdots,K} f_r(x+\delta)=c,\qquad \delta\in\mathcal{T},\ x+\delta\in\mathcal{X}. \tag{1}$$7

and the query budget is 10,000 model queries per attack instance (Williams et al., 2022).

The reported metrics are Targeted Attack Success Rate (ASR), Untargeted ASR, and the average number of queries needed to find a successful adversarial example. The paper compares Art-Attack with GenAttack, Ilyas18, AutoZOOM, and One-Pixel Attack. For GenAttack and Ilyas18, the search space is reduced to an embedding of size $$\arg\max_{r=1,\cdots,K} f_r(x+\delta)=c,\qquad \delta\in\mathcal{T},\ x+\delta\in\mathcal{X}. \tag{1}$$8, following AutoZOOM’s recommendation (Williams et al., 2022).

The paper’s quantitative conclusions are qualitative in form but precise in direction. Art-Attack outperforms both GenAttack and Ilyas18 on all three CIFAR-10 models. Even in its worst shape setting, it still beats these baselines in both success rate and query efficiency. In its strongest setting, it can succeed with up to 50% fewer queries than GenAttack and Ilyas18 (Williams et al., 2022). Against AutoZOOM, the paper reports that AutoZOOM can often find adversarial examples, but in the paper’s setup it fails to produce examples satisfying

$$\arg\max_{r=1,\cdots,K} f_r(x+\delta)=c,\qquad \delta\in\mathcal{T},\ x+\delta\in\mathcal{X}. \tag{1}$$9

within the budget. Against One-Pixel Attack, Art-Attack achieves higher success rates on all three models, although One-Pixel Attack can be more query-efficient in the special case $$\arg\max_{r=1,\cdots,K} f_r(x+\delta)\neq y, \qquad \delta\in\mathcal{T},\ x+\delta\in\mathcal{X}. \tag{3}$$0 (Williams et al., 2022).

A central empirical result is the parameter study on the number and type of shapes. Performance improves dramatically when increasing the number of shapes from very small counts, such as circles from 5 to 20, and success rate increases almost exponentially at first as more overlapping shapes are added. However, too many shapes can hurt by enlarging the search space; performance can plateau or drop, especially for triangles and rectangles (Williams et al., 2022). Circles are especially notable because increasing their number can improve both success rate and efficiency.

These findings place Art-Attack within a broader class of structured black-box attacks that trade full pixel freedom for more constrained but more efficient search spaces. Related work on imperceptible black-box image attacks later emphasized restricting perturbations to salient regions and recursively refining them for improved perceptual quality (Dai et al., 2022). Art-Attack does not use saliency, but both approaches share the premise that the geometry of the search space matters for black-box attack effectiveness.

6. Broader context, related usages, and significance

The term Art-Attack is most properly reserved for the evolutionary-art method of (Williams et al., 2022), but later literature contains several conceptually adjacent uses of “art” or artistic structure in adversarial and safety research.

In face recognition, the attentional adversarial attack generative network $$\arg\max_{r=1,\cdots,K} f_r(x+\delta)\neq y, \qquad \delta\in\mathcal{T},\ x+\delta\in\mathcal{X}. \tag{3}$$1 performs a targeted, impersonation-style face attack by generating a face image that remains visually close to the source but is recognized as a specific target identity. Its architecture combines a conditional variational autoencoder, attention modules, and the face recognition network as an adversarial “third player” (Song et al., 2018). This is a different attack family from Art-Attack, but both replace direct pixel perturbation with structured generative manipulation.

In text-to-image safety, ART denotes Automatic Red-teaming, a framework that discovers safe prompts that still elicit harmful images by coupling a Writer Model with a Guide Model and separate prompt/image judges (Li et al., 2024). The semantic overlap with “Art-Attack” is terminological rather than methodological: ART attacks vulnerabilities in generative-image systems through prompt rewriting rather than geometric perturbation.

In language-model moderation, ASCII-art attacks encode toxic phrases as large visual letterforms so that humans can read them while models fail to recover them reliably. The paper "Read Over the Lines" reports a perfect Attack Success Rate of 1.0 across ten models for its ASCII-art attack families (Berezin et al., 2024). This is again distinct from Art-Attack, but it extends the same broad intuition that structured human-readable representations can expose model weaknesses that are not easily captured by standard token- or pixel-level defenses.

A broader contemporary development is the rise of automated attack discovery. Claudini uses an autoresearch pipeline to discover white-box adversarial attack algorithms that significantly outperform all existing (30+) methods in LLM jailbreaking and prompt injection evaluations (Panfilov et al., 25 Mar 2026). This later work differs sharply from Art-Attack in domain and threat model, but it reinforces a common theme: adversarial performance can improve substantially when the search space is explicitly engineered, whether through geometric primitives, agent-driven algorithm recombination, or multimodal feedback loops.

Art-Attack therefore occupies a specific place in adversarial machine learning. It is a black-box targeted image attack that demonstrates the effectiveness of shape-parameter search under strict query and norm constraints. Its main significance is not only that it outperforms several baselines on CIFAR-10, but that it shows how adversarial optimization can be recast as structured synthesis rather than direct numerical perturbation. This suggests a broader design principle: in black-box settings, carefully chosen representations may be as important as the optimizer itself (Williams et al., 2022).