ScoreAdv: Diffusion-Based Adversarial Framework

• ScoreAdv is a diffusion-based framework that generates unrestricted adversarial examples without traditional l_p norm constraints.
• The method integrates gradient-based adversarial guidance with classifier-free diffusion sampling to maintain natural image quality while targeting specific labels.
• It leverages ScoreCAM for interpretable saliency injection and noise optimization to balance attack efficacy and visual realism.

ScoreAdv is a training-free framework for generating unrestricted adversarial examples with diffusion models, introduced as a method for producing natural-looking targeted attacks that are not constrained by an $\ell_p$ norm and can be generated in unlimited quantity (Huang et al., 8 Jul 2025). It is formulated as score-based targeted sampling: a pretrained diffusion model maintains samples on the natural-image manifold, while gradient-based adversarial guidance steers the denoising trajectory toward a desired target label. The method is designed for both classification and image recognition settings, and its architecture combines classifier-free diffusion guidance, adversarial guidance during reverse diffusion, ScoreCAM-based reference-image injection, and noise optimization (Huang et al., 8 Jul 2025).

1. Conceptual setting and motivation

ScoreAdv was proposed against the background that conventional adversarial attacks typically rely on $\ell_p$-norm perturbation constraints, which the paper argues do not align with human perception (Huang et al., 8 Jul 2025). Within that critique, unrestricted adversarial examples are defined not as minimally perturbed pixels but as plausible images that fool a model while remaining natural under a generative prior. This places the method in the unrestricted adversarial example literature rather than in small-norm perturbation analysis.

The paper positions ScoreAdv against two earlier design tendencies. GAN-based unrestricted attack methods are described as suffering from instability in training, mode collapse, lower image quality, and weaker interpretability. Diffusion-based attacks, while more stable and capable of higher-quality synthesis, are described as still relying on PGD-style perturbation injection at each step and therefore not fully exploiting diffusion’s denoising process (Huang et al., 8 Jul 2025). ScoreAdv is introduced specifically to use iterative denoising itself as the mechanism that preserves realism, rather than treating diffusion as a mere perturbation engine.

The target object is a targeted unrestricted adversarial example generated from a pretrained diffusion model. The paper formalizes the adversarial sample set as

$$A_{UAE} \triangleq \{\boldsymbol{x}_{adv} \in \mathcal{G}(\boldsymbol{z}_{adv}, y, \boldsymbol{x}^{ref}) \mid f(\boldsymbol{x}_{adv}) = y_{tar}\}.$$

Here, $\mathcal{G}$ is the generative model, $f$ is the attacked model, $y$ is the ground-truth label, $y_{tar}$ is the target label, and $\boldsymbol{x}^{ref}$ is an optional reference image (Huang et al., 8 Jul 2025). This formulation makes the attack a constrained sampling problem on the image manifold rather than a direct perturbation problem in pixel space.

A central implication is that ScoreAdv treats adversarial generation as distribution shifting under a diffusion prior. The paper’s claim is not merely that diffusion can synthesize adversarial outputs, but that the denoising trajectory can be gradually biased toward an adversarial distribution while retaining natural image structure (Huang et al., 8 Jul 2025).

2. Diffusion backbone and targeted sampling formulation

The generative backbone uses classifier-free guidance to sample natural images conditioned on the true label $y$. The guided noise estimate is

$$\tilde{\boldsymbol{\epsilon}}_t = (1+s_c)\epsilon_\theta(\boldsymbol{x}_t, y) - s_c \epsilon_\theta(\boldsymbol{x}_t),$$

where $\ell_p$0 is the classifier guidance scale (Huang et al., 8 Jul 2025). This combines conditional and unconditional denoising predictions.

The corresponding reverse diffusion update is

$\ell_p$1

The paper interprets $\ell_p$2 as the natural denoised step, aligned with source semantics before adversarial steering is imposed (Huang et al., 8 Jul 2025). This is the baseline diffusion trajectory that ScoreAdv then perturbs in a targeted direction.

The method’s central claim is that targeted unrestricted attack generation should occur inside the reverse denoising chain rather than after it. In that sense, ScoreAdv differs from diffusion attacks that append PGD-like perturbations to a generative process; it instead adjusts the sampling dynamics themselves (Huang et al., 8 Jul 2025). This makes the adversarial objective endogenous to the reverse process.

The sampling procedure is also explicitly iterative at two levels. The reverse chain runs over timesteps $\ell_p$3, and the full attack can be repeated for $\ell_p$4 outer cycles with latent refinement (Huang et al., 8 Jul 2025). The reported default setting uses a pretrained Guided Diffusion model from OpenAI with $\ell_p$5, $\ell_p$6, $\ell_p$7, and $\ell_p$8 (Huang et al., 8 Jul 2025). Those hyperparameters govern the balance between natural synthesis and adversarial steering rather than defining a norm budget.

3. Adversarial guidance and score-based distribution shift

The defining mechanism of ScoreAdv is the adversarial guidance term inserted after the natural denoising step. The update is

$\ell_p$9

where $$A_{UAE} \triangleq \{\boldsymbol{x}_{adv} \in \mathcal{G}(\boldsymbol{z}_{adv}, y, \boldsymbol{x}^{ref}) \mid f(\boldsymbol{x}_{adv}) = y_{tar}\}.$$0 is the adversarial guidance scale and $$A_{UAE} \triangleq \{\boldsymbol{x}_{adv} \in \mathcal{G}(\boldsymbol{z}_{adv}, y, \boldsymbol{x}^{ref}) \mid f(\boldsymbol{x}_{adv}) = y_{tar}\}.$$1 is the attacked model’s target-class probability (Huang et al., 8 Jul 2025). Rather than applying a one-shot attack in image space, the method adds a targeted bias at every reverse step.

The paper describes this mechanism as gradually shifting the sampling distribution toward the adversarial distribution (Huang et al., 8 Jul 2025). The effect is cumulative: each step slightly increases the likelihood that the sample will be classified as $$A_{UAE} \triangleq \{\boldsymbol{x}_{adv} \in \mathcal{G}(\boldsymbol{z}_{adv}, y, \boldsymbol{x}^{ref}) \mid f(\boldsymbol{x}_{adv}) = y_{tar}\}.$$2, while the diffusion prior continues to regularize the image toward realism. This stepwise structure is also the basis for the paper’s claim that ScoreAdv remains effective under defenses, because the attack is not a single perturbation but a guided generative trajectory.

The supplementary derivation justifies the update by considering the target-conditional sampling distribution

$$A_{UAE} \triangleq \{\boldsymbol{x}_{adv} \in \mathcal{G}(\boldsymbol{z}_{adv}, y, \boldsymbol{x}^{ref}) \mid f(\boldsymbol{x}_{adv}) = y_{tar}\}.$$3

rewriting it with Bayes’ rule, and approximating $$A_{UAE} \triangleq \{\boldsymbol{x}_{adv} \in \mathcal{G}(\boldsymbol{z}_{adv}, y, \boldsymbol{x}^{ref}) \mid f(\boldsymbol{x}_{adv}) = y_{tar}\}.$$4 by Taylor expansion around the mean of the sampling distribution. The resulting shifted mean has the form

$$A_{UAE} \triangleq \{\boldsymbol{x}_{adv} \in \mathcal{G}(\boldsymbol{z}_{adv}, y, \boldsymbol{x}^{ref}) \mid f(\boldsymbol{x}_{adv}) = y_{tar}\}.$$5

providing the theoretical basis for gradient-based guidance inside reverse diffusion (Huang et al., 8 Jul 2025).

A recurrent theme in the paper is the balance between denoising and attack pressure. This balance is controlled by $$A_{UAE} \triangleq \{\boldsymbol{x}_{adv} \in \mathcal{G}(\boldsymbol{z}_{adv}, y, \boldsymbol{x}^{ref}) \mid f(\boldsymbol{x}_{adv}) = y_{tar}\}.$$6, $$A_{UAE} \triangleq \{\boldsymbol{x}_{adv} \in \mathcal{G}(\boldsymbol{z}_{adv}, y, \boldsymbol{x}^{ref}) \mid f(\boldsymbol{x}_{adv}) = y_{tar}\}.$$7, $$A_{UAE} \triangleq \{\boldsymbol{x}_{adv} \in \mathcal{G}(\boldsymbol{z}_{adv}, y, \boldsymbol{x}^{ref}) \mid f(\boldsymbol{x}_{adv}) = y_{tar}\}.$$8, $$A_{UAE} \triangleq \{\boldsymbol{x}_{adv} \in \mathcal{G}(\boldsymbol{z}_{adv}, y, \boldsymbol{x}^{ref}) \mid f(\boldsymbol{x}_{adv}) = y_{tar}\}.$$9, and $\mathcal{G}$0 (Huang et al., 8 Jul 2025). Larger $\mathcal{G}$1 or larger $\mathcal{G}$2 tends to improve attack success rate, but can harm visual quality; moderate $\mathcal{G}$3 is reported as sufficient, while excessive optimization can distort the image. This is not presented as a fixed optimum but as an operating trade-off inside the method’s design (Huang et al., 8 Jul 2025).

4. Reference-image conditioning, interpretable saliency, and noise optimization

A second major component is optional reference-image conditioning through ScoreCAM. The saliency map is defined as

$\mathcal{G}$4

where the purpose is to identify semantically important regions of the reference image (Huang et al., 8 Jul 2025). The paper gives the ScoreCAM construction as

$\mathcal{G}$5

and

$\mathcal{G}$6

This saliency mechanism is presented as interpretable because it identifies which reference regions are preserved or injected during generation (Huang et al., 8 Jul 2025).

The reference image is first diffused to the same timestep,

$\mathcal{G}$7

and then fused with the adversarially guided sample by inpainting-style composition:

$\mathcal{G}$8

The masking semantics are explicit in the paper: $\mathcal{G}$9 preserves adversarially sampled content, while $f$0 injects reference-image structure in salient regions (Huang et al., 8 Jul 2025).

The paper argues that ScoreCAM is preferable to gradient-based CAM variants such as GradCAM because it is more stable and avoids gradient issues, and it reports an ablation in which ScoreCAM outperforms GradCAM and GradCAM++ in both attack success and image quality (Huang et al., 8 Jul 2025). That claim is specific to the paper’s experimental setup, but it establishes saliency choice as a substantive design variable rather than a cosmetic addition.

After synthesis of $f$1, ScoreAdv further refines the initial latent via noise optimization:

$f$2

The algorithmic summary also describes the latent update as

$f$3

This refinement is performed for $f$4 cycles and is intended to pre-shape the latent noise so that the ensuing denoising trajectory is more favorable to the target-class objective (Huang et al., 8 Jul 2025).

5. Empirical evaluation, model coverage, and defensive robustness

The evaluation spans both generic image classification and face recognition. The paper reports experiments on ImageNet, using 1,000 validation images from 1,000 classes resized to $f$5, and on CelebA, with LFW used for recognition experiments; for recognition, 1,000 images are randomly selected (Huang et al., 8 Jul 2025). Ten target classification models are tested across three families: CNNs, Transformers, and MLPs. The listed CNNs are ResNet-50, VGG19, Inception-v3, WideResNet50-2, and ConvNeXt; the listed Transformers are ViT-B/16, Swin-B, and DeiT-B; the listed MLPs are Mixer-B/16 and Mixer-L/16. For recognition, the paper evaluates IR152, FaceNet, and MobileFace (Huang et al., 8 Jul 2025).

Evaluation aspect	Reported setting
Classification datasets	ImageNet
Recognition datasets	CelebA, LFW
Classification model families	CNNs, Transformers, MLPs
Recognition models	IR152, FaceNet, MobileFace
Attack regimes	White-box and black-box

The paper states that ScoreAdv achieves state-of-the-art attack success rates in most settings and improves transferability relative to prior baselines (Huang et al., 8 Jul 2025). Concrete white-box examples reported from ImageNet include 97.6% on ResNet-50, 100.0% on VGG19, and 93.4% on Inception-v3. It also reports strong cross-model transfer to Transformers and MLPs, and particularly strong performance in black-box recognition attacks (Huang et al., 8 Jul 2025).

Image quality is assessed with FID, LPIPS, PSNR, and SSIM. The paper reports FID $f$6, LPIPS $f$7, PSNR $f$8, and SSIM $f$9, and states that compared with the best prior diffusion baseline, AdvDiffuser, ScoreAdv improves all four reported image-quality indicators: lower FID, lower LPIPS, higher PSNR, and higher SSIM (Huang et al., 8 Jul 2025). In the paper’s interpretation, these values indicate more natural and coherent adversarial images.

Robustness is evaluated against a diverse set of defenses: AdvProp, R&P, RandS, Bit-Red, Adv-Inc-v3, IncRes-v2 ensemble, SR, NRP, and DiffPure (Huang et al., 8 Jul 2025). The paper reports that ScoreAdv remains strong under these defenses and is slightly more robust than previous methods, including AdvDiff. The stated explanation is the interaction between diffusion denoising and adversarial guidance, together with the method’s distribution-shifting behavior (Huang et al., 8 Jul 2025).

6. Methodological significance, constraints, and related directions

The principal significance claimed for ScoreAdv is that diffusion models can serve as a principled adversarial generation engine rather than as post hoc image priors (Huang et al., 8 Jul 2025). The method combines score-based targeted sampling, stepwise adversarial guidance, interpretable saliency-based reference injection, and noise optimization into a single training-free framework. A plausible implication is that ScoreAdv reframes unrestricted adversarial example generation as controlled generative inference, with realism preserved by the denoising prior instead of by explicit perturbation constraints.

The paper is also explicit about constraints. ScoreAdv relies on a pretrained diffusion model and is primarily developed for targeted attacks (Huang et al., 8 Jul 2025). The reference-image branch is optional, which means that content control can depend on whether a suitable reference image is available. Its practical behavior depends on balancing $y$0, $y$1, $y$2, and $y$3; too much adversarial guidance or too many optimization steps can degrade image quality. As with many gradient-based attacks, best performance in targeted settings depends on white-box access or a strong surrogate model (Huang et al., 8 Jul 2025).

A common misconception in unrestricted attack research is that removing an $y$4 budget eliminates all structure. ScoreAdv does not support that interpretation. Its unrestrictedness refers to the absence of an explicit norm constraint, but the attack remains strongly structured by the diffusion prior, the class-conditioning mechanism, and optionally the ScoreCAM mask (Huang et al., 8 Jul 2025). Another misconception is that diffusion is used only to beautify adversarial perturbations. The paper explicitly argues the opposite: denoising is the primary mechanism that keeps samples natural, while adversarial steering is injected as a controlled bias inside the reverse process (Huang et al., 8 Jul 2025).

In a broader score-guided generation context, related work includes “Score-Guided Generative Adversarial Networks” (Lee et al., 2020), which used an evaluator-derived score as an auxiliary signal for GAN training rather than for inference-time adversarial sampling. That earlier work is relevant because it addresses how score-based objectives can guide generation without collapsing into trivial exploitation of the scoring model (Lee et al., 2020). ScoreAdv differs in both objective and mechanism: it is training-free, diffusion-based, targeted, and directly concerned with unrestricted adversarial image generation (Huang et al., 8 Jul 2025).

Taken together, ScoreAdv occupies a specific position in adversarial machine learning: it is a diffusion-native, inference-time, targeted unrestricted attack framework whose novelty lies in integrating denoising, adversarial guidance, interpretable reference injection, and latent refinement into a single generative attack pipeline (Huang et al., 8 Jul 2025).