ResPA: Residual Perturbation Attack

• ResPA is a transfer-based adversarial attack that uses the residual between current and EMA-based historical gradients to guide flatness probing.
• It mitigates overfitting to sharp surrogate-specific loss landscapes by steering perturbations toward flatter regions during iterative optimization.
• Empirical evaluations demonstrate that ResPA consistently improves attack success rates and complements existing input transformation methods.

Searching arXiv for ResPA and adjacent residual-perturbation papers to ground the article in current literature. Residual Perturbation Attack (ResPA) is a transfer-based adversarial attack for black-box settings that improves adversarial transferability by changing how local flatness is probed during iterative optimization. Instead of using only the current input gradient to define the perturbation direction, ResPA builds a reference gradient from historical gradients via an exponential moving average (EMA), then uses the residual between the current gradient and that reference as the perturbation direction. The method is intended to guide adversarial examples toward flatter regions of the surrogate loss landscape, reduce overfitting to sharp surrogate-specific directions, and improve attack success on unseen target models (Peng et al., 6 Aug 2025).

1. Problem setting and motivation

ResPA is formulated in the standard transfer-attack setting: an attacker crafts adversarial examples on a surrogate model $f^s$ and evaluates them on a black-box target model $f^t$. The optimization problem is stated as

$$\max _{x^{a d v} } J\left(x^{a d v}, y\right) \quad \text { s.t. } \left\|x - x^{a d v}\right\|_{\infty}<\epsilon,$$

where $x$ is the clean input, $y$ is the true label, $x^{adv}$ is the adversarial example, $J(x,y)$ is the cross-entropy loss on the surrogate, and $\epsilon$ is the perturbation budget (Peng et al., 6 Aug 2025). Transferability is measured by the attack success rate

$$A S R=\frac{1}{|\mathcal{X}|} \sum_{x \in \mathcal{X}} \mathbb{I}[f^t(x) \neq f^t(x^{adv})].$$

The motivating claim is that iterative attacks often overfit the surrogate model’s sharp local loss geometry. High white-box success therefore does not necessarily imply high black-box transferability. A recent line of work seeks flatter local maxima, but ResPA argues that prior flatness-based attacks still rely too heavily on the current gradient when selecting the direction in which local flatness is evaluated. In the paper’s account, this can bias the search toward the sharpest local region within the perturbation radius $\rho$, limiting transferability even when flatness regularization is present (Peng et al., 6 Aug 2025).

The surrogate loss is written as

$f^t$0

and the classical flatness term considered by prior methods is

$f^t$1

Because the exact minimizer is difficult to compute, prior methods approximate it by moving along the current gradient: $f^t$2 ResPA is introduced specifically as a replacement for this current-gradient-only direction choice (Peng et al., 6 Aug 2025).

2. Residual perturbation mechanism

The defining feature of ResPA is its reference-gradient and residual-gradient construction. The reference gradient is formed from historical gradients via EMA: $f^t$3 with

$f^t$4

where $f^t$5 is the EMA decay factor and $f^t$6 is the average gradient over sampled points in the current iteration (Peng et al., 6 Aug 2025).

The residual gradient is then defined as

$f^t$7

This residual is the perturbation direction used to probe flatness. The flatness term becomes

$f^t$8

so the perturbed neighbor is chosen along the residual direction rather than the raw current gradient (Peng et al., 6 Aug 2025).

The regularized objective over sampled points is

$f^t$9

where

$$\max _{x^{a d v} } J\left(x^{a d v}, y\right) \quad \text { s.t. } \left\|x - x^{a d v}\right\|_{\infty}<\epsilon,$$0

and $$\max _{x^{a d v} } J\left(x^{a d v}, y\right) \quad \text { s.t. } \left\|x - x^{a d v}\right\|_{\infty}<\epsilon,$$1 is the penalty coefficient. The approximate gradient used in optimization is

$$\max _{x^{a d v} } J\left(x^{a d v}, y\right) \quad \text { s.t. } \left\|x - x^{a d v}\right\|_{\infty}<\epsilon,$$2

In the paper’s interpretation, the residual direction captures the difference between the current local direction and the historically accumulated direction. This is intended to reflect global perturbation-direction change rather than only instantaneous local sharpness. A plausible implication is that ResPA functions less as a pure momentum variant than as a geometry-sensitive modification of flatness estimation: the novelty lies in how the neighbor $$\max _{x^{a d v} } J\left(x^{a d v}, y\right) \quad \text { s.t. } \left\|x - x^{a d v}\right\|_{\infty}<\epsilon,$$3 is selected, not in the final sign-step rule (Peng et al., 6 Aug 2025).

3. Optimization procedure and algorithmic workflow

ResPA samples neighborhood points around the current adversarial example: $$\max _{x^{a d v} } J\left(x^{a d v}, y\right) \quad \text { s.t. } \left\|x - x^{a d v}\right\|_{\infty}<\epsilon,$$4 where

$$\max _{x^{a d v} } J\left(x^{a d v}, y\right) \quad \text { s.t. } \left\|x - x^{a d v}\right\|_{\infty}<\epsilon,$$5

The average gradient over the sampled points is

$$\max _{x^{a d v} } J\left(x^{a d v}, y\right) \quad \text { s.t. } \left\|x - x^{a d v}\right\|_{\infty}<\epsilon,$$6

After updating the EMA state $$\max _{x^{a d v} } J\left(x^{a d v}, y\right) \quad \text { s.t. } \left\|x - x^{a d v}\right\|_{\infty}<\epsilon,$$7, ResPA uses an MI-style momentum accumulator: $$\max _{x^{a d v} } J\left(x^{a d v}, y\right) \quad \text { s.t. } \left\|x - x^{a d v}\right\|_{\infty}<\epsilon,$$8 followed by the adversarial update

$$\max _{x^{a d v} } J\left(x^{a d v}, y\right) \quad \text { s.t. } \left\|x - x^{a d v}\right\|_{\infty}<\epsilon,$$9

Thus ResPA is not a separate outer optimization framework from iterative momentum attacks; it is a flatness-regularized iterative attack whose distinctive component is the residual perturbation direction used inside the flatness term (Peng et al., 6 Aug 2025).

The default attack settings reported in the paper are $x$0, $x$1, $x$2, and $x$3. For sampling-based methods, the paper uses $x$4 and $x$5. For ResPA specifically, the stated settings are $x$6, $x$7, $x$8, and $x$9. The paper also notes a discrepancy in the hyperparameter section, where “default” values are stated as $y$0 for simplified analysis, while the main experiments use $y$1 (Peng et al., 6 Aug 2025).

The method is described as compatible with input transformation methods. In the reported experiments, it is combined with DIM, TIM, SIM, Admix, and SSA. This suggests that ResPA is intended as a plug-in attack component for transferability enhancement rather than as a stand-alone replacement for all iterative attack design choices (Peng et al., 6 Aug 2025).

4. Empirical performance

The evaluation uses 1,000 images from the ILSVRC 2012 validation set and the Market-1501 person re-identification dataset. The standard source and target models are Inception-v3, ResNet-50, DenseNet-121, VGG-19, Vision Transformer, and Swin Transformer. The paper also evaluates adversarially trained models Inc-v3$y$2 and Inc-v3$y$3, as well as defense models HGD, Bit-Red, FD, JPEG, NRP, random resizing and padding, and randomized smoothing (Peng et al., 6 Aug 2025).

For single-model transfer attacks, ResPA consistently attains the best or near-best average ASR. With Inc-v3 as the surrogate, the reported average ASRs are MI 45.0, VMI 58.2, GRA 59.6, PGN 63.6, AdaMSI 46.6, TPA 60.3, and ResPA 64.6. With Res-50 as the surrogate, the reported averages are MI 64.9, VMI 78.7, GRA 82.9, PGN 84.2, AdaMSI 64.0, TPA 81.7, and ResPA 84.6. With Den-121 as the surrogate, the reported averages are MI 68.5, VMI 81.9, GRA 86.0, PGN 87.2, AdaMSI 71.0, TPA 86.3, and ResPA 87.5 (Peng et al., 6 Aug 2025).

The combination results are a central empirical claim. On Res-50, average ASR improves from 79.6 to 91.0 for DIM, from 70.6 to 90.9 for TIM, from 77.3 to 89.0 for SIM, from 70.2 to 81.8 for Admix, and from 82.4 to 86.6 for SSA when ResPA is added (Peng et al., 6 Aug 2025). This supports the paper’s claim that ResPA is complementary to current input transformation methods.

Under an ensemble surrogate of Res-50 + Vgg-19 + Den-121, the average ASRs are MI 84.2, VMI 91.8, GRA 95.4, PGN 95.8, AdaMSI 83.4, TPA 93.8, and ResPA 96.0. Against defended models with adversarial examples generated on Inc-v3, the reported average ASRs are MI 36.8, VMI 52.0, GRA 54.8, PGN 58.8, AdaMSI 35.4, TPA 56.5, and ResPA 59.8 (Peng et al., 6 Aug 2025).

On Market-1501 person re-identification, lower Rank-1 and mAP indicate stronger attack performance. Averaged over four black-box Re-ID models, the reported results are MI 45.32 / 32.66, VMI 32.08 / 23.09, PGN 30.24 / 21.57, GRA 28.88 / 20.81, BSR 31.14 / 22.12, TPA 27.03 / 19.20, and ResPA 26.45 / 18.77, making ResPA the strongest method in that evaluation (Peng et al., 6 Aug 2025).

5. Relation to adjacent residual-centered methods

ResPA is specifically the method in “Boosting Adversarial Transferability via Residual Perturbation Attack” (Peng et al., 6 Aug 2025). It should not be conflated with several adjacent lines of work that also use the language of residuals, perturbation, or reversibility.

“Robust Adversarial Perturbation on Deep Proposal-based Models” proposes R-AP, explicitly named Robust Adversarial Perturbation, and attacks the Region Proposal Network in proposal-based detectors and instance segmentation models. Its objective combines a label loss and a shape loss, and its transfer mechanism is based on attacking the shared proposal generator rather than the final prediction head (Li et al., 2018). This is an intermediate-stage proposal attack, not a residual-gradient flatness attack.

“BadRes: Reveal the Backdoors through Residual Connection” exploits residual connections as a backdoor attack surface by replacing one residual merge with a subtraction-based BadRes block during training. Its mechanism is a trigger-based training-time backdoor, not a transfer-based test-time adversarial attack (He et al., 2022). “Deep Learning with Data Privacy via Residual Perturbation” uses Gaussian noise injection into each residual mapping of ResNets as a privacy-preserving defense with differential privacy claims, rather than an attack mechanism (Tao et al., 2024).

A different group of papers uses “residual” in privacy or optimization senses. “Reminiscence Attack on Residuals” studies approximate machine unlearning and proposes ReA, which amplifies residual traces of unlearned data through targeted fine-tuning processes (Xiao et al., 28 Jul 2025). “The Unseen Threat: Residual Knowledge in Machine Unlearning under Perturbed Samples” formalizes residual knowledge revealed by perturbations of forget samples and proposes RURK as a mitigation (Hsu et al., 29 Jan 2026). “Residual-Evasive Attacks on ADMM in Distributed Optimization” studies attacks that preserve the monitored ADMM residual while perturbing iterates, which is mechanically close to a residual-preserving stealth attack, but belongs to distributed optimization rather than adversarial image transfer (Bruckmeier et al., 22 Apr 2025).

This suggests that the term “residual perturbation” has become polysemous. In ResPA, “residual” refers to the residual between the current gradient and the EMA-based reference gradient. In other works, “residual” may instead refer to residual network mappings, residual traces left by approximate unlearning, monitored optimization residuals, or recoverable side information in reversible perturbation systems (Peng et al., 6 Aug 2025).

6. Limitations, caveats, and significance

The paper’s strengths are simplicity of the core idea, strong empirical performance across standard transfer, ensemble transfer, defended models, CNN and transformer targets, and person re-identification, and compatibility with DIM, TIM, SIM, Admix, and SSA (Peng et al., 6 Aug 2025). Its practical message is that transferability can be improved not only by input diversification or momentum smoothing, but also by changing the direction used to evaluate local flatness.

The paper also exposes several limitations. ResPA is more expensive than plain I-FGSM or MI-FGSM because it requires neighborhood sampling and averaging. It introduces additional hyperparameters, including $y$4, $y$5, $y$6, $y$7, and the flatness radius $y$8. Incremental gains over the strongest flatness-based baselines can be modest in some settings; for example, the improvement over PGN is sometimes about 1 point on average. The mechanism is justified primarily through geometric intuition, ablation trends, and loss-surface visualization rather than a formal theorem. The manuscript also contains typographical corruption in several formulas and the noted mismatch in default $y$9 values (Peng et al., 6 Aug 2025).

The ablations reported in the paper indicate that transferability improves initially as $x^{adv}$0 increases and declines when $x^{adv}$1; that increasing $x^{adv}$2 improves transferability with diminishing returns; that transferability is almost stable when $x^{adv}$3; and that good performance is obtained for $x^{adv}$4, while performance drops sharply when $x^{adv}$5 (Peng et al., 6 Aug 2025). These observations suggest that the method is not hyper-fragile once the history term is sufficiently strong and the flatness penalty is balanced rather than dominant.

In the current literature, ResPA is best understood as a flatness-regularized, momentum-based transfer attack whose main novelty is the residual perturbation direction

$x^{adv}$6

By making the flatness probe depend on the discrepancy between the current local gradient and a history-based reference, it seeks flatter and less surrogate-specific adversarial regions than current-gradient-only methods, and its empirical evaluations indicate that this direction choice improves black-box transferability across a broad range of targets (Peng et al., 6 Aug 2025).