Fast Gradient Sign Method (FGSM)

• FGSM is a single-step adversarial attack that uses the gradient's sign within an L∞ constraint to perturb inputs and mislead neural networks.
• Its computational efficiency and linear approximation make it a foundational tool for scalable adversarial training and defense strategies.
• Despite its simplicity, FGSM faces challenges like catastrophic overfitting and gradient masking, spurring various enhancements and variants.

The Fast Gradient Sign Method (FGSM) is a foundational single-step adversarial attack and training method for neural networks, derived as a linear approximation of an inner maximization problem in robust optimization. FGSM constructs adversarial examples by perturbing inputs in the direction that maximally increases the model’s loss, providing a simple, computationally efficient tool for both attacking and robustifying deep learning models. It serves as the methodological and theoretical basis for numerous advanced adversarial learning and defense pipelines, and its properties—efficiency, linearity, sensitivity to curvature, and failure modes such as catastrophic overfitting—have influenced both attack research and the development of scalable adversarial training recipes.

1. Formal Definition and Principle

FGSM was introduced by Goodfellow et al. as a white-box attack that exploits local loss gradients to efficiently generate input perturbations that maximize classification error under an $L_\infty$ constraint. Given an input $x \in \mathbb{R}^d$, true label $y$, model parameters $\theta$, and loss function $J(\theta, x, y)$ (often cross-entropy):

$$x_{\mathrm{adv}} = x + \varepsilon \cdot \mathrm{sign}(\nabla_x J(\theta, x, y))$$

Here, $\varepsilon$ is the maximum allowed $L_\infty$ norm of the perturbation, and $\mathrm{sign}(\cdot)$ is the elementwise sign function. FGSM optimizes the first-order Taylor expansion of the loss:

$$J(\theta, x+\eta, y) \approx J(\theta, x, y) + \eta^\top \nabla_x J(\theta, x, y)$$

and selects the perturbation $\eta$ that maximizes the loss within the $L_\infty$ ball of radius $\varepsilon$ (Wong et al., 2020, Waghela et al., 20 Aug 2024, Musa et al., 2022, Dou et al., 2018).

2. Variants, Extensions, and Algorithmic Enhancements

Several FGSM variants and extensions address its limitations or adapt it for different architectures and threat models:

• Targeted FGSM: Instead of maximizing loss for the true class, minimizes loss toward a specific target class, using a negative gradient sign (Musa et al., 2022, Dou et al., 2018).
• Iterative FGSM (I-FGSM): Applies FGSM steps iteratively with smaller step sizes and projection back to the $\ell_\infty$-ball; leads to stronger attacks (Milton, 2018).
• Momentum (MI-FGSM) and Diverse Input (DI$^2$-FGSM): Combine with momentum accumulation and random input transformations to enhance black-box transferability; the M-DI$^2$-FGSM attack is an example of this synergy (Milton, 2018).
• Embedding-Space FGSM: Applies normalized gradient perturbations in the embedding space for neural retrieval architectures under $L_2$ norm constraints (Lupart et al., 2023).
• Saliency- or Aesthetics-Aware FGSM: Modifies FGSM by weighting perturbations using saliency or image quality gradients to preserve perceptual quality (Linardos et al., 2019).

Pseudocode for canonical FGSM adaptation and adversarial example generation is widely standardized (see (Waghela et al., 20 Aug 2024, Wong et al., 2020)). The method typically consists of a forward pass to compute the loss, then one backward pass to obtain the gradient with respect to input, and finally a sign-and-scale operation followed by optional clipping to maintain input validity.

3. Theoretical Properties and Implications

FGSM’s effectiveness stems from the local linearity hypothesis of deep networks in high-dimensional input spaces (Dou et al., 2018, Waghela et al., 20 Aug 2024). In sufficiently small $\varepsilon$ regimes, FGSM increases the loss of any ReLU-CNN monotonically for untargeted attacks and decreases the loss toward a target class for targeted attacks as long as local linearity holds (Dou et al., 2018). This local linearity can break for larger perturbation radii, leading to deviations from optimal attack directions and diminished effectiveness.

FGSM-based adversarial training can be interpreted as a data-driven penalized likelihood, inducing an $\ell_1$-like penalty analogous to LASSO regularization. In the limit, this leads to provable statistical properties (root-$n$ consistency, weak oracle property) under certain GLM settings (Zuo, 2018). However, domain-specific bias can emerge in non-sign-neutral data regimes.

Curvature along the FGSM direction is a critical factor penalizing the method’s accuracy relative to stronger, multi-step attacks like PGD: large curvature causes FGSM to underexplore the threat space (Huang et al., 2020). Regularization approaches such as curvature penalties or alignment constraints between gradients have been proposed to bridge this gap (Huang et al., 2020, Andriushchenko et al., 2020).

4. Adversarial Training and Failure Modes

FGSM is central to scalable adversarial training (“FGSM-AT”). This training replaces clean samples by FGSM-perturbed versions in the outer optimization loop, minimizing the expected worst-case loss:

$$\min_\theta \mathbb{E}_{(x,y) \sim D} \left[ \max_{||\delta||_\infty \leq \epsilon} J(\theta, x+\delta, y) \right]$$

Key empirical findings:

• Computational Efficiency: FGSM-based training achieves robust models orders of magnitude faster than PGD-based adversarial training, making it feasible at ImageNet scale (Wong et al., 2020, Jia et al., 2022).
• Catastrophic Overfitting (CO): FGSM-AT, especially with zero or random init, is vulnerable to sudden collapses in robust accuracy—PGD-robustness can go to zero within a single epoch while FGSM-robustness remains high. This overfitting is tied to local nonlinearity and loss of gradient alignment (Andriushchenko et al., 2020, Wong et al., 2020, Xie et al., 2021).
• Mitigations: Several regularizers and strategies mitigate CO:
  • Random initialization within the threat ball (“FGSM-RS”) (Wong et al., 2020)
  • Gradient alignment penalties (GradAlign) (Andriushchenko et al., 2020)
  • PGD-based logit pairing regularizers (FGSMPR) (Xie et al., 2021), and curvature penalties (FGSMR) (Huang et al., 2020)
  • Prior-guided FGSM initialization (PGI) using historical perturbations (FGSM-MEP/EP/BP) (Jia et al., 2022)
  • Data and architectural modifications, e.g., masked input pixels, smooth activations, stride adjustments, and weight regularization (Li et al., 2022)
• FGSM in Transfer Learning: In adversarially robust transfer learning, FGSM-AT is intrinsically more stable and can match PGD-level robustness without CO at standard threat budgets when fine-tuning pre-trained models, further accelerated by parameter-efficient fine-tuning (Zhao et al., 27 Jun 2025).

5. Empirical Performance and Benchmarking

FGSM’s attack strength, training efficiency, and generalization impact are empirically well-characterized:

• Under increasing $\varepsilon$, FGSM can rapidly drive model accuracy to chance, especially on MNIST- or CIFAR-type tasks (Waghela et al., 20 Aug 2024, Dou et al., 2018, Musa et al., 2022).
• Experiments demonstrate that standard (vanilla) FGSM-AT is insufficient for high-$\varepsilon$ robustness; refined strategies (e.g., FGSM-MEP, GradAlign, PGI, or hybrid data/architecture regularizers) attain PGD-level robustness at 1/3–1/4 the computational cost (Li et al., 2022, Jia et al., 2022, Andriushchenko et al., 2020, Xie et al., 2021).
• On ResNet/CIFAR-10, well-tuned FGSM-AT with batch-prior or momentum prior achieves $>$49% PGD-50 accuracy and $>$45% AutoAttack accuracy under $\epsilon=8/255$, closely tracking PGD-AT (Jia et al., 2022, Li et al., 2022).
• In robust transfer, FGSM fine-tuning closes the gap to PGD while reducing wall-time by 4×, with no catastrophic overfitting for $\varepsilon\leq8$ (Zhao et al., 27 Jun 2025).
• On neural retrieval and ranking models, FGSM adversarial training applied to embedding spaces yields consistent robustness and generalization improvements, including resilience to typos and out-of-domain noise (Lupart et al., 2023).

6. Defensive Strategies and Unlearning Methods

Traditional defenses against FGSM include adversarial training, input preprocessing, and regularization-based methods. Recent work investigates machine unlearning as a defense mechanism: adversarial points with highest loss (i.e., those with greatest contribution to model vulnerability) are iteratively “unlearned” (removed) and the model retrained, restoring robust accuracy without resorting to data augmentation (Khorasani et al., 3 Nov 2025).

Other notable defenses are:

• Input preprocessing (e.g., JPEG compression, feature squeezing) (Musa et al., 2022)
• Architectural modifications (stride increase, smooth activations, masking) (Li et al., 2022)
• Regularizers for gradient norm or logit agreement between FGSM and multi-step attacks (Xie et al., 2021, Huang et al., 2020)
• Curvature minimization in loss landscape (Huang et al., 2020)
• Historical-perturbation priors in fast adversarial training (Jia et al., 2022)

7. Limitations, Open Challenges, and Broader Impact

FGSM’s simplicity and speed come with trade-offs:

• Linearity Breakdown: In high-curvature regions or with larger $\varepsilon$, FGSM poorly approximates the best attack direction, leading to a robustness gap relative to PGD-AT (Huang et al., 2020).
• Gradient Masking Risk: FGSM-AT can promote gradient masking, making models appear robust to single-step attacks while remaining vulnerable to multi-step or adaptive adversaries (Andriushchenko et al., 2020, Wong et al., 2020).
• Transferability: FGSM’s effectiveness in the black-box regime is limited compared to momentum/diverse/iterative extensions, or when facing real-world, distribution-shifted attacks (Milton, 2018, Linardos et al., 2019).
• Task and Architecture Dependence: The success of FGSM-AT and required tricks are architecture- and data-dependent; recommendations for $\varepsilon$, initialization, and regularization must be tailored accordingly.

Despite these caveats, FGSM remains a cornerstone tool for both adversarial attack design and scalable adversarial training, shaping both empirical best practices and the theoretical understanding of robustness in deep networks (Wong et al., 2020, Waghela et al., 20 Aug 2024, Zhao et al., 27 Jun 2025). Newer work continues to refine its stability and effectiveness, expanding its reach and applicability in security-critical domains and large-scale robust transfer learning.