--- title: Adversarial Attacks in Machine Learning url: https://www.emergentmind.com/topics/adversarial-attacks type: topic --- # Adversarial Attacks in Machine Learning Adversarial attacks are perturbation-based methodologies designed to manipulate machine learning models, particularly deep neural networks (DNNs), such that they produce incorrect or attacker-specified predictions. These perturbations are typically constrained to ensure imperceptibility under an appropriate metric (e.g., $L_p$ norm for signals or semantic similarity for text). While originally studied in image classification, adversarial attacks are now established across vision, speech, reinforcement learning, NLP, and power systems. They have become central to security and reliability assessments of machine learning systems, revealing intrinsic vulnerabilities, transfer behavior, and limitations of current defenses. ## 1. Formal Definition and Taxonomy An adversarial example is an input $x' = x + \delta$ such that the perturbation $\delta$ is imperceptible ($\|\delta\|_p \leq \epsilon$), but the induced prediction $f(x')$ differs from the correct label or is pushed toward a designated target. Formally, for a classifier $f: \mathbb{R}^d\to \{1,\dots,K\}$ and input $x$ with label $y$, the canonical objectives are: - **Non-targeted attack**: $$\min_{\delta} \|\delta\|_p \;\mathrm{s.t.}\; f(x+\delta)\neq y$$ - **Targeted attack**: $$\min_{\delta} \|\delta\|_p \;\mathrm{s.t.}\; f(x+\delta) = t \neq y$$ Attacks can be classified along three primary axes [2112.02797, 2308.07673]: - **Knowledge**: white-box (full access to model parameters/gradients), black-box (query access only, either to probabilities—score-based—or just class labels—decision-based). - **Objective**: targeted (force $f(x+\delta)=t$), untargeted (force $f(x+\delta)\neq y$). - **Setting**: evasion (inference-time perturbations), poisoning (injecting malicious data during training). Common norm constraints include $L_0$ (sparse pixel changes), $L_2$ (Euclidean), and $L_\infty$ (maximum component change), with typical $\epsilon$ calibrated for dataset granularity (e.g., $8/255$ for images) [2308.07673]. ## 2. Attack Algorithms: Core Methodologies The literature has established several attack paradigms: - **FGSM (Fast Gradient Sign Method)**: A single-step, linearized attack under $L_\infty$ norm, $\delta = \epsilon\, \mathrm{sign}(\nabla_x L(x, y))$. - **PGD (Projected Gradient Descent)**: Iterative $L_\infty$-constrained attack, repeatedly stepping along the gradient and projecting back into the feasible ball [2112.02797, 2308.07673]. - **C&W Attack**: Unconstrained optimization of $L_2$ or $L_\infty$ distance subject to classification constraints, often via the logit margin [2112.02797]. - **DeepFool**: Iterative approach for minimal $L_2$-norm perturbations via local linearization [2202.07421]. - **Sparse and Imperceivable Attacks**: Black-box, score-based algorithms that optimize $L_0$ count, e.g., CornerSearch, and extensions with per-pixel adaptive bounds [1909.05040]. - **Entropy-Based Methods**: Allocate perturbation budget to high-entropy regions to evade human perception while maintaining attack success [1902.09286]. - **GAN-Based Attacks**: Use adversarial generative models ($G$) to synthesize imperceptible yet highly effective perturbations optimized against both a discriminator ($D$) and the target classifier ($C$) [2412.16662]. - **Algebraic Attacks on Explanations**: Exploit network symmetry groups to guarantee identical predictions but divergent explanations, avoiding traditional constrained optimization [2503.12683]. - **Domain-Specific and Multimodal Attacks**: Attacks on 3D rendering parameters, ISP/optics pipelines, reinforcement-learning state observations, and speech recognition (audio-visual correlation targeting) [1711.07183, 2102.03728, 1712.03632, 1912.08639]. In NLP, attacks are constructed by transforming token sequences through synonym swaps, paraphrasing, or character-level mutations, subject to semantic and syntactic constraints, as in TextAttack's modular pipeline [2005.05909]. ## 3. Empirical Findings and Transferability Extensive evaluation reveals state-of-the-art DNNs are highly brittle: - FGSM on MNIST ($\epsilon=0.3$) raises error from ~1% to >20%; PGD on CIFAR-10 ($\epsilon_\infty=8/255$) yields nearly 100% attack success; C&W on ImageNet achieves >99% attack success with imperceptible distortions [2112.02797, 2308.07673]. - Iterative or ensemble-based black-box attacks achieve high transfer rates: perturbations crafted for one model frequently mislead diverse architectures (e.g., FGSM/PGD adversarial inputs have 84–96% transfer success on black-box APIs) [2308.07673]. - Sparse attacks (e.g., CornerSearch with adaptive per-pixel bounds) can achieve high misclassification rates (e.g., median 2–7 pixels perturbed yields >95% non-targeted attack success rate) while avoiding detectability [1909.05040]. Physical attacks, such as perturbing 3D properties, camera pipelines, or embedding patterns in scenes or speech, demonstrate practical feasibility in safety-critical domains. Success rates remain high (e.g., >90% for targeted ISP/optics attacks) when the attack is tailored to actual acquisition conditions [2102.03728, 1711.07183]. Recent work on transform-dependent attacks introduces "metamorphic" perturbations that reveal vulnerabilities not just to input changes but to compositional transformation pipelines (e.g., scaling, blur, gamma). A single perturbation can control the adversarial outcome as a function of the applied transformation parameter, achieving up to 99% attack success depending on target and architecture [2406.08443]. ## 4. Defense Mechanisms and Robustness Defenses to adversarial attacks remain a topic of active research and challenge: - **Adversarial Training**: Incorporates adversarial examples (often via PGD) in the training loop, widely regarded as the most effective empirical defense, but only yields robustness near the sampled perturbation regime and often at substantial computational cost or reduction in clean accuracy [1705.09764, 2308.07673]. - MAT (Multi-strength Adversarial Training) extends training to multiple perturbation strengths, improving coverage [1705.09764]. - **Input Transformations and Denoising**: JPEG compression, random resizing/padding, and learned denoisers (HGD) can partially disrupt attacks but are typically circumventable by adaptive strategies [1804.00097]. - **Certified Defenses**: Interval Bound Propagation and randomized smoothing provide provable but scaling-limited robustness certificates [2112.02797]. - **Purification Mechanisms**: In NLP, ensemble-based masked language model purification mitigates word substitution attacks without access to the attacker's candidate set, raising after-attack accuracy by >60 percent points against strong word-substitution attacks [2203.14207]. - **Hedge Defense**: Applies a second, general $L_p$-bounded perturbation to adversarially trained models, exploiting differences in Lipschitz continuity across class scores to reverse many adversarial errors and boost accuracy under attack by up to 7% on CIFAR-10/ImageNet [2106.04938]. A perennial theme is the trade-off between robustness and generalization, the curse of high-dimensional geometry (the existence of thin adversarial subspaces), and the intrinsic limitation of gradient-masking or obfuscated approaches. ## 5. Extensions and Emerging Directions Recent advancements expand the adversarial landscape: - **Multimodal and Cross-Domain Attacks**: Adversarial examples targeting correlations between modalities (e.g., audio-visual sync in speech recognition) or features crossing domains (e.g., geometric, photometric, and ISP transformations) [2102.03728, 1912.08639]. - **Universal and Signal-Agnostic Attacks**: Construction of single perturbations that generalize over a large portion of the data manifold (e.g., universal perturbations with 74% fooling rates in power systems) [2202.07421]. - **Explainability and Interpretability Attacks**: Craft attacks to diverge explanation while preserving model prediction using symmetry and group-theoretic insights [2503.12683]. - **Adaptive, Perception-Evading Attacks**: Use entropy or structural priors to minimize human detection while maximizing machine misclassification, validated by user studies [1902.09286]. - **GAN-Driven Adversarial Example Synthesis**: Exploit adversarial generative models to discover realistic, imperceptible patterns evading both classifier and discriminator, consistently outperforming FGSM/BIM methods [2412.16662]. - **Textual Adversarial Attacks**: Modularized approaches (TextAttack) enable transformation-constraint-search pipelines, supporting grammar/semantic-aware attacks at scale [2005.05909]. ## 6. Representative Benchmarks and Quantitative Results Empirical results highlight the consistently high effectiveness and transferability of adversarial attacks against SOTA models: | Dataset | Model/Method | Attack | Attack Success/Accuracy | Reference | |----------------|------------------------|----------------|------------------------|--------------------| | MNIST | DNN, PGD-trained | FGSM ($\epsilon=0.3$) | Error >20% (from 1%) | [2112.02797] | | CIFAR-10 | ResNet, Adv. Training | PGD ($\epsilon=8/255$) | Acctest $62.7\%$ (Adv-pNML $67.2\%$) | [2109.01945] | | ImageNet | ResNet-50, Fast AT | PGD ($\epsilon=8/255$) | Accbase $16.0\%$, Adv-pNML $20.0\%$ | [2109.01945] | | Power Quality | ConvNet | SAA (universal) | Misclassification up to $74\%$ | [2202.07421] | | Speech (LRW) | AV-SR DNN | FGSM ($\epsilon^A{1024},\epsilon^V{16}$) | Top1acc $10.4\%$, detection AUC $0.99$ | [1912.08639] | | Text (IMDB) | BERT | TextFooler | No defense: $2.8\%$ acc; Purification: $51.0\%$ | [2203.14207] | Defended models, even those using adversarial training or multi-strength methods, remain vulnerable to advanced adaptive, physical, or domain-specific attacks. ## 7. Open Problems and Future Challenges Despite extensive defenses, no single approach is universally effective, and research illustrates defenders must anticipate attacks that exploit transformation, perception, and group-theoretic vulnerabilities [2406.08443, 2503.12683]. Robustness in deployed systems will require closing gaps in certified guarantees, generalizing to unseen transformations and modalities, and integrating multi-layered defense frameworks. Transform-dependent and algebraic attacks underscore that robust models must secure not just fixed $(x, \delta)$ neighborhoods, but also compositional transformations and symmetry-induced invariants, requiring fundamentally new techniques in model design, training, and certification. --- **Key References:** 2109.01945, 2102.03728, 1705.09764, 2112.02797, 1804.00097, 2406.08443, 1712.03632, 2202.07421, 1812.02885, 2412.16662, 1912.08639, 2106.04938, 1902.09286, 1711.07183, 2001.11137, 1909.05040, 2005.05909, 2308.07673, 2503.12683, 2203.14207 Source: https://www.emergentmind.com/topics/adversarial-attacks