---
title: Adversarial ML Taxonomy
url: https://www.emergentmind.com/topics/adversarial-machine-learning-aml-taxonomy
type: topic
---

# Adversarial ML Taxonomy

Adversarial Machine Learning (AML) investigates vulnerabilities in machine learning systems to input or training data manipulation, formalizing attacks and defenses in a unified, hierarchical framework. AML encompasses settings where the predictions or learned models can be intentionally perturbed by adversaries to achieve malicious objectives, under various knowledge, capability, and domain constraints. A rigorous taxonomy identifies the compositional structure of attacks—classified by stages of the ML life cycle, threat models, perturbation bounds, and attacker goals—and connects these to a spectrum of defensive methodologies, spanning empirical tactics to provable robustness and formal verification. The field is characterized by a persistent arms race between increasingly adaptive attacks and more sophisticated defenses, with key open challenges centered on certified robustness, scalability, and adapting to nonstationary real-world environments [2502.05637, 2302.09457].

## 1. Formal Threat Models and Unifying Notation

At the core of AML taxonomy lies the formal definition of threat models delineated by adversary knowledge $K$, capabilities $C$, and goals $G$:

- **Knowledge $K$**:  
  - *White-box*: adversary has complete access to model parameters $\theta$, architecture, gradients $\nabla_x \mathcal{L}(f_\theta(x), y)$, and training data.  
  - *Black-box*: adversary may only query $f_\theta(x)$ or obtain loss outputs, lacking direct access to model internals [2502.05637, 1810.07339, 2302.09457].

- **Capabilities $C$**: 
  - *Evasion*: perturbation of inputs at inference time such that $\| \delta \|_p \le \epsilon$.
  - *Poisoning*: injection or modification of up to $N_\mathrm{poison}$ training samples, possibly under $\ell_p$ constraints (feature or problem space).
  - *Model Extraction/Inference*: query-based reconstruction or privacy inference (membership, data extraction) [2502.05637, 1911.02621, 2007.02407].

- **Goals $G$**:  
  - *Untargeted*: induce any misclassification $f_\theta(x') \neq y$.
  - *Targeted*: force classification to a specific class $t$.
  - *Backdoor/Trigger*: produce targeted misclassification only when an embedded trigger is present.
  - *Confidentiality/Privacy*: extract internal details or verify membership of data [2502.05637, 2302.09457, 2201.06093].

The attack space is further stratified by labeling attacks as feature-space (direct manipulation of input vectors) or problem-space (domain-constrained modifications to raw data objects) [1911.02621].

## 2. Taxonomy of AML Attacks Across the ML Lifecycle

A comprehensive AML taxonomy emerges from analyzing attacks by their stage of occurrence and mechanism [2502.05637, 2302.09457, 1810.07339]:

| ML Lifecycle Stage    | Attack Paradigm      | Mechanism/Goal                |
|----------------------|---------------------|-------------------------------|
| Pre-training         | Data-poisoning backdoors | Stateless poisoning with additive/static triggers; e.g. BadNets, blended triggers. Target: backdoor mapping trigger $\rightarrow$ $y_e$. |
| In-training          | Training-controllable backdoors | Dynamic optimization of jointly learned triggers and weights; input-aware, semantic, or federated triggers. |
| Post-training        | Weight/Parameter attacks | White/gray-box optimization or physical bit-flip to induce malicious behavior in deployed models. |
| Deployment           | Device-level parameter tampering | Rowhammer, laser fault; bit-flip attack on stored weights. |
| Inference            | Evasion (adversarial examples) | White-box: direct gradient methods; black-box: transfer, finite difference, decision/score-based [2502.05637, 2302.09457]. Also: backdoor or weight-attack activation, model extraction, membership inference. |

Prominent attack methods include:
- **FGSM**: $\delta = \epsilon \cdot \operatorname{sign}(\nabla_x \mathcal{L}(f_\theta(x), y))$ (white-box, $\ell_\infty$).
- **PGD**: $x_{k+1} = \mathrm{Proj}_{\| \cdot \|_p \le \epsilon}(x_k + \alpha \operatorname{sign}(\nabla_x \mathcal{L}(f_\theta(x_k), y)))$ (multi-step, white-box).
- **Carlini–Wagner**: minimize $ \| \eta \|_2^2 + c \cdot \phi(x+\eta)$.
- **Finite-difference/Black-box**: $\hat{g}_i = [\mathcal{L}(f_\theta(x+\delta e_i)) - \mathcal{L}(f_\theta(x))]/\delta$.

Evasion, poisoning, transfer (surrogate) attacks, and privacy/model extraction strategies all map to distinct positions in the hierarchical taxonomy.

## 3. Defense Methodologies: Empirical to Certified Robustness

Defense taxonomy is structured by the stage of intervention and level of robustness guarantee [2502.05637, 2007.00753, 2302.09457].

- **Input Preprocessing & Gradient Masking**:  
  - *Defensive Quantization*: quantize gradients to $b$ bits, $\nabla_x^\mathrm{quant} = \operatorname{round}(\nabla_x 2^{b-1}) / 2^{b-1}$, aiming to obfuscate gradients.
  - *Randomized Smoothing*: addition of Gaussian noise to the input, forming a smoothed classifier $\tilde{f}$ with certified $\ell_2$-robustness radius $r = (\sigma/2)(\Phi^{-1}(p_A) - \Phi^{-1}(p_B))$ [2502.05637, 2007.00753].

- **Adversarial Training**:  
  - Empirical minimization of worst-case loss within the perturbation budget:
    $$
    \theta^* = \arg\min_\theta \mathbb{E}_{(x, y) \sim \mathcal{D}} \left[ \max_{\| \delta \|_p \le \epsilon} \mathcal{L}(f_\theta(x+\delta), y) \right]
    $$
  - Variations include min–max PGD training, ensemble adversarial training, logit pairing [2502.05637, 2007.00753].

- **Certified Robustness Techniques**:
  - *Interval Bound Propagation (IBP)*: layer-wise propagation of activation bounds, guaranteeing no decision change within $\epsilon$.
  - *Randomized Smoothing*: as above, with provable certificates in the $\ell_2$-norm.
  - *Exact/LP/SDP relaxations*: Reluplex (SMT), convex polytope theory, abstract interpretation for smaller models or relaxed bounds [2502.05637, 2007.00753].

- **Other Tactics**:
  - *Data Sanitization*: detection/removal of anomalous samples via influence functions or stat tests.
  - *Differential Privacy*: restriction of single-sample influence on model outputs, mitigating poisoning.
  - *Detection/Reactive Measures*: feature squeezing, anomaly detectors, statistical discrepancy tests [2502.05637, 1810.07339].

## 4. Evaluation Metrics and Robustness Criteria

Robustness in AML is quantified via metrics that capture classifier accuracy under threat constraints:

- **Robust Accuracy**:
  $$
  \mathrm{RobustAccuracy} = \frac{1}{N} \sum_{i=1}^N \mathbf{1}\left( f(x_i + \delta_i^*) = y_i \right)
  $$
  with $\delta_i^* = \arg\max_{\| \delta \|_p \le \epsilon} \mathcal{L}(f_\theta(x_i+\delta), y_i)$ [1810.07339].
- **Certified Radius**: Size of the neighborhood (as in IBP, randomized smoothing) guaranteeing unchanged prediction.
- **Risk Assessment** (domain-specific): $\mathrm{Risk} \approx \mathrm{Effectiveness} \times \mathrm{Impact} \times \mathrm{Likelihood}$, combining attack effectiveness and threat actor attributes [2201.06093].

Best practices include clearly specified threat models (knowledge, capability, goal, perturbation norm), evaluation against adaptive adversaries, provable guarantees when possible, and reproducibility.

## 5. Hierarchical and Multi-Domain Taxonomy Structures

AML taxonomy is inherently hierarchical, accommodating multiple dimensions [2502.05637, 2302.09457, 2405.08834]:

- **Attack taxonomy**:  
  1. Threat model (knowledge, capability, goal)
  2. Evasion attacks (untargeted, targeted, optimization, transfer)
  3. Poisoning attacks (untargeted, targeted/backdoor)
  4. Model extraction and privacy inference

- **Defense taxonomy**:
  1. Input preprocessing and gradient masking
  2. Adversarial training, ensemble methods
  3. Certified defenses (IBP, randomized smoothing, LP relaxations)

- **Domain-specific extensions**:
  - Network security: risk grid based on discriminative and directive autonomy [1911.02621].
  - Cyber security: seven-axis taxonomy including attacker knowledge, training-set access, targeting, and output format [2007.02407].
  - Spacecraft AML: axes on mission objectives, resource constraints, learning stage, storage architecture, C&DH access, model exposure [2405.08834].
  - Wireless communications: axes of influence, phase, knowledge, attack goals; mirrored in layered defense taxonomy [2012.14392].

## 6. Open Challenges and Research Directions

Key open problems in AML reflect both foundational and engineering barriers [2502.05637, 2007.00753]:

- **Adaptive Attacks vs. Gradient Obfuscation**: Defenses based on gradient masking (e.g., defensive quantization, non-differentiable layers) are commonly bypassed using BPDA or transfer-based methods. There is an unresolved tension between masking for robustness and the need for clean gradients during training and verification.

- **Verification Scalability**: The complexity of exact formal verification ($O((2n)^L)$ for $L$-layer, $n$-neuron networks) precludes its application to large real-world models, forcing a trade-off to approximation or probabilistic bounds.

- **Certified Robustness vs. Model Capacity**: Tight bounds on robustness often require simplifying model architectures, incurring a trade-off between robustness radius, expressivity, accuracy, and computational load.

- **Distribution Shift and Real-World Threats**: Existing evaluations are primarily on clean digital benchmarks with $\ell_p$-bounded adversarial examples; real-world adversaries exploit physical attacks, sensor noise, and distributional shifts, challenging simplistic threat models.

- **Unified Theories and Benchmarks**: Lack of a unified theory of adversarial transferability or standardized, diverse benchmarks impedes reproducibility and principled progress [2302.09457, 2012.14392].

## 7. Synthesis and Outlook

AML taxonomy provides a rigorous scaffold for understanding and combating the evolving spectrum of adversarial threats. Hierarchical, multi-dimensional frameworks—rooted in formal threat models, life-cycle analysis, and robust optimization—distinguish attack and defense paradigms, clarify trade-offs, and expose gaps. The interplay between attack sophistication (e.g., adaptive, physical, or domain-constrained perturbations) and defensive innovation (empirical, certified, combined) ensures the AML landscape remains dynamic. Scalability, real-world deployment, and standardized, attack-agnostic robustness guarantees remain central targets for ongoing and future research [2502.05637, 2302.09457, 2405.08834].

Source: https://www.emergentmind.com/topics/adversarial-machine-learning-aml-taxonomy