Bi-Level Adversarial Training

• Bi-level adversarial training is defined as a nested optimization framework where a defender minimizes task loss while an adversary maximizes perturbations, leading to improved robustness.
• It employs specialized techniques like margin-based attacks and learnable adversaries to overcome the limitations of traditional min-max approaches.
• Empirical outcomes reveal reduced robust overfitting, faster convergence, and enhanced resistance to adversarial manipulations across diverse domains.

Bi-level adversarial training defines a class of machine learning defense and adaptation methodologies wherein the process of robust model development is formalized as a bi-level (nested) optimization problem. In this framework, two coupled players interact: the upper-level ("defender") seeks optimal model parameters or representations to minimize task loss or maximize robustness, while the lower-level ("attacker" or "adversary") seeks instead to maximize the model's loss through data manipulations or parameter perturbations within prescribed constraints. This contrasts with single-level or simple min-max approaches by decoupling the objectives at each level, often granting both theoretical and empirical improvements in out-of-domain robustness, adversarial resistance, and generalization. Recent advances employ bi-level adversarial training across domains including deep learning, federated learning, vision, generative modeling, and natural language processing, with growing emphasis on algorithmic tractability, tight theoretical guarantees, and domain-adaptive attack models.

1. Formalization and Core Bi-Level Structure

In canonical bi-level adversarial training, the problem is written as:

$$\min_{\theta} \ \mathbb{E}_{(x,y)\sim\mathcal{D}} \Bigl[ \ell(f_\theta(x+\eta^*),y)\Bigr] \quad \text{subject to} \quad \eta^* \in \arg\max_{\|\eta\| \leq \epsilon} \mathcal{J}(f_\theta,x+\eta,y)$$

Here, $\theta$ denotes model parameters, $f_\theta$ is the prediction function, $\ell(\cdot)$ is the upper-level loss (typically a differentiable surrogate for 0–1 risk or task loss), and the constraint is that the adversary selects $\eta$ to maximize a lower-level criterion $\mathcal{J}$ under feasible perturbations (e.g., norm-bounded $\epsilon$). Unlike zero-sum min-max formulations, $\mathcal{J}$ need not match $\ell$, and in modern methods, it is chosen to tightly align with true adversarial goals (such as misclassification margin) rather than mere loss inflation (Robey et al., 2023).

Generalizations allow the adversarial player to represent non-differentiable or combinatorial operations (e.g., label flips, input corruptions, network modules) and may be parameterized by learnable neural networks, policy controllers, or distributional agents.

2. Theoretical Analysis and Pitfalls of Surrogate Relaxations

Conventional adversarial training, as in Madry et al.'s min-max (zero-sum) paradigm, often approximates the inner maximization by maximizing a smooth surrogate loss such as cross-entropy. However, this introduces two core pathologies (Robey et al., 2023):

• Weak Attacks: Maximizing the surrogate loss may not guarantee true misclassification, as the surrogate can be inflated via irrelevant logit manipulations.
• Defender Misalignment: The defender, when trained against such weak attacks, may not achieve reductions in worst-case 0–1 risk but may instead overfit to non-adversarial failure modes, leading to robust overfitting and unreliable certified robustness.

By formalizing the lower-level adversary in terms of negative-margin maximization, exact misclassification-inducing perturbations can be located whenever they exist. For multiclass classifiers, solving $K-1$ margin subproblems and selecting the maximal violation yields adversarial examples that are always label-flipping when feasible.

Crucially, the resulting bi-level formulation yields a non-zero-sum game: the attacker drives the genuine decision boundary, while the defender operates over a surrogate upper bound, thereby restoring the adversarial game's integrity and eliminating robust overfitting observed in prior methods (Robey et al., 2023).

3. Algorithmic Instantiations and Practical Optimization

Algorithmic realizations of bi-level adversarial training vary according to the attacker's expressivity, the defender's surrogate, and computational constraints.

• Margin-based Attacks and BETA Algorithm: The Best Targeted Attack (BETA) computes, for each class $\theta$0, the perturbation maximally increasing the negative margin, then selects the most effective one. BETA is invoked on each SGD step, and a smoothed log-sum-exp variant (SBETA) enables gradient-based optimization throughout (Robey et al., 2023).
• Learned Attackers (Learning-to-Learn/L2L): Instead of hand-crafted solvers (e.g., PGD), the inner maximizer can be parameterized as a neural network $\theta$1, mapping inputs (and possibly gradients) to perturbations. The inner player is updated by gradient ascent to maximize downstream adversarial loss, while the outer player proceeds by (mini-batch) descent. Such architectures demonstrate improved efficiency and robustness over standard approaches (Jiang et al., 2018).
• Corruption Policy Generators (Hierarchical RL): In structured data corruption or generative settings, the inner problem may be implemented via policy-gradient reinforcement learning. This is essential when the space of manipulations is discrete or non-differentiable, e.g., selecting image noise, blur, or transmission artifacts for vision networks (Liu et al., 2023).
• Outlier Exclusion and Robust Training: By composing the upper-level loss as a robust rank-based objective (e.g., the Average of Ranked Range or AoRR), bi-level adversarial training can simultaneously excise outliers and focus learning on informative adversarial samples. Dual auxiliary thresholds can be maintained and updated by joint gradient methods, with provable statistical consistency (Hu et al., 2023).
• Fast Approximate and Single-Step Methods: Bi-level formulations reveal why standard single-step (Fast-AT/FGSM) approaches are prone to catastrophic overfitting. Advances such as Fast-BAT and FGSM-PCO introduce explicit or implicit gradient corrections, fused adversarial examples, and regularization terms to stabilize single-step adversarial optimization while preserving efficiency and robustness (Wang et al., 2024, Zhang et al., 2021).

4. Applications and Domain-Specific Adaptations

Bi-level adversarial training is not limited to standard classification but has been adapted to a diverse spectrum of machine learning tasks:

Domain/Task	Lower-Level Adversary	Upper-Level Defender / Objective
Image Classification, Vision	Margin maximization, PGD, learned attacker	Surrogate loss minimization (CE, margin)
Structured Corruption (Vision)	Discrete corruption policy via RL	IoU/detection loss minimization (e.g., Soft-IoU)
Generative Modeling (Face, LLMs)	Adversarial LoRA patch, GAN loss	Image realism/feature preservation, DPO, utility
Federated Learning	Poisoned agent models, consensus-based SDE	Robust loss on low-loss sublevel set
Clustering/Unsupervised	Data perturbation to change assignments	Robustness w.r.t. deviation measure $\theta$2

In open-weight LLMs, bi-level adversarial training can employ a state-aware adversarial hypernetwork generating malicious LoRA interventions, with the defender LoRA optimized to neutralize harmful effects while preserving utility. This architecture (AntiDote) achieves a substantial improvement in tamper-resistance without significant compromise in capabilities (Sanyal et al., 6 Sep 2025). Similar dual-level designs—e.g., BLAN for face verification—combine pixel and feature-level GAN discriminators, serving both sample realism and identification robustness (Li et al., 2017).

5. Theoretical Guarantees and Robustness Analysis

Bi-level adversarial training methodologies offer, in various settings, formal guarantees beyond empirical robustness:

• Surrogate alignment and risk minimization: Theoretical analysis shows that maintaining exact adversarial misclassification in the lower level is critical for aligning the upper-level surrogate minimization with true worst-case 0–1 risk reduction (Robey et al., 2023).
• Statistical consistency: Rank-based losses in the adversarial context satisfy $\theta$3-consistency with respect to adversarial 0/1 loss, ensuring that the surrogate minimization directly translates to actual adversarial performance (Hu et al., 2023).
• Uniform generalization: High-probability convergence rates for empirical risk minimization extend to the adversarial, bi-level setting for appropriately regularized losses.
• Global convergence in distributed settings: In consensus-based federated optimization, rigorous mean-field analysis guarantees exponential concentration of benign particles around the bi-level minimizer, provided hyperparameters such as quantile and robustness weight are tuned according to benign/malicious agent proportion (Trillos et al., 2024).
• Convergence rates with non-smooth or composite levels: For certain linearized or smoothed bi-level reductions, provable convergence to $\theta$4-stationarity matching single-level SGD can be established, with practical variations based on Hessian-free approximations (Zhang et al., 2021).

6. Empirical Outcomes, Limitations, and Best Practices

Empirical studies highlight the practical value of bi-level adversarial training:

• Robustness without overfitting: Bi-level non-zero-sum formulations eliminate the robust overfitting endemic to standard surrogate-based AT, displaying consistent last-iterate robustness across learning rate schedules (Robey et al., 2023).
• Superior attack resistance and efficiency: BETA-AT and its variants match or outperform widely used evaluators (AutoAttack) and are up to $\theta$5 faster by obviating repeated restarts (Robey et al., 2023). L2L methods deliver 2–3$\theta$6 speedup and superior robust accuracy versus traditional PGD (Jiang et al., 2018).
• Plug-and-play robustness in vision: Sample-adaptive corruption generators and spatial-frequency decoupling modules systematically enhance performance across a range of corruption types without substantial accuracy drop (Liu et al., 2023).
• Combined defense against data noise and adversaries: ORAT outperforms both adversarial training and re-weighting approaches for datasets with noisy labels or outliers, achieving consistent gains under multiple attack strategies (Hu et al., 2023).
• Mitigation of catastrophic overfitting: Fast-BAT and FGSM-PCO stabilize single-step AT methods, both preventing and recovering from collapse, and allow practical robustness on large-scale tasks (Wang et al., 2024, Zhang et al., 2021).
• Tamper resistance in open LLMs: AntiDote achieves up to 27.4% error reduction on a suite of 52 red-teaming attacks with under 0.5% utility degradation, surpassing prior fine-tuning defense and unlearning baselines (Sanyal et al., 6 Sep 2025).

Best practices emerging from these studies include:

• Substituting margin-based inner maximization for surrogate losses in the lower-level problem.
• Distinct objectives at each level: attacker must always target the true task boundary, while the defender employs a well-calibrated surrogate.
• Employing learnable attack parameterizations only when sufficient care is exercised to avoid attacker-defender degeneracy ("cycling").
• In distributed or federated contexts, explicit robust aggregation and hyperparameter adaptation are essential to sustained collective robustness (Trillos et al., 2024).

7. Open Problems and Future Developments

Despite empirical and theoretical successes, several challenges remain:

• Efficient and scalable inner maximization: For high-dimensional deep networks, exact adversarial margin maximization is non-trivial and can become a computational bottleneck.
• General-purpose solution strategies: While specific domains (vision, LLMs, federated learning) have crafted domain-adapted attack and defense pairs, unified algorithms for general bi-level adversarial programs are lacking. Recent theoretical frameworks have yet to deliver tractable solvers for all bi-level configurations, especially in clustering or unsupervised learning (Zheng et al., 29 Oct 2025).
• Provable convergence for non-convex or non-smooth settings: Theoretical analyses are incomplete for most practical deep-network instantiations, with few rigorous convergence proofs or finite-sample guarantees outside idealized cases.
• Adaptive or state-aware adversaries: Conditioning the inner adversary on rich model state (as in AntiDote) is now recognized as essential for defeating "static" attackers but increases the risk of instability or mode collapse without careful schedule management.

Ongoing research is expanding bi-level adversarial training into new problem classes (e.g., medical imaging, self-supervised learning, robust optimization for discrete combinatorics) and integrating advanced optimization and game-theoretic techniques to address the above bottlenecks. The principled design of bi-level games, informed by the precise alignment of theoretical surrogates, remains a central theme in the field.