Stochastic Extended Adversarial Optimization

• Stochastic Extended Adversarial Optimization is a research umbrella that fuses adversarial objectives with stochastic gradients, dynamics, and losses to balance worst-case robustness with exploitable structure.
• SEA techniques leverage optimism, adaptive step sizing, and variance‐and‐drift decompositions to derive regret bounds that improve performance across online convex and min-max optimization settings.
• The framework has practical implications for robust training in adversarial MDPs, transfer attacks, and stochastic min-max games, advancing both theoretical guarantees and empirical robustness.

Stochastic Extended Adversarial Optimization denotes, in the most explicit formal usage, the Stochastically Extended Adversarial (SEA) model for online convex optimization, and more broadly a recurring design pattern in which adversarial objectives are extended by stochastic losses, stochastic dynamics, stochastic gradients, or stochastic computation paths. The common goal is to preserve adversarial robustness while exploiting structure that is weaker than i.i.d. stochasticity but stronger than a fully unconstrained adversary. This suggests a research umbrella rather than a single canonical formalism, spanning online convex optimization, episodic MDPs, stochastic min-max games, adversarial training, and black-box transfer attacks (Sachs et al., 2022, Tiapkin et al., 2024, Ramirez et al., 2023).

1. Formal models and problem quantities

The most formal instantiation is the SEA model for online convex optimization. At round $t$, the learner chooses $x_t \in X \subseteq \mathbb{R}^d$, nature chooses a distribution $\mathcal{D}_t$ over losses, a random function $f_t(\cdot)$ is drawn from $\mathcal{D}_t$, and the learner incurs $f_t(x_t)$. The expected loss is

$$F_t(x) = \mathbb{E}_{f_t \sim \mathcal{D}_t}[f_t(x)],$$

and expected static regret against a comparator $u \in X$ is

$$\mathbb{E}[\mathrm{Reg}_T(u)] = \mathbb{E}\Big[\sum_{t=1}^T f_t(x_t) - \sum_{t=1}^T f_t(u)\Big].$$

The interpolation between stochastic and adversarial regimes is quantified by the cumulative stochastic gradient variance

$$\sigma_{1:T}^2 = \mathbb{E}\Big[\sum_{t=1}^T \sigma_t^2\Big], \qquad \sigma_t^2 = \sup_{x \in X}\mathbb{E}\big[\|\nabla f_t(x)-\nabla F_t(x)\|_2^2\big],$$

and the cumulative adversarial gradient variation

$x_t \in X \subseteq \mathbb{R}^d$0

The same decomposition also appears in the earlier smooth-expected OCO formulation of the stochastically extended adversary, which uses averaged quantities such as $x_t \in X \subseteq \mathbb{R}^d$1 and $x_t \in X \subseteq \mathbb{R}^d$2 and recovers fully i.i.d., fully adversarial, and adversarially corrupted i.i.d. regimes as special cases (Sachs et al., 2022, Chen et al., 2023).

In this formalism, fully adversarial data corresponds to $x_t \in X \subseteq \mathbb{R}^d$3, so $x_t \in X \subseteq \mathbb{R}^d$4 and temporal difficulty is carried by $x_t \in X \subseteq \mathbb{R}^d$5. Fully i.i.d. data corresponds to $x_t \in X \subseteq \mathbb{R}^d$6 for all $x_t \in X \subseteq \mathbb{R}^d$7, so $x_t \in X \subseteq \mathbb{R}^d$8 and only $x_t \in X \subseteq \mathbb{R}^d$9 remains. Intermediate regimes include adversarial corruptions of i.i.d. losses, random-order models, distribution shift, and distribution switching (Sachs et al., 2022).

A related but distinct formalization appears in adversarial MDPs with stochastic transitions. There the transition kernels are fixed across episodes and unknown to the learner, while the reward functions are chosen by an oblivious adversary and revealed in full only at the end of each episode. The learner then competes with the best fixed nonstationary policy in hindsight, and the stochastic-adversarial split is between unknown dynamics and adversarial rewards rather than between gradient noise and gradient drift (Tiapkin et al., 2024).

2. Online convex optimization: optimism, smoothness, and parameter-free adaptation

The core algorithmic mechanism in SEA OCO is optimism. In the smooth-expected convex setting, Optimistic Follow-the-Regularized-Leader uses the previous gradient as prediction, $\mathcal{D}_t$0, together with an adaptive step size

$\mathcal{D}_t$1

Under convexity, bounded gradients, and $\mathcal{D}_t$2-smooth expected losses, the resulting regret bound replaces the classical $\mathcal{D}_t$3 scale by a variance-and-drift scale:

$\mathcal{D}_t$4

When $\mathcal{D}_t$5 is $\mathcal{D}_t$6-strongly convex, the regret becomes logarithmic in $\mathcal{D}_t$7 and depends on $\mathcal{D}_t$8 rather than on $\mathcal{D}_t$9 (Sachs et al., 2022).

Optimistic Online Mirror Descent sharpens this picture. For convex and smooth expected losses it achieves the same $f_t(\cdot)$0 regret order without requiring convexity of the individual losses $f_t(\cdot)$1. For strongly convex and smooth expected losses it improves the logarithmic dependence from $f_t(\cdot)$2 to

$f_t(\cdot)$3

and for exp-concave and smooth losses it yields

$f_t(\cdot)$4

The same framework also gives the first dynamic regret guarantee for the SEA model,

$f_t(\cdot)$5

and extends to non-smooth convex losses via implicit optimistic updates (Chen et al., 2023).

A complementary line removes prior knowledge of problem constants. Parameter-free SEA algorithms based on Optimistic Online Newton Step eliminate the need to know the domain diameter $f_t(\cdot)$6 and the Lipschitz constant $f_t(\cdot)$7. In the unknown-diameter setting, the comparator-adaptive regret is

$f_t(\cdot)$8

and the more general comparator- and Lipschitz-adaptive construction preserves the same SEA dependence while using clipped gradients and dynamic radius growth (Wang et al., 6 Oct 2025).

Historically, this line extends earlier “best-of-both-worlds” results in which Squint and MetaGrad retained adversarial worst-case guarantees while automatically adapting to favorable stochastic environments satisfying a Bernstein condition. In that setting the fast-rate exponent is

$f_t(\cdot)$9

which interpolates between the adversarial $\mathcal{D}_t$0 regime at $\mathcal{D}_t$1 and logarithmic-type behavior at $\mathcal{D}_t$2 (Koolen et al., 2016).

3. Adversarial MDPs with stochastic dynamics

In episodic finite-horizon MDPs, Stochastic Extended Adversarial Optimization takes the form of unknown but fixed stochastic transitions combined with adversarially chosen rewards. The setting consists of $\mathcal{D}_t$3 episodes of horizon $\mathcal{D}_t$4, state space size $\mathcal{D}_t$5, action space size $\mathcal{D}_t$6, a fixed initial state $\mathcal{D}_t$7, and reward functions $\mathcal{D}_t$8 with $\mathcal{D}_t$9. The adversary is oblivious: the full reward sequence is fixed before interaction begins. Feedback is full-information at the end of each episode, since the entire $f_t(x_t)$0 is revealed (Tiapkin et al., 2024).

For any policy $f_t(x_t)$1 and episode $f_t(x_t)$2, the value, action-value, and advantage functions are

$f_t(x_t)$3

$f_t(x_t)$4

$f_t(x_t)$5

Regret is measured against the best fixed nonstationary policy in hindsight:

$f_t(x_t)$6

The algorithm APO-MVP alternates between transition estimation, epoch freezing, dynamic programming, and black-box online linear optimization over advantage vectors. Epochs are triggered by power-of-two visit counts. Within an epoch, empirical transitions and bonuses are frozen:

$f_t(x_t)$7

with $f_t(x_t)$8. Backward dynamic programming then computes

$f_t(x_t)$9

and the advantages $$F_t(x) = \mathbb{E}_{f_t \sim \mathcal{D}_t}[f_t(x)],$$0 are fed to independent OLO instances on $$F_t(x) = \mathbb{E}_{f_t \sim \mathcal{D}_t}[f_t(x)],$$1.

With polynomial-potential or exponential-potential OLO, APO-MVP satisfies, with probability at least $$F_t(x) = \mathbb{E}_{f_t \sim \mathcal{D}_t}[f_t(x)],$$2,

$$F_t(x) = \mathbb{E}_{f_t \sim \mathcal{D}_t}[f_t(x)],$$3

$$F_t(x) = \mathbb{E}_{f_t \sim \mathcal{D}_t}[f_t(x)],$$4

Up to logarithms, the leading scaling is $$F_t(x) = \mathbb{E}_{f_t \sim \mathcal{D}_t}[f_t(x)],$$5. This improves the best previous adversarial bound UC-O-REPS by a factor $$F_t(x) = \mathbb{E}_{f_t \sim \mathcal{D}_t}[f_t(x)],$$6, matches the minimax lower bound $$F_t(x) = \mathbb{E}_{f_t \sim \mathcal{D}_t}[f_t(x)],$$7 in its dependence on $$F_t(x) = \mathbb{E}_{f_t \sim \mathcal{D}_t}[f_t(x)],$$8, $$F_t(x) = \mathbb{E}_{f_t \sim \mathcal{D}_t}[f_t(x)],$$9, and $u \in X$0, and narrows the adversarial-versus-stochastic gap to polynomial factors in $u \in X$1 (Tiapkin et al., 2024).

A distinctive methodological feature is that the analysis is occupancy-measure free. Instead, it combines OLO-based policy optimization in the style of Jonckheere–Mertikopoulos–Stoltz with refined martingale and optional-skipping arguments for transition estimation. This makes the procedure both black-box in the policy-update layer and explicit in the dynamic-programming layer (Tiapkin et al., 2024).

4. Stochastic min-max optimization and distributed game dynamics

In stochastic saddle-point optimization, the same theme appears as an attempt to retain the stabilizing effect of optimism while reducing its sensitivity to gradient noise. Omega replaces the optimistic correction term by an exponential moving average of historical gradients:

$u \in X$2

$u \in X$3

This preserves the one-call cost of independent-samples optimistic gradient, stores only an EMA state of size $u \in X$4, and empirically improves robustness to noise in stochastic bilinear and quadratic-linear games. The method does not come with new formal convergence guarantees, and the momentum variant OmegaM can diverge on bilinear games (Ramirez et al., 2023).

A different extension is Randomized SGDA, which turns the deterministic inner maximization loop of epoch SGDA into a geometric random variable. At each iteration, a descent step in $u \in X$5 is taken with probability $u \in X$6, and an ascent step in $u \in X$7 with probability $u \in X$8, so the number of inner ascent steps between two descent steps is geometrically distributed with expectation $u \in X$9. The crucial step-size coupling is

$$\mathbb{E}[\mathrm{Reg}_T(u)] = \mathbb{E}\Big[\sum_{t=1}^T f_t(x_t) - \sum_{t=1}^T f_t(u)\Big].$$0

where $$\mathbb{E}[\mathrm{Reg}_T(u)] = \mathbb{E}\Big[\sum_{t=1}^T f_t(x_t) - \sum_{t=1}^T f_t(u)\Big].$$1. This loopless design yields the first almost sure convergence rates for SGDA-type methods in the nonconvex–strongly concave setting, with deterministic-gradient rate $$\mathbb{E}[\mathrm{Reg}_T(u)] = \mathbb{E}\Big[\sum_{t=1}^T f_t(x_t) - \sum_{t=1}^T f_t(u)\Big].$$2, stochastic fixed-step complexity $$\mathbb{E}[\mathrm{Reg}_T(u)] = \mathbb{E}\Big[\sum_{t=1}^T f_t(x_t) - \sum_{t=1}^T f_t(u)\Big].$$3, and large-minibatch complexity $$\mathbb{E}[\mathrm{Reg}_T(u)] = \mathbb{E}\Big[\sum_{t=1}^T f_t(x_t) - \sum_{t=1}^T f_t(u)\Big].$$4 (Sebbouh et al., 2021).

Diffusion Stochastic Same-Sample Optimistic Gradient addresses a different failure mode: the large-batch requirement of conventional stochastic optimistic methods in nonconvex min-max problems. Its same-sample optimistic gradients reuse the current stochastic sample when recomputing the “past” gradient:

$$\mathbb{E}[\mathrm{Reg}_T(u)] = \mathbb{E}\Big[\sum_{t=1}^T f_t(x_t) - \sum_{t=1}^T f_t(u)\Big].$$5

$$\mathbb{E}[\mathrm{Reg}_T(u)] = \mathbb{E}\Big[\sum_{t=1}^T f_t(x_t) - \sum_{t=1}^T f_t(u)\Big].$$6

In the distributed version, each agent performs an adapt-then-combine diffusion step under a left-stochastic communication protocol. Under a nonconvex–PL setting, the method attains a primal best-iterate rate $$\mathbb{E}[\mathrm{Reg}_T(u)] = \mathbb{E}\Big[\sum_{t=1}^T f_t(x_t) - \sum_{t=1}^T f_t(u)\Big].$$7, a dual last-iterate optimality-gap rate $$\mathbb{E}[\mathrm{Reg}_T(u)] = \mathbb{E}\Big[\sum_{t=1}^T f_t(x_t) - \sum_{t=1}^T f_t(u)\Big].$$8, and asynchronous joint $$\mathbb{E}[\mathrm{Reg}_T(u)] = \mathbb{E}\Big[\sum_{t=1}^T f_t(x_t) - \sum_{t=1}^T f_t(u)\Big].$$9-stationarity after $$\sigma_{1:T}^2 = \mathbb{E}\Big[\sum_{t=1}^T \sigma_t^2\Big], \qquad \sigma_t^2 = \sup_{x \in X}\mathbb{E}\big[\|\nabla f_t(x)-\nabla F_t(x)\|_2^2\big],$$0 iterations, while avoiding the conventional large-batch requirement by keeping the batch size at $$\sigma_{1:T}^2 = \mathbb{E}\Big[\sum_{t=1}^T \sigma_t^2\Big], \qquad \sigma_t^2 = \sup_{x \in X}\mathbb{E}\big[\|\nabla f_t(x)-\nabla F_t(x)\|_2^2\big],$$1 (Cai et al., 2024).

5. Robustness, purification, and transfer under stochastic extension

In adversarial robustness, the stochastic extension often moves from the optimization protocol to the model, the prior, or the ensemble over attack surrogates. ScoreOpt exemplifies a test-time generative-prior formulation: given an adversarial image $$\sigma_{1:T}^2 = \mathbb{E}\Big[\sum_{t=1}^T \sigma_t^2\Big], \qquad \sigma_t^2 = \sup_{x \in X}\mathbb{E}\big[\|\nabla f_t(x)-\nabla F_t(x)\|_2^2\big],$$2, it optimizes a purified image by maximizing the posterior under a score-based diffusion prior, using either the diffusion-prior loss

$$\sigma_{1:T}^2 = \mathbb{E}\Big[\sum_{t=1}^T \sigma_t^2\Big], \qquad \sigma_t^2 = \sup_{x \in X}\mathbb{E}\big[\|\nabla f_t(x)-\nabla F_t(x)\|_2^2\big],$$3

or the score-regularized objective

$$\sigma_{1:T}^2 = \mathbb{E}\Big[\sum_{t=1}^T \sigma_t^2\Big], \qquad \sigma_t^2 = \sup_{x \in X}\mathbb{E}\big[\|\nabla f_t(x)-\nabla F_t(x)\|_2^2\big],$$4

The optimization is stochastic because each step samples a noise level and Gaussian perturbations, but it avoids sequential reverse-SDE simulation. On CIFAR-10 under BPDA+EOT and $$\sigma_{1:T}^2 = \mathbb{E}\Big[\sum_{t=1}^T \sigma_t^2\Big], \qquad \sigma_t^2 = \sup_{x \in X}\mathbb{E}\big[\|\nabla f_t(x)-\nabla F_t(x)\|_2^2\big],$$5, ScoreOpt-N reports $$\sigma_{1:T}^2 = \mathbb{E}\Big[\sum_{t=1}^T \sigma_t^2\Big], \qquad \sigma_t^2 = \sup_{x \in X}\mathbb{E}\big[\|\nabla f_t(x)-\nabla F_t(x)\|_2^2\big],$$6 standard accuracy and $$\sigma_{1:T}^2 = \mathbb{E}\Big[\sum_{t=1}^T \sigma_t^2\Big], \qquad \sigma_t^2 = \sup_{x \in X}\mathbb{E}\big[\|\nabla f_t(x)-\nabla F_t(x)\|_2^2\big],$$7 robust accuracy (Zhang et al., 2023).

A second line uses stochastic model averaging during adversarial training. SWAAT aggregates temporal weight states over a sliding window,

$$\sigma_{1:T}^2 = \mathbb{E}\Big[\sum_{t=1}^T \sigma_t^2\Big], \qquad \sigma_t^2 = \sup_{x \in X}\mathbb{E}\big[\|\nabla f_t(x)-\nabla F_t(x)\|_2^2\big],$$8

and periodically replaces the working model by the average while recalibrating batch normalization. This yields an ensemble-like effect without training multiple networks. On CIFAR-10 with WRN-28-10, PGD-20 robustness increases from $$\sigma_{1:T}^2 = \mathbb{E}\Big[\sum_{t=1}^T \sigma_t^2\Big], \qquad \sigma_t^2 = \sup_{x \in X}\mathbb{E}\big[\|\nabla f_t(x)-\nabla F_t(x)\|_2^2\big],$$9 for PGD-AT to $x_t \in X \subseteq \mathbb{R}^d$00 for SWAAT, while natural accuracy rises from $x_t \in X \subseteq \mathbb{R}^d$01 to $x_t \in X \subseteq \mathbb{R}^d$02 (Hwang et al., 2020).

A third line randomizes internal computation directly. Stochastic combinatorial ensembles insert denoising operators such as VAEs at random intermediate positions, generating an exponentially large ensemble with linear expected cost. The transferability analysis is explicitly gradient-geometric: Pearson correlations between cosine similarity and transfer success reach $x_t \in X \subseteq \mathbb{R}^d$03 for FGS, $x_t \in X \subseteq \mathbb{R}^d$04 for IGS, and $x_t \in X \subseteq \mathbb{R}^d$05 for CW2, indicating that gradient alignment largely controls cross-model attack transfer (Adam et al., 2018). Stochastic Local Winner-Takes-All networks similarly randomize the active subnetwork through sampled winners in local competition blocks, trained under a variational Bayesian formulation and combined with PGD-based adversarial training. On CIFAR-10 with WRN-34-10, the resulting model reports $x_t \in X \subseteq \mathbb{R}^d$06 natural accuracy and $x_t \in X \subseteq \mathbb{R}^d$07 AutoAttack robustness (Panousis et al., 2021).

The stochastic extension also appears on the attack side. SVRE treats ensemble transfer attacks as stochastic optimization over the surrogate-model index and applies an SVRG-style control variate:

$x_t \in X \subseteq \mathbb{R}^d$08

This reduces cross-model gradient variance and improves black-box transfer. On nine defense models, SVRE-SI-TI-DIM raises average transfer success from $x_t \in X \subseteq \mathbb{R}^d$09 for Ens-SI-TI-DIM to $x_t \in X \subseteq \mathbb{R}^d$10 (Xiong et al., 2021).

6. Structured bandits, stochastic decision sets, and open problems

The extension from stochastic losses to stochastic feasibility appears in online combinatorial optimization with stochastic decision sets and adversarial losses. There the available action set $x_t \in X \subseteq \mathbb{R}^d$11 is drawn each round from a fixed unknown distribution, losses are chosen by an adaptive adversary, and regret is measured against the best fixed policy $x_t \in X \subseteq \mathbb{R}^d$12 that maps each available set to a feasible action. FTPL combined with the Counting Asleep Times estimator yields expected regret bounds under full information, restricted information, and semi-bandit feedback. In the restricted-information setting,

$x_t \in X \subseteq \mathbb{R}^d$13

which implies the universal rate

$x_t \in X \subseteq \mathbb{R}^d$14

In the semi-bandit setting, CAT plus geometric resampling gives

$x_t \in X \subseteq \mathbb{R}^d$15

and improves the sleeping-bandit guarantee to

$x_t \in X \subseteq \mathbb{R}^d$16

under stochastic availability (Neu et al., 28 Apr 2026).

A related application-level perspective appears in hierarchical adversarial bandits for hyperparameter tuning. HyperArm Bandit Optimization treats each hyperparameter as a super-arm and its candidate values as sub-arms, using EXP3 at both levels. This is motivated by the claim that hyperparameter rewards may be noisy or non-stationary because of interactions among hyperparameters and changing model states, so purely stochastic bandit assumptions are often too rigid. The method inherits $x_t \in X \subseteq \mathbb{R}^d$17 adversarial-bandit regret at each layer and is positioned as a robust alternative when the optimization landscape exhibits pseudo-adversarial drift (Karroum et al., 13 Mar 2025).

Several limitations recur across the literature. In adversarial MDPs, extending the $x_t \in X \subseteq \mathbb{R}^d$18-type dependence to bandit reward feedback remains open, fully matching the lower bound in $x_t \in X \subseteq \mathbb{R}^d$19 remains open, and adaptive adversaries are expected to reintroduce the $x_t \in X \subseteq \mathbb{R}^d$20 penalty (Tiapkin et al., 2024). In SEA OCO, bandit feedback is not covered by the smooth-expected analyses, and even the parameter-free algorithms retain comparator-dependent polynomial terms on unbounded domains (Sachs et al., 2022, Wang et al., 6 Oct 2025). In stochastic min-max games, Omega and its momentum variant remain largely empirical, with no new formal convergence bounds (Ramirez et al., 2023). In distributed same-sample optimism, the theory depends on dual-side PL structure, unbiased gradient oracles, and fixed left-stochastic communication topologies (Cai et al., 2024).

Taken together, these directions establish a consistent technical theme. Stochastic Extended Adversarial Optimization is characterized by optimism, variance-or-drift decompositions, epoch freezing or same-sample coupling, and explicit mechanisms for converting adversarially robust procedures into ones that exploit stochastic regularity without collapsing to purely stochastic assumptions. The most mature theory currently lies in SEA online convex optimization and adversarial MDPs, while the richest empirical developments lie in adversarial robustness, stochastic purification, and transfer attacks.