Random Adversarial Training (RAT)

• Random Adversarial Training (RAT) is a family of methods that introduce stochasticity into adversarial objectives via noise injection on inputs, weights, or through batch sampling and transformations.
• RAT variants use different randomization mechanisms—such as Gaussian perturbations, Bernoulli sampling, and adversarial transformations—to influence the robustness–accuracy–cost trade-off across various norms and applications.
• RAT methods enhance multi-norm robustness and improve generalization in tasks like image classification and NLP by dynamically balancing computational cost with defense performance.

Searching arXiv for papers on “Random Adversarial Training” and related RAT variants. First, I’ll look for papers explicitly titled with “Random Adversarial Training” or close variants. Random Adversarial Training (RAT) is not a single canonical algorithm but an overloaded label used for several randomized variants of adversarial training and adversarial regularization. Across the literature, the common motif is the deliberate insertion of stochasticity into an adversarial objective—through input noise, weight noise, random batch selection for adversarial-example generation, or random label-preserving transformations—in order to alter the robustness–accuracy–cost trade-off. The term therefore covers at least four distinct lines of work: multi-norm robust training with injected noise (Araujo et al., 2019), Taylor-expanded adversarial training with Gaussian weight perturbations (Jin et al., 2023), Bernoulli-sampled adversarial training in NLP and biomedical information extraction (Chen et al., 14 Sep 2025, Chen et al., 2 Sep 2025), and semi-supervised consistency regularization based on adversarial transformations (Suzuki et al., 2019).

1. Terminological scope and core design pattern

The central pattern underlying RAT is a minimax objective in which the adversarial component is randomized rather than applied deterministically in the same way on every update. In some papers, the randomization is inserted into the model evaluation map, as in $\tilde f_\theta(x)=f_\theta(x+\eta)$ with $\eta$ sampled from a noise distribution (Araujo et al., 2019). In other papers, the randomization is placed in parameter space via $\tilde w=w+u$ with $u\sim\mathcal N(0,\sigma^2 I)$ (Jin et al., 2023), or at the batch level through a Bernoulli variable $k\sim\mathrm{Bernoulli}(p)$ that decides whether adversarial loss is included for a minibatch (Chen et al., 14 Sep 2025, Chen et al., 2 Sep 2025). A separate semi-supervised line uses randomized or adversarially optimized transformations $f_\phi$ rather than additive perturbations (Suzuki et al., 2019).

RAT variant	Randomization mechanism	Primary setting
Randomized adversarial training	Noise injected into inputs during adversarial training	Multi-norm image robustness
Randomized adversarial training via Taylor expansion	Gaussian perturbation of weights	Adversarial training with flat-minima emphasis
Random adversarial training	Bernoulli sampling of adversarial batches	BioIE and text classification
Regularization based on adversarial transformations	Adversarially chosen label-preserving transforms	Semi-supervised learning

This suggests that “RAT” is best treated as a family resemblance term rather than a uniquely specified method. A plausible implication is that comparisons across RAT papers are only meaningful after fixing the randomization locus, the threat model, and the optimization target.

2. Noise-injected randomized adversarial training for multi-norm robustness

A prominent early formulation appears in “Robust Neural Networks using Randomized Adversarial Training” (Araujo et al., 2019). There, standard single-norm adversarial training is written as

$$\min_\theta \; \mathbb E_{(x,y)\sim D}\Big[\max_{\|\tau\|_p\le \epsilon} L(f_\theta(x+\tau),y)\Big],$$

with separate instantiations for $p=\infty$ and $p=2$. RAT modifies the classifier by adding random noise at inference and training time,

$\tilde f_\theta(x)=f_\theta(x+\eta),$

where the noise can be Gaussian$\eta$0 or Uniform$\eta$1, and then performs adversarial training against $\eta$2 rather than against the deterministic model (Araujo et al., 2019).

The motivating theoretical claim is a “no free lunch” result between $\eta$3 and $\eta$4 defenses: in high dimension, the overlap between equal-volume $\eta$5 and $\eta$6 balls vanishes exponentially. The paper states a theorem showing that if $\eta$7 is the unit $\eta$8 ball and $\eta$9 is the equal-volume $\tilde w=w+u$0 ball, then

$\tilde w=w+u$1

Within that framework, the authors also study Mixed Adversarial Training (MAT), including MAT-Rand and MAT-Max, but report that these mixed schemes fail badly on C&W$\tilde w=w+u$2 attacks, whereas RAT better balances robustness across $\tilde w=w+u$3 and $\tilde w=w+u$4 attack families (Araujo et al., 2019).

The empirical evaluation is on CIFAR-10 with Wide-ResNet 28-10, white-box attacks with EOT for noise-injecting defenses, PGD inner loops of 10 iterations at training time, and 100 epochs on $\tilde w=w+u$5V100 GPUs. The reported attack suite includes PGD$\tilde w=w+u$6 with $\tilde w=w+u$7, PGD$\tilde w=w+u$8 with $\tilde w=w+u$9 chosen so that $u\sim\mathcal N(0,\sigma^2 I)$0, and C&W$u\sim\mathcal N(0,\sigma^2 I)$1 for 60 iterations. In the excerpted Table 3, RAT-$u\sim\mathcal N(0,\sigma^2 I)$2 + Uniform obtains Natural $u\sim\mathcal N(0,\sigma^2 I)$3, PGD$u\sim\mathcal N(0,\sigma^2 I)$4 $u\sim\mathcal N(0,\sigma^2 I)$5, PGD$u\sim\mathcal N(0,\sigma^2 I)$6 $u\sim\mathcal N(0,\sigma^2 I)$7, C&W$u\sim\mathcal N(0,\sigma^2 I)$8 $u\sim\mathcal N(0,\sigma^2 I)$9, and Min-Acc $k\sim\mathrm{Bernoulli}(p)$0; RAT-$k\sim\mathrm{Bernoulli}(p)$1 + Gaussian yields Min-Acc $k\sim\mathrm{Bernoulli}(p)$2; and the worst-case robustness of RAT-$k\sim\mathrm{Bernoulli}(p)$3 + Uniform exceeds the next best defense in that table (Araujo et al., 2019).

In this line of work, the role of randomization is not computational efficiency but cross-norm robustness. The method combines adversarial training with a smoothed classifier, making RAT conceptually close to randomized smoothing, but the objective remains an adversarial-training objective rather than a purely certification-oriented smoothing procedure.

3. Randomized weights and Taylor-expanded robust objectives

A distinct formulation appears in “Randomized Adversarial Training via Taylor Expansion” (Jin et al., 2023). Here the source of randomness is not input noise but Gaussian perturbation of model weights: $k\sim\mathrm{Bernoulli}(p)$4 The robust objective is TRADES-like: it combines a standard clean-data classification term with a robustness regularizer defined by the worst-case shift under a small $k\sim\mathrm{Bernoulli}(p)$5 attack, averaged over the weight noise. The paper writes

$k\sim\mathrm{Bernoulli}(p)$6

For small $k\sim\mathrm{Bernoulli}(p)$7, the method expands $k\sim\mathrm{Bernoulli}(p)$8 in a Taylor series and derives a surrogate loss with zeroth-, first-, and second-order terms. The first-order term depends on $k\sim\mathrm{Bernoulli}(p)$9, and the second-order term depends on $f_\phi$0 (Jin et al., 2023).

Algorithmically, each minibatch samples one noise vector $f_\phi$1, initializes adversarial examples with Gaussian noise $f_\phi$2, performs $f_\phi$3-step PGD under the perturbed model $f_\phi$4, and then updates $f_\phi$5 by minimizing a batch objective of the form

$f_\phi$6

The paper states that no high-order autograd is needed, only standard Jacobians and Hessians extracted via vector-Jacobian products; it also reports that the first- plus second-order variant takes roughly $f_\phi$7–$f_\phi$8 longer than TRADES alone and uses about $f_\phi$9 the memory (Jin et al., 2023).

The empirical results cover CIFAR-10, CIFAR-100, and SVHN with ResNet-18, WRN-34-10, VGG16, and MobileNetV2. On CIFAR-10 with ResNet-18 under the $$\min_\theta \; \mathbb E_{(x,y)\sim D}\Big[\max_{\|\tau\|_p\le \epsilon} L(f_\theta(x+\tau),y)\Big],$$0 threat model, the reported numbers for the first+second-order regularizer at $$\min_\theta \; \mathbb E_{(x,y)\sim D}\Big[\max_{\|\tau\|_p\le \epsilon} L(f_\theta(x+\tau),y)\Big],$$1 are Clean $$\min_\theta \; \mathbb E_{(x,y)\sim D}\Big[\max_{\|\tau\|_p\le \epsilon} L(f_\theta(x+\tau),y)\Big],$$2, PGD-20 $$\min_\theta \; \mathbb E_{(x,y)\sim D}\Big[\max_{\|\tau\|_p\le \epsilon} L(f_\theta(x+\tau),y)\Big],$$3, and AutoAttack $$\min_\theta \; \mathbb E_{(x,y)\sim D}\Big[\max_{\|\tau\|_p\le \epsilon} L(f_\theta(x+\tau),y)\Big],$$4. On WideResNet-34-10, adding RAT to AWP-TRADES raises AutoAttack on CIFAR-10 from $$\min_\theta \; \mathbb E_{(x,y)\sim D}\Big[\max_{\|\tau\|_p\le \epsilon} L(f_\theta(x+\tau),y)\Big],$$5 to $$\min_\theta \; \mathbb E_{(x,y)\sim D}\Big[\max_{\|\tau\|_p\le \epsilon} L(f_\theta(x+\tau),y)\Big],$$6 and on CIFAR-100 from $$\min_\theta \; \mathbb E_{(x,y)\sim D}\Big[\max_{\|\tau\|_p\le \epsilon} L(f_\theta(x+\tau),y)\Big],$$7 to $$\min_\theta \; \mathbb E_{(x,y)\sim D}\Big[\max_{\|\tau\|_p\le \epsilon} L(f_\theta(x+\tau),y)\Big],$$8, while also boosting clean accuracy by about $$\min_\theta \; \mathbb E_{(x,y)\sim D}\Big[\max_{\|\tau\|_p\le \epsilon} L(f_\theta(x+\tau),y)\Big],$$9 (Jin et al., 2023).

This version of RAT is explicitly motivated by loss-landscape flattening. The paper states that figures of the adversarial-loss surface show flatter landscapes in both input and weight directions compared with TRADES, and interprets the randomized-weight procedure as a mechanism for finding flat minima (Jin et al., 2023). That interpretation is specific to the Taylor-expansion formulation and should not be conflated with batch-sampling RAT variants.

4. Bernoulli-sampled adversarial loss in NLP and biomedical information extraction

In recent NLP work, “random adversarial training” refers to stochastic inclusion of adversarial loss on only a subset of minibatches. “RanAT4BIE: Random Adversarial Training for Biomedical Information Extraction” defines the batch loss as

$p=\infty$0

with $p=\infty$1 and attack generation performed only when $p=\infty$2 (Chen et al., 14 Sep 2025). The paper contrasts this with standard adversarial training using FGM, PGD, FreeLB, or SMART; uses PubMedBERT as the base encoder; adds perturbations to the input-token embeddings only; and freezes model parameters during inner maximization. Its experimental setup covers BioNER on BC5-chem, BC5-disease, BC2GM, and NCBI-disease, plus BioRE on CHEMPROT and DDI, with micro-F1 as the metric, batch size $p=\infty$3, max length $p=\infty$4, learning rate $p=\infty$5, $p=\infty$6, $p=\infty$7, $p=\infty$8, and sampling probability $p=\infty$9 (Chen et al., 14 Sep 2025).

The main quantitative claim in that paper is that adversarial training on PubMedBERT yields an average absolute F1 gain of $p=2$0 across six datasets, while RAT at $p=2$1 preserves nearly all of the improvement. In the reported table, BC5-chem is $p=2$2 baseline, $p=2$3 AT average, and $p=2$4 RAT average; DDI is $p=2$5 baseline, $p=2$6 AT average, and $p=2$7 RAT average. Computational cost is measured in xFP units, where baseline is $p=2$8, FGM AT is $p=2$9, RAT FGM is $\tilde f_\theta(x)=f_\theta(x+\eta),$0 with CRR $\tilde f_\theta(x)=f_\theta(x+\eta),$1, and RAT PGD is $\tilde f_\theta(x)=f_\theta(x+\eta),$2 with CRR $\tilde f_\theta(x)=f_\theta(x+\eta),$3; the paper states that with $\tilde f_\theta(x)=f_\theta(x+\eta),$4, multi-step RAT saves about $\tilde f_\theta(x)=f_\theta(x+\eta),$5 of adversarial overhead (Chen et al., 14 Sep 2025).

A closely related formulation appears in “Abex-rat: Synergizing Abstractive Augmentation and Adversarial Training for Classification of Occupational Accident Reports” (Chen et al., 2 Sep 2025). There the classifier $\tilde f_\theta(x)=f_\theta(x+\eta),$6 is a lightweight MLP over embeddings, the base loss is focal loss,

$\tilde f_\theta(x)=f_\theta(x+\eta),$7

with $\tilde f_\theta(x)=f_\theta(x+\eta),$8, and the adversarial perturbation is a one-step FGM perturbation under an $\tilde f_\theta(x)=f_\theta(x+\eta),$9 constraint,

$\eta$00

using $\eta$01 and a Bernoulli gate $\eta$02 with $\eta$03. The paper reports hyperparameters $\eta$04, batch size $\eta$05, and learning rate $\eta$06 with Adam, and states that only half of the batches incur the extra adversarial branch on average (Chen et al., 2 Sep 2025).

The reported ablation numbers in ABEX-RAT are Macro-F1 $\eta$07 for “w/o ABEX” (RAT on original data only), $\eta$08 for “w/o RAT” (ABEX plus standard training), and $\eta$09 for full ABEX-RAT on the OSHA dataset. The paper further reports minority-class recall improving from $\eta$10 without RAT to $\eta$11 in ABEX-RAT (Chen et al., 2 Sep 2025).

Within this NLP-oriented family, RAT is primarily an efficiency device. The stated motivation is that full adversarial training doubles or triples training cost, while Bernoulli-sampled adversarial updates reduce expected adversarial-sample generations by factor $\eta$12 and preserve nearly all of the measured generalization gain when $\eta$13 (Chen et al., 14 Sep 2025). This is operationally very different from the multi-norm and Taylor-expansion RATs.

5. Adversarial transformations in semi-supervised learning

Another established use of the acronym appears in “Adversarial Transformations for Semi-Supervised Learning,” where RAT stands for a regularization framework based on adversarial transformations rather than random adversarial example selection (Suzuki et al., 2019). The model output is $\eta$14, the data are split into labeled and unlabeled subsets, and the inner maximization seeks a worst-case label-preserving transformation

$\eta$15

The per-point regularizer is then

$\eta$16

and the semi-supervised objective combines supervised cross-entropy with an unlabeled consistency term and, in practice, an entropy penalty (Suzuki et al., 2019).

The admissible transformation family $\eta$17 includes additive noise, affine transformations, thin-plate spline transformations with a $\eta$18 control grid, flow-field deformations, and color distortion. Composite transformations can be built by chaining these operations. The training procedure uses an $\eta$19-rampup schedule based on the sigmoid ramp from Mean Teacher: $\eta$20 so that the perturbation region grows gradually during early training (Suzuki et al., 2019).

The reported experiments use a Wide ResNet 28$\eta$212 backbone trained for $\eta$22K iterations with Adam. On CIFAR-10 with $\eta$23 labels, the paper reports test error $\eta$24 for supervised only, $\eta$25 for VAT with $\eta$26-rampup, $\eta$27 for RAT with noise + affine and no rampup, and $\eta$28 for RAT with $\eta$29-rampup. On SVHN with $\eta$30 labels, the corresponding errors are $\eta$31, $\eta$32, $\eta$33, and $\eta$34. The paper also states that combining noise injection with affine transformation produced the largest gain over VAT in its CIFAR-10 ablation (Suzuki et al., 2019).

This line broadens the meaning of adversarial perturbation from additive norm-bounded noise to label-invariant transformations along the data manifold. It is therefore closer to adversarial consistency regularization than to classical robust optimization against norm-bounded attacks.

6. Theoretical context, recurring trade-offs, and limitations

A major theoretical backdrop for RAT-style methods is the difficulty of achieving robust generalization under adversarial training. “The curse of overparametrization in adversarial training: Precise analysis of robust generalization for random features regression” studies adversarially trained random-features regression under an $\eta$35-bounded adversary in the proportional high-dimensional limit $\eta$36 with $\eta$37 and $\eta$38 (Hassani et al., 2022). The paper derives an asymptotically exact robust-risk formula through a five-dimensional saddle-point problem and reports that, unlike standard generalization, robust generalization can degrade as overparameterization increases. Its summarized insights include: standard risk exhibits double descent with respect to $\eta$39, but for any $\eta$40 the robust-risk curve loses the safe descent in the overparameterized region; the worst robust error can occur at $\eta$41; and for strong adversaries or large $\eta$42, the global minimum of robust risk can even occur in the underparameterized regime $\eta$43 (Hassani et al., 2022).

Although this paper is not itself a RAT proposal, it sharpens the context in which RAT methods are designed. Across the surveyed RAT variants, three recurring trade-offs are explicit. First is the robustness-versus-cost trade-off: batch-sampled RAT in BioIE reduces adversarial overhead by roughly the sampling probability $\eta$44 while retaining nearly all measured gains at $\eta$45 (Chen et al., 14 Sep 2025), and ABEX-RAT states that random adversarial training enhances robustness without significant overhead (Chen et al., 2 Sep 2025). Second is the robustness-versus-clean-accuracy trade-off: the Taylor-expansion RAT is explicitly proposed to improve both robustness and clean accuracy by flattening the loss landscape (Jin et al., 2023). Third is the threat-model trade-off: the multi-norm RAT paper argues that a defense tailored to one norm does not automatically transfer to another, motivating noise-injected adversarial training as a way to improve the worst-case profile across attacks (Araujo et al., 2019).

Several limitations are also explicit in the source literature. RanAT4BIE notes that its randomness is currently at batch level, with no per-token or per-dimension sampling, and that a fixed $\eta$46 may be suboptimal; it suggests dynamic or adaptive scheduling, stratified sampling of embedding subspaces or token positions, and combination with gradient accumulation or mixed precision as future directions (Chen et al., 14 Sep 2025). The Taylor-expansion RAT incurs additional runtime and memory relative to TRADES (Jin et al., 2023). In the semi-supervised RAT setting, effectiveness depends on the availability of transformation families that genuinely preserve class labels (Suzuki et al., 2019).

A common misconception is that “RAT” names one established benchmark method. The literature instead uses the acronym for multiple non-equivalent constructions. This suggests that any technical discussion of RAT should specify, at minimum, whether the randomization acts in input space, weight space, attack scheduling, or transformation space; whether the objective is robust optimization, consistency regularization, or efficiency-oriented approximation; and whether the target evaluation is worst-case adversarial accuracy, clean accuracy, micro-F1 or macro-F1, or semi-supervised test error.