---
title: Random Adversarial Training (RAT)
url: https://www.emergentmind.com/topics/random-adversarial-training-rat
type: topic
---

# Random Adversarial Training (RAT)

Searching arXiv for papers on “Random Adversarial Training” and related RAT variants.
First, I’ll look for papers explicitly titled with “Random Adversarial Training” or close variants.
Random Adversarial Training (RAT) is not a single canonical algorithm but an overloaded label used for several randomized variants of adversarial training and adversarial regularization. Across the literature, the common motif is the deliberate insertion of stochasticity into an adversarial objective—through input noise, weight noise, random batch selection for adversarial-example generation, or random label-preserving transformations—in order to alter the robustness–accuracy–cost trade-off. The term therefore covers at least four distinct lines of work: multi-norm robust training with injected noise [1903.10219], Taylor-expanded adversarial training with Gaussian weight perturbations [2303.10653], Bernoulli-sampled adversarial training in NLP and biomedical information extraction [2509.11191, 2509.02072], and semi-supervised consistency regularization based on adversarial transformations [1911.06181].

## 1. Terminological scope and core design pattern

The central pattern underlying RAT is a minimax objective in which the adversarial component is randomized rather than applied deterministically in the same way on every update. In some papers, the randomization is inserted into the model evaluation map, as in $\tilde f_\theta(x)=f_\theta(x+\eta)$ with $\eta$ sampled from a noise distribution [1903.10219]. In other papers, the randomization is placed in parameter space via $\tilde w=w+u$ with $u\sim\mathcal N(0,\sigma^2 I)$ [2303.10653], or at the batch level through a Bernoulli variable $k\sim\mathrm{Bernoulli}(p)$ that decides whether adversarial loss is included for a minibatch [2509.11191, 2509.02072]. A separate semi-supervised line uses randomized or adversarially optimized transformations $f_\phi$ rather than additive perturbations [1911.06181].

| RAT variant | Randomization mechanism | Primary setting |
|---|---|---|
| Randomized adversarial training | Noise injected into inputs during adversarial training | Multi-norm image robustness |
| Randomized adversarial training via Taylor expansion | Gaussian perturbation of weights | Adversarial training with flat-minima emphasis |
| Random adversarial training | Bernoulli sampling of adversarial batches | BioIE and text classification |
| Regularization based on adversarial transformations | Adversarially chosen label-preserving transforms | Semi-supervised learning |

This suggests that “RAT” is best treated as a family resemblance term rather than a uniquely specified method. A plausible implication is that comparisons across RAT papers are only meaningful after fixing the randomization locus, the threat model, and the optimization target.

## 2. Noise-injected randomized adversarial training for multi-norm robustness

A prominent early formulation appears in “Robust Neural Networks using Randomized Adversarial Training” [1903.10219]. There, standard single-norm adversarial training is written as
\[
\min_\theta \; \mathbb E_{(x,y)\sim D}\Big[\max_{\|\tau\|_p\le \epsilon} L(f_\theta(x+\tau),y)\Big],
\]
with separate instantiations for $p=\infty$ and $p=2$. RAT modifies the classifier by adding random noise at inference and training time,
\[
\tilde f_\theta(x)=f_\theta(x+\eta),
\]
where the noise can be Gaussian$(0,\sigma_1^2 I)$ or Uniform$([-\sigma_2,\sigma_2]^d)$, and then performs adversarial training against $\tilde f_\theta$ rather than against the deterministic model [1903.10219].

The motivating theoretical claim is a “no free lunch” result between $\ell_\infty$ and $\ell_2$ defenses: in high dimension, the overlap between equal-volume $\ell_\infty$ and $\ell_2$ balls vanishes exponentially. The paper states a theorem showing that if $B_{\infty,d}(1)$ is the unit $\ell_\infty$ ball and $B_{2,d}(r_2(d))$ is the equal-volume $\ell_2$ ball, then
\[
\frac{\mathrm{Vol}(B_{2,d}(r_2(d))\cap B_{\infty,d}(1))}{\mathrm{Vol}(B_{\infty,d}(1))}\to 0
\quad\text{exponentially as } d\to\infty.
\]
Within that framework, the authors also study Mixed Adversarial Training (MAT), including MAT-Rand and MAT-Max, but report that these mixed schemes fail badly on C\&W$_2$ attacks, whereas RAT better balances robustness across $\ell_\infty$ and $\ell_2$ attack families [1903.10219].

The empirical evaluation is on CIFAR-10 with Wide-ResNet 28-10, white-box attacks with EOT for noise-injecting defenses, PGD inner loops of 10 iterations at training time, and 100 epochs on $4\times$V100 GPUs. The reported attack suite includes PGD$_\infty$ with $\epsilon_\infty=0.031$, PGD$_2$ with $\epsilon_2=0.83$ chosen so that $\mathrm{Vol}_2(\epsilon_2)=\mathrm{Vol}_\infty(\epsilon_\infty)$, and C\&W$_2$ for 60 iterations. In the excerpted Table 3, RAT-$\infty$ + Uniform obtains Natural $0.80$, PGD$_\infty$ $0.40$, PGD$_2$ $0.43$, C\&W$_2$ $0.41$, and Min-Acc $0.39$; RAT-$2$ + Gaussian yields Min-Acc $0.30$; and the worst-case robustness of RAT-$\infty$ + Uniform exceeds the next best defense in that table [1903.10219].

In this line of work, the role of randomization is not computational efficiency but cross-norm robustness. The method combines adversarial training with a smoothed classifier, making RAT conceptually close to randomized smoothing, but the objective remains an adversarial-training objective rather than a purely certification-oriented smoothing procedure.

## 3. Randomized weights and Taylor-expanded robust objectives

A distinct formulation appears in “Randomized Adversarial Training via Taylor Expansion” [2303.10653]. Here the source of randomness is not input noise but Gaussian perturbation of model weights:
\[
\tilde w = w + u,\qquad u\sim\mathcal N(0,\sigma^2 I).
\]
The robust objective is TRADES-like: it combines a standard clean-data classification term with a robustness regularizer defined by the worst-case shift under a small $\ell_p$ attack, averaged over the weight noise. The paper writes
\[
\min_w \; \mathbb E_u\Big[\mathbb E_{s\sim D}[L(f_w(s),y)] + \mathbb E_{s\sim D}\max_{\|s'-s\|_p\le \epsilon}\frac{1}{\lambda}L(f_{w+u}(s),f_{w+u}(s'))\Big].
\]
For small $u$, the method expands $g_s(w+u)$ in a Taylor series and derives a surrogate loss with zeroth-, first-, and second-order terms. The first-order term depends on $g'_s(w)^T u$, and the second-order term depends on $u^T g''_s(w) u$ [2303.10653].

Algorithmically, each minibatch samples one noise vector $u$, initializes adversarial examples with Gaussian noise $\mathcal N(0,10^{-3}I)$, performs $K$-step PGD under the perturbed model $f_{w+u}$, and then updates $w$ by minimizing a batch objective of the form
\[
L_{\rm batch}=\frac1n\sum_i\Big[L(f_w(s_i),y_i)+\frac1\lambda\big(L_0+L_1+\tfrac12 L_2\big)\Big].
\]
The paper states that no high-order autograd is needed, only standard Jacobians and Hessians extracted via vector-Jacobian products; it also reports that the first- plus second-order variant takes roughly $1.4$–$1.6\times$ longer than TRADES alone and uses about $2\times$ the memory [2303.10653].

The empirical results cover CIFAR-10, CIFAR-100, and SVHN with ResNet-18, WRN-34-10, VGG16, and MobileNetV2. On CIFAR-10 with ResNet-18 under the $\ell_\infty$ threat model, the reported numbers for the first+second-order regularizer at $\eta=0.2$ are Clean $83.55\to84.13\%$, PGD-20 $54.86\to54.91\%$, and AutoAttack $48.8\to50.3\%$. On WideResNet-34-10, adding RAT to AWP-TRADES raises AutoAttack on CIFAR-10 from $56.2\%$ to $57.4\%$ and on CIFAR-100 from $28.6\%$ to $30.2\%$, while also boosting clean accuracy by about $1\%$ [2303.10653].

This version of RAT is explicitly motivated by loss-landscape flattening. The paper states that figures of the adversarial-loss surface show flatter landscapes in both input and weight directions compared with TRADES, and interprets the randomized-weight procedure as a mechanism for finding flat minima [2303.10653]. That interpretation is specific to the Taylor-expansion formulation and should not be conflated with batch-sampling RAT variants.

## 4. Bernoulli-sampled adversarial loss in NLP and biomedical information extraction

In recent NLP work, “random adversarial training” refers to stochastic inclusion of adversarial loss on only a subset of minibatches. “RanAT4BIE: Random Adversarial Training for Biomedical Information Extraction” defines the batch loss as
\[
\mathcal L_{\rm batch}(\theta)=\mathcal L(f(x;\theta),y)+\mathbf 1[k_i=1]\cdot \mathcal L(f(x_{\rm adv};\theta),y),
\]
with $k_i\sim\mathrm{Bernoulli}(p)$ and attack generation performed only when $k_i=1$ [2509.11191]. The paper contrasts this with standard adversarial training using FGM, PGD, FreeLB, or SMART; uses PubMedBERT as the base encoder; adds perturbations to the input-token embeddings only; and freezes model parameters during inner maximization. Its experimental setup covers BioNER on BC5-chem, BC5-disease, BC2GM, and NCBI-disease, plus BioRE on CHEMPROT and DDI, with micro-F1 as the metric, batch size $32$, max length $128$, learning rate $3\times10^{-5}$, $\epsilon_{\rm FGM}=10^{-5}$, $\epsilon_{\rm PGD}=10^{-2}$, $T=3$, and sampling probability $p=0.5$ [2509.11191].

The main quantitative claim in that paper is that adversarial training on PubMedBERT yields an average absolute F1 gain of $+0.5\%$ across six datasets, while RAT at $p=0.5$ preserves nearly all of the improvement. In the reported table, BC5-chem is $93.33\%$ baseline, $93.60\%$ AT average, and $93.60\%$ RAT average; DDI is $82.36\%$ baseline, $82.93\%$ AT average, and $83.39\%$ RAT average. Computational cost is measured in xFP units, where baseline is $3$, FGM AT is $6$, RAT FGM is $4.5$ with CRR $25\%$, and RAT PGD is $3+1.5S$ with CRR $50\%$; the paper states that with $S\to\infty$, multi-step RAT saves about $50\%$ of adversarial overhead [2509.11191].

A closely related formulation appears in “Abex-rat: Synergizing Abstractive Augmentation and Adversarial Training for Classification of Occupational Accident Reports” [2509.02072]. There the classifier $f(\mathbf x;\theta)$ is a lightweight MLP over embeddings, the base loss is focal loss,
\[
\mathrm{FL}(p_t)=-\alpha_t(1-p_t)^\gamma \log(p_t),
\]
with $\gamma=3.0$, and the adversarial perturbation is a one-step FGM perturbation under an $\ell_2$ constraint,
\[
\mathbf r_{\mathrm{adv}}=\epsilon\;\frac{\nabla_{\mathbf x}\mathcal L(f(\mathbf x;\theta),y)}{\|\nabla_{\mathbf x}\mathcal L(f(\mathbf x;\theta),y)\|_2},
\]
using $\epsilon=0.1$ and a Bernoulli gate $k\sim\mathrm{Bernoulli}(p_{\rm rat})$ with $p_{\rm rat}=0.5$. The paper reports hyperparameters $N_{\rm epochs}=100$, batch size $16$, and learning rate $10^{-4}$ with Adam, and states that only half of the batches incur the extra adversarial branch on average [2509.02072].

The reported ablation numbers in ABEX-RAT are Macro-F1 $74.08\%$ for “w/o ABEX” (RAT on original data only), $86.68\%$ for “w/o RAT” (ABEX plus standard training), and $90.32\%$ for full ABEX-RAT on the OSHA dataset. The paper further reports minority-class recall improving from $84.70\%$ without RAT to $91.88\%$ in ABEX-RAT [2509.02072].

Within this NLP-oriented family, RAT is primarily an efficiency device. The stated motivation is that full adversarial training doubles or triples training cost, while Bernoulli-sampled adversarial updates reduce expected adversarial-sample generations by factor $p$ and preserve nearly all of the measured generalization gain when $p=0.5$ [2509.11191]. This is operationally very different from the multi-norm and Taylor-expansion RATs.

## 5. Adversarial transformations in semi-supervised learning

Another established use of the acronym appears in “Adversarial Transformations for Semi-Supervised Learning,” where RAT stands for a regularization framework based on adversarial transformations rather than random adversarial example selection [1911.06181]. The model output is $p_\theta(y\mid x)$, the data are split into labeled and unlabeled subsets, and the inner maximization seeks a worst-case label-preserving transformation
\[
\phi_{T\text{-adv}}(x;\theta)=\arg\max_{\|\phi\|\le \epsilon}
\mathrm{KL}\big[p_\theta(y\mid x)\,\|\,p_\theta(y\mid f_\phi(x))\big].
\]
The per-point regularizer is then
\[
L_{\rm RAT}(x;\theta)=\mathrm{KL}\big[p_\theta(y\mid x)\,\|\,p_\theta(y\mid f_{\phi_{T\text{-adv}}}(x))\big],
\]
and the semi-supervised objective combines supervised cross-entropy with an unlabeled consistency term and, in practice, an entropy penalty [1911.06181].

The admissible transformation family $\mathcal S(\epsilon)$ includes additive noise, affine transformations, thin-plate spline transformations with a $4\times4$ control grid, flow-field deformations, and color distortion. Composite transformations can be built by chaining these operations. The training procedure uses an $\epsilon$-rampup schedule based on the sigmoid ramp from Mean Teacher:
\[
\mathrm{ramp}(s)=\exp(-5(1-s)^2),\qquad \epsilon(t)=\epsilon_{\max}\cdot \mathrm{ramp}(t/T_{\rm ramp}),
\]
so that the perturbation region grows gradually during early training [1911.06181].

The reported experiments use a Wide ResNet 28$\times$2 backbone trained for $500$K iterations with Adam. On CIFAR-10 with $4{,}000$ labels, the paper reports test error $20.35\pm0.14\%$ for supervised only, $13.26\pm0.20\%$ for VAT with $\epsilon$-rampup, $11.70\pm0.32\%$ for RAT with noise + affine and no rampup, and $11.26\pm0.34\%$ for RAT with $\epsilon$-rampup. On SVHN with $1{,}000$ labels, the corresponding errors are $12.33\pm0.25\%$, $5.17\pm0.26\%$, $3.10\pm0.12\%$, and $2.86\pm0.07\%$. The paper also states that combining noise injection with affine transformation produced the largest gain over VAT in its CIFAR-10 ablation [1911.06181].

This line broadens the meaning of adversarial perturbation from additive norm-bounded noise to label-invariant transformations along the data manifold. It is therefore closer to adversarial consistency regularization than to classical robust optimization against norm-bounded attacks.

## 6. Theoretical context, recurring trade-offs, and limitations

A major theoretical backdrop for RAT-style methods is the difficulty of achieving robust generalization under adversarial training. “The curse of overparametrization in adversarial training: Precise analysis of robust generalization for random features regression” studies adversarially trained random-features regression under an $\ell_2$-bounded adversary in the proportional high-dimensional limit $n,d,N\to\infty$ with $N/d\to\psi_1$ and $n/d\to\psi_2$ [2201.05149]. The paper derives an asymptotically exact robust-risk formula through a five-dimensional saddle-point problem and reports that, unlike standard generalization, robust generalization can degrade as overparameterization increases. Its summarized insights include: standard risk exhibits double descent with respect to $\psi_1$, but for any $\epsilon>0$ the robust-risk curve loses the safe descent in the overparameterized region; the worst robust error can occur at $\psi_1>\psi_2$; and for strong adversaries or large $\psi_2$, the global minimum of robust risk can even occur in the underparameterized regime $\psi_1<1$ [2201.05149].

Although this paper is not itself a RAT proposal, it sharpens the context in which RAT methods are designed. Across the surveyed RAT variants, three recurring trade-offs are explicit. First is the robustness-versus-cost trade-off: batch-sampled RAT in BioIE reduces adversarial overhead by roughly the sampling probability $p$ while retaining nearly all measured gains at $p=0.5$ [2509.11191], and ABEX-RAT states that random adversarial training enhances robustness without significant overhead [2509.02072]. Second is the robustness-versus-clean-accuracy trade-off: the Taylor-expansion RAT is explicitly proposed to improve both robustness and clean accuracy by flattening the loss landscape [2303.10653]. Third is the threat-model trade-off: the multi-norm RAT paper argues that a defense tailored to one norm does not automatically transfer to another, motivating noise-injected adversarial training as a way to improve the worst-case profile across attacks [1903.10219].

Several limitations are also explicit in the source literature. RanAT4BIE notes that its randomness is currently at batch level, with no per-token or per-dimension sampling, and that a fixed $p$ may be suboptimal; it suggests dynamic or adaptive scheduling, stratified sampling of embedding subspaces or token positions, and combination with gradient accumulation or mixed precision as future directions [2509.11191]. The Taylor-expansion RAT incurs additional runtime and memory relative to TRADES [2303.10653]. In the semi-supervised RAT setting, effectiveness depends on the availability of transformation families that genuinely preserve class labels [1911.06181].

A common misconception is that “RAT” names one established benchmark method. The literature instead uses the acronym for multiple non-equivalent constructions. This suggests that any technical discussion of RAT should specify, at minimum, whether the randomization acts in input space, weight space, attack scheduling, or transformation space; whether the objective is robust optimization, consistency regularization, or efficiency-oriented approximation; and whether the target evaluation is worst-case adversarial accuracy, clean accuracy, micro-F1 or macro-F1, or semi-supervised test error.

Source: https://www.emergentmind.com/topics/random-adversarial-training-rat