Learning to Perturb Activations (LPA)

• Learning to Perturb Activations (LPA) is a method that leverages learned, class-aware perturbations of intermediate features to improve model robustness.
• It systematically employs both additive and multiplicative perturbations, linking activation noise to flat minima and boundary densification.
• Empirical results demonstrate that LPA enhances performance in balanced, long-tail, and domain generalization settings, outperforming standard regularizers.

Learning to Perturb Activations (LPA) denotes a family of methods that act directly on intermediate hidden representations rather than restricting perturbation to inputs, labels, or final logits. In the literature summarized here, the term has both a broad and a narrow sense. Broadly, it names the idea of learning, parameterizing, or systematically studying perturbations of layer activations; narrowly, it refers to the class-aware PGD-based method introduced under the name “Learning to Perturb Activations” in 2026. Across these formulations, perturbations may be additive or multiplicative, stochastic or deterministic, sample-level or class-level, and may be optimized for few-shot generalization, adversarial robustness, anomaly detection, universal transfer attacks, or inference-time probing (Li, 28 May 2026, Lee et al., 2019).

1. Conceptual scope and taxonomy

A general hidden-perturbation view writes a network representation at layer $l$ as $a_i^{(l)}$, then replaces it by

$$\tilde{a}_i^{(l)} = a_i^{(l)} + \delta_i^{(l)}, \qquad \|\delta_i^{(l)}\|_p \leq \epsilon.$$

Within this template, Dropout, DropBlock, Shake-Shake, ShakeDrop, Manifold Mixup, and adversarial feature perturbation can all be interpreted as specific activation perturbations, but typically with random or class-agnostic rules rather than a learned class-aware perturbation policy (Li, 28 May 2026).

This viewpoint also clarifies the relation between activation, logit, and input perturbation. Logit perturbation is the deepest special case: if logits satisfy $z = Wh + b$, then perturbing logits by $\tilde z = z + \delta$ is equivalent to perturbing the final activation by some $\Delta h$ such that $W\Delta h = \delta$ (Li et al., 2022). Input perturbation is a constrained special case from the opposite direction: small input changes induce layer-$l$ activation changes $$\Delta_\ell(x,\varepsilon) \approx J_{a_\ell}(x)\varepsilon$$, so the reachable perturbations lie in the image of a Jacobian rather than the full activation space (Ren et al., 3 Feb 2026).

Method	Perturbation space and form	Learned component
Meta-dropout	Hidden activations; multiplicative, input-dependent noise	Noise generator $\phi$ (Lee et al., 2019)
Learn2Perturb	Hidden activations; additive Gaussian noise	Layer-wise noise scales $a_i^{(l)}$0 (Jeddi et al., 2020)
LPA (2026)	Selected hidden layer; class-level additive PGD perturbation	Class perturbation vectors $a_i^{(l)}$1 (Li, 28 May 2026)
LPL	Logits; class-level additive perturbation	Class logit perturbations $a_i^{(l)}$2 (Li et al., 2022)
PNN	Hidden activations; fixed additive masks	Linear combination weights $a_i^{(l)}$3 (Juefei-Xu et al., 2018)
APEX	Hidden activations; additive Gaussian inference-time noise	No learned perturbation (Ren et al., 3 Feb 2026)

2. From perturbative layers to meta-learned feature noise

One early activation-centric formulation is the perturbation layer of Perturbative Neural Networks. Instead of a convolutional kernel, the layer forms multiple perturbed activation maps $a_i^{(l)}$4 and outputs

$a_i^{(l)}$5

Here the perturbation masks are fixed random tensors, while learning occurs through the scalar mixing weights $a_i^{(l)}$6. Once the masks are sampled, the layer is deterministic; perturbation is architectural rather than a training-only regularizer. Empirically, Perturbative Neural Networks perform comparably with standard CNNs on MNIST, CIFAR-10, PASCAL VOC, and ImageNet with fewer parameters, which suggests that learned combinations of perturbed activations can substitute for explicit spatial filters in some regimes (Juefei-Xu et al., 2018).

A more explicit learning-to-perturb formulation appears in meta-dropout. In few-shot meta-learning, meta-dropout learns a noise generator that outputs a multiplicative noise distribution for latent features, conditioned on the current example and layer. For a layer $a_i^{(l)}$7, noise is produced as

$a_i^{(l)}$8

and the perturbed activation becomes

$a_i^{(l)}$9

Support examples in the inner loop are perturbed stochastically, while the query objective in the outer loop uses deterministic mean perturbations; the meta-objective differentiates through the inner-loop adaptation and requires second-order derivatives, in a standard MAML-style bilevel computation (Lee et al., 2019).

This meta-learned perturbation policy is directly optimized for generalization across tasks. On Omniglot 20-way classification, reproduced MAML improves from $$\tilde{a}_i^{(l)} = a_i^{(l)} + \delta_i^{(l)}, \qquad \|\delta_i^{(l)}\|_p \leq \epsilon.$$0 to $$\tilde{a}_i^{(l)} = a_i^{(l)} + \delta_i^{(l)}, \qquad \|\delta_i^{(l)}\|_p \leq \epsilon.$$1 in 1-shot and from $$\tilde{a}_i^{(l)} = a_i^{(l)} + \delta_i^{(l)}, \qquad \|\delta_i^{(l)}\|_p \leq \epsilon.$$2 to $$\tilde{a}_i^{(l)} = a_i^{(l)} + \delta_i^{(l)}, \qquad \|\delta_i^{(l)}\|_p \leq \epsilon.$$3 in 5-shot when meta-dropout is added. On miniImageNet 5-way classification, MAML improves from $$\tilde{a}_i^{(l)} = a_i^{(l)} + \delta_i^{(l)}, \qquad \|\delta_i^{(l)}\|_p \leq \epsilon.$$4 to $$\tilde{a}_i^{(l)} = a_i^{(l)} + \delta_i^{(l)}, \qquad \|\delta_i^{(l)}\|_p \leq \epsilon.$$5 in 1-shot and from $$\tilde{a}_i^{(l)} = a_i^{(l)} + \delta_i^{(l)}, \qquad \|\delta_i^{(l)}\|_p \leq \epsilon.$$6 to $$\tilde{a}_i^{(l)} = a_i^{(l)} + \delta_i^{(l)}, \qquad \|\delta_i^{(l)}\|_p \leq \epsilon.$$7 in 5-shot. In the reported comparisons, meta-dropout also exceeds mixup, manifold mixup, VIB, and Information Dropout, and its visualizations place perturbed support examples near decision boundaries, consistent with the interpretation that the learned perturbations densify decision-critical regions of feature space (Lee et al., 2019).

3. Explicit class-aware Learning to Perturb Activations

The 2026 method named LPA makes the class structure of activation perturbation explicit. It selects a hidden layer $$\tilde{a}_i^{(l)} = a_i^{(l)} + \delta_i^{(l)}, \qquad \|\delta_i^{(l)}\|_p \leq \epsilon.$$8, assigns a class-level perturbation vector $$\tilde{a}_i^{(l)} = a_i^{(l)} + \delta_i^{(l)}, \qquad \|\delta_i^{(l)}\|_p \leq \epsilon.$$9 to each class, and partitions classes into $z = Wh + b$0 for positive augmentation and $z = Wh + b$1 for negative augmentation. The training objective is

$z = Wh + b$2

This yields the paper’s distinction between expansive perturbation, which increases activation norm and is conjectured to act as positive augmentation, and contractive perturbation, which decreases activation norm and is conjectured to act as negative augmentation (Li, 28 May 2026).

The perturbations are optimized by PGD on class-averaged gradients. For positive augmentation,

$z = Wh + b$3

and for negative augmentation the sign is reversed, followed by projection onto an $z = Wh + b$4-ball. Perturbation radii are class- and layer-dependent: $z = Wh + b$5 In practice the reported setup uses $z = Wh + b$6 constraints, $z = Wh + b$7 PGD steps, step size $z = Wh + b$8, and a low-rank variant that retains the top $z = Wh + b$9 directions (Li, 28 May 2026).

The method is supported by two propositions and one reduction result. First, activation perturbation is linked to flat minima by showing equivalence to a structured perturbation of the parameters at the perturbed layer. Second, perturbation amplification across layers motivates smaller radii for shallower layers. Third, when the perturbed layer is the logit layer, LPA reduces to Learning to Perturb Logits (LPL), while for $\tilde z = z + \delta$0 it strictly enlarges the attainable logit perturbation set because $\tilde z = z + \delta$1 is typically much larger than the number of classes (Li, 28 May 2026).

Empirically, the reported gains are consistent across balanced classification, long-tail learning, and domain generalization. On CIFAR-100 with ResNet-110, Top-1 error drops from 24.56 for CE, 23.85 for Dropout, 23.28 for Manifold Mixup, and 22.65 for LPL to 21.92 for LPA. On CIFAR-10 with WRN-28-10, error drops from 3.89 for CE and 3.28 for LPL to 3.06 for LPA. On CIFAR-100-LT with imbalance $\tilde z = z + \delta$2, LPA reaches about $\tilde z = z + \delta$3 overall accuracy and is reported to exceed LPL by 5.4 percentage points and LPG by 2.2 points, with especially large gains on tail classes. In DomainBed, LPA improves over SWAD on PACS, VLCS, OfficeHome, and TerraIncognita, with reported numbers $\tilde z = z + \delta$4 vs $\tilde z = z + \delta$5, $\tilde z = z + \delta$6 vs $\tilde z = z + \delta$7, $\tilde z = z + \delta$8 vs $\tilde z = z + \delta$9, and $\Delta h$0 vs $\Delta h$1, respectively (Li, 28 May 2026).

4. Robustness-oriented activation perturbation and logit-space special cases

Learn2Perturb instantiates LPA as additive Gaussian feature noise with learnable scale. At layer $\Delta h$2,

$\Delta h$3

so the perturbed activation is $\Delta h$4 with $\Delta h$5. In the ResNet experiments, perturbation-injection modules are added to the output of every block and before the ReLU activation. Training alternates between updating network weights $\Delta h$6 and noise parameters $\Delta h$7, with a regularizer $\Delta h$8 and a lower bound $\Delta h$9 to prevent collapse of the noise scales to zero. The reported training recipe uses a 20-epoch warmup without noise, then alternating back-propagation together with PGD adversarial training and a balanced loss with $W\Delta h = \delta$0 (Jeddi et al., 2020).

A distinctive feature of Learn2Perturb is that perturbation remains active at inference. The resulting stochastic mapping is meant to harden the network against both input attacks and internal uncertainty. On CIFAR-10 with ResNet-V1(20), vanilla PGD training yields clean accuracy $W\Delta h = \delta$1, PGD robustness $W\Delta h = \delta$2, and FGSM robustness $W\Delta h = \delta$3, whereas Learn2Perturb yields $W\Delta h = \delta$4, $W\Delta h = \delta$5, and $W\Delta h = \delta$6, respectively. The reported comparisons state that Learn2Perturb exceeds PNI by $W\Delta h = \delta$7 on FGSM and PGD, is best across all confidence levels on $W\Delta h = \delta$8 C&W, retains $W\Delta h = \delta$9 robustness under EOT PGD, and passes Athalye’s obfuscated-gradient checks (Jeddi et al., 2020).

LPL occupies the boundary between activation and output-space perturbation. It perturbs class logits by $l$0, with $l$1 obtained by maximizing loss for positive augmentation or minimizing loss for negative augmentation. In single-label classification the update is proportional to the class-average of $l$2; in multi-label classification it becomes a scalar shift per label dimension. Because LPL perturbs only logits, it is a plug-in method with no additional parameters at inference. Reported results include a reduction on CIFAR-100 ResNet-110 from $l$3 error for ISDA+Dropout to $l$4 for LPL, and in long-tail learning a complementary effect with logit adjustment, such as CIFAR100-LT100 improving from $l$5 for LA and $l$6 for LPL to $l$7 for LA+LPL (Li et al., 2022).

5. Anomaly detection, universal attacks, and inference-time probing

PLAD transfers the learning-to-perturb idea to one-class anomaly detection. A perturbator $l$8 outputs multiplicative and additive factors $l$9, producing

$$\Delta_\ell(x,\varepsilon) \approx J_{a_\ell}(x)\varepsilon$$0

while a classifier $$\Delta_\ell(x,\varepsilon) \approx J_{a_\ell}(x)\varepsilon$$1 separates original from perturbed points. The perturbator is regularized toward the identity transform through

$$\Delta_\ell(x,\varepsilon) \approx J_{a_\ell}(x)\varepsilon$$2

Implemented in input space, PLAD is nevertheless formulated in a way that can be transplanted almost directly to activation space by replacing $$\Delta_\ell(x,\varepsilon) \approx J_{a_\ell}(x)\varepsilon$$3 with a hidden representation. Reported performance includes best average AUC on CIFAR-10 at $$\Delta_\ell(x,\varepsilon) \approx J_{a_\ell}(x)\varepsilon$$4, best average AUC on Fashion-MNIST at $$\Delta_\ell(x,\varepsilon) \approx J_{a_\ell}(x)\varepsilon$$5, and Arrhythmia F1 of $$\Delta_\ell(x,\varepsilon) \approx J_{a_\ell}(x)\varepsilon$$6, where it substantially exceeds the best cited baseline at about $$\Delta_\ell(x,\varepsilon) \approx J_{a_\ell}(x)\varepsilon$$7 (Cai et al., 2022).

Pre-trained Adversarial Perturbations and the Low-Level Layer Lifting Attack pursue a different goal: they learn a universal input perturbation whose defining objective is activation-space. The base loss is

$$\Delta_\ell(x,\varepsilon) \approx J_{a_\ell}(x)\varepsilon$$8

so optimization maximizes the Frobenius norm of feature maps at a selected low-level layer. The method also studies a fused objective over layers $$\Delta_\ell(x,\varepsilon) \approx J_{a_\ell}(x)\varepsilon$$9 and $\phi$0, and Uniform Gaussian Sampling, which augments the activation-lifting objective with Gaussian noise inputs having uniformly sampled mean and variance. The central empirical observation is that low-level layers change less under fine-tuning and therefore yield better cross-task transferability. On ResNet101, the best baseline is around $\phi$1 ASR, whereas L4A$\phi$2 reaches $\phi$3, L4A$\phi$4 reaches $\phi$5, and L4A$\phi$6 reaches $\phi$7. On ViT-B/16, L4A$\phi$8 reaches $\phi$9 (Ban et al., 2022).

APEX shifts the emphasis from training to probing. It perturbs post-activation representations at inference by

$a_i^{(l)}$00

then estimates the prediction distribution $a_i^{(l)}$01 through Monte Carlo sampling. Its key theoretical decomposition states that $a_i^{(l)}$02, where $a_i^{(l)}$03 is independent of $a_i^{(l)}$04 and $a_i^{(l)}$05 is uniformly bounded, implying a transition from sample-dependent behavior at small $a_i^{(l)}$06 to model-dependent behavior at large $a_i^{(l)}$07. In the small-noise regime, APEX defines escape noise as the smallest $a_i^{(l)}$08 for which the original top-1 prediction probability falls below $a_i^{(l)}$09; this quantity aligns with memorization score and C-score. In the large-noise regime, APEX reveals stationary output distributions and exposes model-level biases, including pronounced concentration on the target class in backdoored models (Ren et al., 3 Feb 2026).

6. Recurring themes, misconceptions, and open directions

Several themes recur across these works. First, layer choice is not incidental. The 2026 LPA paper argues that shallow perturbation behaves more like input-level augmentation, while deeper perturbation behaves more like logit manipulation; APEX proves that input perturbation is a constrained activation perturbation; PAPs shows that low-level perturbation is best for pre-training-to-fine-tuning universality; and meta-dropout and Learn2Perturb both obtain their reported benefits by perturbing hidden features inside the learning dynamics rather than only at the input or label layer (Li, 28 May 2026, Ren et al., 3 Feb 2026, Ban et al., 2022, Lee et al., 2019, Jeddi et al., 2020).

Second, LPA is not a single algorithmic recipe. A common misconception is to equate activation perturbation with dropout-like random masking during training. The surveyed methods contradict that simplification. PNN uses fixed perturbation masks and learns only how to combine them; meta-dropout learns an input-dependent multiplicative distribution over latent features; Learn2Perturb learns additive Gaussian scales and keeps perturbation active at test time; LPL perturbs only the logits; and APEX uses non-learned inference-time noise solely for probing. This suggests that LPA is best understood as a design space defined by perturbation parameterization, optimization target, and deployment phase rather than by any single stochastic mechanism (Juefei-Xu et al., 2018, Lee et al., 2019, Jeddi et al., 2020, Ren et al., 3 Feb 2026).

Third, the reported limitations are structurally similar even when objectives differ. Meta-dropout incurs computational overhead from evaluating noise networks and differentiating through second-order inner loops; Learn2Perturb requires careful control of warmup, annealing, and lower-bounded noise scales; explicit LPA adds PGD steps and layer-selection heuristics; PLAD may become heavy in full-dimensional input space; and APEX does not learn the perturbation operator at all. The cited works explicitly propose richer noise families, explicit task embeddings, extension to sequence models and non-few-shot settings, multi-layer perturbators, domain generalization and continual-learning variants, and learned structured probe operators as natural next steps. A plausible implication is that future LPA research will continue to move from random or hand-designed perturbations toward objective-aware, geometry-aware, and architecture-aware perturbation policies (Lee et al., 2019, Li, 28 May 2026, Cai et al., 2022, Ren et al., 3 Feb 2026).