---
title: Adaptive Swish (ASH) Activation
url: https://www.emergentmind.com/topics/adaptive-swish-ash
type: topic
---

# Adaptive Swish (ASH) Activation

Adaptive Swish (ASH) is an activation function designed to impose adaptive, data-dependent sparsity in neural network layers by modulating activations based on feature-map statistics such as mean and standard deviation. Originating as a generalization of the Swish activation, ASH introduces learnable, context-aware thresholds that differentiate it from traditional pointwise nonlinearities. Its formulation is motivated both by biological observations of variable neuronal thresholds and by the need to mitigate interference in continual learning scenarios through sparse representations. ASH and its variants have demonstrated effectiveness across a range of tasks, including class-incremental learning and high-dimensional visual classification, providing empirical improvements over established activation functions and regularization methods [2404.17651, 2210.11672].

## 1. Mathematical Formulation and Mechanism

ASH extends the Swish activation, which is $\mathrm{Swish}(x) = x \cdot \sigma(x)$, with an adaptive, per-layer (or per-channel) threshold. The core formula, as defined in [2210.11672], is:
\[
\mathrm{ASH}(x) = x \cdot \sigma\left(\alpha \cdot (x - \mu_X - z_k \sigma_X)\right)
\]
where $x$ is the pre-activation, $\sigma$ is the logistic sigmoid, $\mu_X$ and $\sigma_X$ are the mean and standard deviation of the activation vector (per layer or feature-map), $\alpha > 0$ controls the steepness or hardness of the gating, and $z_k$ sets the threshold as a Z-score (number of standard deviations above the mean). This construction ensures that only units with activations exceeding a dynamic, data-driven threshold contribute non-trivially to the layer's output.

Variants include a smooth gate (sigmoid or tanh-based) and a hard gate, as detailed later. The per-layer percentile-thresholding is justified by the approximate Gaussianity of pre-activations found in convolutional and feedforward layers, allowing the threshold $z_k$ to be interpreted as selecting the top-$k\%$ of activations [2210.11672].

## 2. Hard Adaptive Swish (Hard ASH) Variant

Hard ASH modifies the smooth gating to a discretized, nearly piecewise-constant regime. The gating function is replaced with a hard sigmoid, and the activations themselves are clipped within a maximum bound:
\[
\mathrm{Hard}\ \mathrm{ASH}(x) = \mathrm{clip}(x, 0, x_{\max}) \cdot \mathrm{HardSigmoid}(\alpha (x - \mu_X - z_k \sigma_X))
\]
where
\[
\mathrm{HardSigmoid}(u) = \frac{\mathrm{clip}(u + 3,\, 0,\, 6)}{6}
\]
and $\mathrm{clip}(x, a, b)$ restricts $x$ to $[a, b]$. This regime effectively divides neurons into three groups: those totally suppressed (output and gradient zero), those passing maximal output (again, gradient zero), and a narrow transition whose location and width are controlled by $\alpha$ and $z_k$. This further sparsifies both activations and gradients, sharply limiting interference during sequential task learning [2404.17651].

## 3. Context-Adaptive Thresholding and Trainability

The central feature of ASH is its context- and batch-adaptive gating. Unlike ReLU or Swish, which use fixed thresholds or fixed gating dynamics, ASH computes its thresholds on the fly per forward pass using mini-batch statistics. The sparsity control parameter $z_k$ (optionally trainable per layer or channel) sets the desired activation density, while $\alpha$ modulates gating steepness. The combination yields trainable, context-adaptive sparsification—activating only the most informative subset of units in response to current task or input statistics.

Empirical ablations show that trainable $z_k$ outperforms fixed percentile sparsification. A hard threshold gating function is intractable for gradient-based learning, so ASH employs smooth approximations (sigmoid or tanh), preserving differentiability for optimization [2210.11672].

## 4. Empirical Evaluation and Comparative Results

ASH has been tested in both vision and continual learning settings:

- **Vision benchmarks**: On ImageNet (ResNet-164, WRN-28-10, DenseNet-100-12 backbones), ASH attains top-1 accuracies of **78.6% ± 0.07** versus Swish's 77.5% ± 0.07 and ReLU's 76.4% ± 0.1. For CIFAR-10, ASH achieves **96.1% ± 0.05** (Swish: 95.6%, ReLU: 94.7%). Across other tasks (detection, segmentation, image generation), ASH consistently yields higher accuracy, faster convergence, and improved mean average precision or mIoU scores [2210.11672].

- **Continual learning (Split-MNIST)**: With a one-hidden-layer MLP (1,000 neurons, weight normalization), Hard ASH + Adagrad achieves **78.3% ± 1.4** after five sequential tasks (one epoch per task), notably outperforming ReLU + Adam (**49.2% ± 7.9**), Top-K subtraction, and Elastic Weight Consolidation (EWC, 500 epochs: 61%). ASH and Hard ASH exhibit resilience across optimizers, with SGD+Hard ASH (52.9%) far surpassing SGD+ReLU (19.8%). Increasing $z_k$ or $\alpha$ boosts retention for early tasks without degrading learning on later tasks [2404.17651].

A representative table of comparative Split-MNIST results:

| Activation + Optimizer  | Final Test Acc. (5 runs, %) | Training Regime             |
|------------------------|-----------------------------|-----------------------------|
| Hard ASH + Adagrad     | 78.3 ± 1.4                  | 5 epochs (1 per task)       |
| ASH + Adagrad          | 76.4 ± 1.4                  | 5 epochs (1 per task)       |
| Top-K Subtract + Adagrad | 76.0 ± 1.6                | 5 epochs (1 per task)       |
| ReLU + Adam            | 49.2 ± 7.9                  | 5 epochs (1 per task)       |
| EWC                    | 61                          | 500 epochs                  |
| SDMLP                  | 69                          | 500 epochs                  |
| SDMLP + EWC            | 83                          | 500 epochs                  |

## 5. Implementation Details and Computational Considerations

ASH requires computation of per-layer (or per-feature-map) mean and standard deviation, introducing minimal additional overhead relative to popular normalization layers. In standard vision models, ASH is "plug-and-play," replacing ReLU or Swish with no other architectural changes. For MLP continual learning, the canonical setup uses Kaiming-normalized weights and weight normalization on the first layer, with Adagrad as the primary optimizer (learning rate ≈ $2 \times 10^{-4}$, initial accumulator $1 \times 10^{-6}$) [2404.17651, 2210.11672].

ASH achieves high sparsity (typically 97–99% of units suppressed per layer) in $O(n)$ per-layer time, sidestepping explicit Top-K sorting ($O(n \log n)$). Backpropagation through the adaptive gating is handled via standard autodiff, with explicit formulas for gradient flow with respect to both $x$ and $z_k$. Sample pseudo-code for forward/backward computation is provided in [2210.11672].

## 6. Theoretical Properties and Intuitions

ASH leverages the approximate Gaussianity of deep-layer pre-activations to justify its adaptive Z-thresholding (Proposition 1, 2 in [2210.11672]). The gating function is infinitely differentiable (for finite $\alpha$), strictly increasing for positive $x$, and bounded between $0$ and $x$. Hard ASH introduces larger regions of zero gradient, promoting stability by constraining most unit updates to the narrow transition band around the adaptive threshold.

In continual learning, ASH and Hard ASH directly mitigate the stability–plasticity dilemma. High sparsity reduces overlap between units active during different tasks, thus lowering interference (stability), while active units for new tasks receive full gradient signals (plasticity). This focused gradient flow prevents widespread drift of parameters tied to inactive units, curbing catastrophic forgetting.

## 7. Practical Benefits, Limitations, and Open Issues

Empirical studies highlight several advantages of ASH:

- Context- and batch-aware gating adapts to input statistics per forward pass.
- Trainable thresholding enables each neuron or channel to regulate its own effective sparsity.
- Rapid convergence and competitive accuracy on diverse benchmarks, including classification, detection, and segmentation.

Documented limitations include slight computational and memory overhead for per-map statistics, absence of formal generalization bounds for arbitrary $a, b$ in the generalized Swish form, and untested application to NLP/transformer models. The dynamics of $z_k$ evolution across deeper architectures remain an open research avenue. A plausible implication is that further theoretical analysis of adaptive thresholding could yield insights amenable to broader classes of neural architectures [2210.11672].

## References

- "Hard ASH: Sparsity and the right optimizer make a continual learner" [2404.17651]
- "Stochastic Adaptive Activation Function" [2210.11672]

Source: https://www.emergentmind.com/topics/adaptive-swish-ash