---
title: Parametric Swish Activation
url: https://www.emergentmind.com/topics/parametric-swish-activation
type: topic
---

# Parametric Swish Activation

Parametric Swish activation functions constitute a class of non-linearities in neural networks designed to enhance flexibility, gradient flow, and representation power compared to fixed functions such as ReLU or the vanilla Swish. These functions introduce explicit, trainable or hyperparameterized degrees of freedom—most commonly via slope/scaling, bias shift, input-dependent thresholding, or context adaptation—that allow each layer or unit to interpolate between linearity, non-monotonic gating, and sharp rectification. The family includes several variants used across vision, NLP, and generative models, with substantial empirical evidence showing reliable, often state-of-the-art, performance improvements in large-scale benchmarks.

## 1. Mathematical Foundations and Canonical Parametric Forms

The canonical Swish activation is defined as
$$
\mathrm{Swish}(x; \beta) = x \cdot \sigma(\beta x)
$$
with $\sigma(z) = 1/(1 + e^{-z})$ as the logistic sigmoid, and $\beta \geq 0$ a shape parameter. The non-parametric version sets $\beta=1$ ("SiLU"), while the parametric (trainable or hyperparameter) variant allows $\beta$ to vary globally, per-layer, or per-channel. As $\beta \to \infty$, $\mathrm{Swish}$ approaches $\mathrm{ReLU}$; $\beta \to 0$ yields a scaled linear identity. The derivative has a closed form:
$$
\frac{d}{dx}\mathrm{Swish}(x; \beta) = \sigma(\beta x) + \beta x \sigma(\beta x)(1 - \sigma(\beta x))
$$
yielding smooth, non-monotonic, and gradient-preserving transitions across $x$ [1710.05941].

The E-swish extension introduces an output slope parameter:
$$
\mathrm{E\text{-}swish}_\beta(x) = \beta x \cdot \sigma(x)
$$
where $\beta > 0$ amplifies (or contracts) the overall gain. This scaling can be interpreted as a "depth knob," controlling gradient magnitude and activation nonlinearity based on architecture depth [1801.07145].

Further parametric forms generalize Swish to:
- Learnable vertical shifts: $x \cdot \sigma(\beta x) - \gamma$, with $\gamma \in \mathbb{R}$ providing re-centering and explicit bias (as in SG-Blend's $\mathrm{SSwish}_{\beta, \gamma}$) [2505.23942].
- Context-adaptive gates: $x \cdot \sigma(a x + b)$, with $a$ and $b$ modulating the sharpness and the position of the gating boundary (as in Adaptive SwisH, ASH) [2210.11672].
- Piecewise or hybrid regimes where Swish is augmented via additive biases or composed with Tanh, as in the Swish-T family [2407.01012].

## 2. Parametric Swish Variants and Theoretical Properties

Table: Representative Parametric Swish Activations

| Name             | Formula                                         | Principal Parameter(s)         |
|------------------|-------------------------------------------------|-------------------------------|
| Swish            | $x \cdot \sigma(\beta x)$                       | $\beta$ (trainable/fixed)     |
| E-swish          | $\beta x \cdot \sigma(x)$                       | $\beta$ (hyperparameter)      |
| SSwish           | $x \cdot \sigma(\beta x) - \gamma$              | $\beta,\,\gamma$ (learnable)  |
| Swish-T$_C$      | $\sigma(\beta x)\big(x + \frac{2\alpha}{\beta}\big) - \frac{\alpha}{\beta}$| $\beta$ (learned), $\alpha$ (fixed) |
| ASH              | $x \cdot \sigma(a x + b)$, $a,b$ adapt to stats | $a,\,b$ (contextual, learned) |
| PFTS             | $x\cdot\sigma(x)+t$ for $x\ge0$, else $t$       | $t$ (learnable, piecewise)    |

All variants retain the essential properties of Swish: $C^\infty$ smoothness, non-monotonicity (negative "dip" near $x=0$), gradient non-sparsity, and unbounded positive support. Additional degrees of freedom enable:
- Tunable sharpness and symmetry ($\beta$, $\gamma$)
- Controlled mean activation (bias shift, via $t$ or $\gamma$)
- Adaptive sparsity and top-$k$ filtering (ASH)
- Context-aware or learnable gating regions (via $a$, $b$)
- Flexible negative-region behavior (via Tanh bias in Swish-T or flat constant in PFTS)

## 3. Parametric Swish in Empirical Benchmarking

Parametric Swish activations have demonstrated consistent increases in accuracy, faster convergence, and improved loss landscape properties over both ReLU and fixed Swish, especially in convolutional and transformer architectures. Key findings include:

- **CIFAR-10/100, ImageNet:** Swish with trainable or appropriately chosen $\beta$ exceeds ReLU by $+0.7$–$+1.4$\% top-1 on ResNet, MobileNet, and Inception/ResNet models [1710.05941].
- **E-swish:** Largest accuracy boosts on Wide ResNet are obtained for $\beta = 1.25$–$1.75$, with larger $\beta$ needed for shallow nets, and smaller $\beta$ for deeper or batch-normalized networks [1801.07145].
- **SSwish/SG-Blend:** Learning both $\beta$ and $\gamma$ outperforms both Swish and GELU, with SG-Blend (blending SSwish and GELU using a learned $\alpha$) achieving state-of-the-art on CIFAR-10, BERT, and WMT14 BLEU [2505.23942].
- **ASH:** Adaptive, context-dependent thresholding provides 0.5–1\% higher accuracy than Swish or GELU on vision and detection benchmarks, with accelerated convergence [2210.11672].
- **Swish-T$_C$:** Non-parametric variant matches or marginally surpasses learned $\beta$ Swish-T, indicating that in some architectures a fixed, high $\beta$ (e.g., $\beta = 6$) suffices for optimal performance [2407.01012].
- **PFTS:** On SVHN, performance gains of up to $+71.8$\% over ReLU in deep fully-connected architectures, and highest mean rank among parametric activations [2011.03155].
- **Zorro family:** The Sloped-Zorro variant ($m$ tuned to $0.7$) numerically approximates Swish but further improves accuracy on vision tasks (e.g., $+5.3$\% on CIFAR-10 over classic Swish) and enables explicit central slope control to mitigate vanishing gradients [2409.19239].

## 4. Parameterization, Learning Schemes, and Regularization

Different variants instantiate parameter learning distinctly:

- **Fixed vs. learned $\beta$:** Swish can use a global fixed $\beta$, but per-channel (or per-layer) learnable $\beta$ consistently improves accuracy and adaptation [1710.05941].
- **Symmetry and centering ($\gamma$, Tanh bias):** Adding a learnable vertical offset (as in SSwish) or a Tanh bias (Swish-T) enables the activation's mean to be controlled, aiding convergence and negative region expressivity [2407.01012, 2505.23942].
- **Adaptive gating (ASH):** Per-feature/mini-batch statistics ($\mu_X$, $\sigma_X$) and a learned percentile gate ($z_k$) are updated alongside network weights, providing context-sensitive activation sparsity or density [2210.11672].
- **Piecewise regimes (PFTS):** Trainable threshold $t$ is optimized via standard back-prop, with careful initialization ($t_0 \approx -0.2$) leading to stable mean-zero activations and reduced bias shift [2011.03155].

All parameters may be optimized using standard SGD/Adam, with optional clamping or softplus constraints for stability, and regularizers (e.g., weight decay, penalties on $\|\alpha-0.5\|$) as needed to avoid degenerate solutions [2505.23942].

## 5. Theoretical Insights and Functional Flexibility

The introduction of parametric degrees of freedom serves to:
- Smoothly interpolate between identity, ReLU, and saturating or non-monotonic regimes.
- Adapt nonlinearity and gradient scale to network depth (avoiding vanishing or exploding gradients as in E-swish and Zorro).
- Restore information flow or gradient signals in negative activation regimes (Swish-T, PFTS, SSwish).
- Implement biologically inspired or sampling-based gating (ASH), linking to the behavior variability of real neuron populations.

These properties facilitate globally smoother loss landscapes, fewer plateaus, data- and architecture-adaptive nonlinearity, and—in deep or wide networks—greater resilience against training instabilities [1710.05941, 1801.07145, 2210.11672, 2409.19239].

## 6. Comparative Experimental Evaluation and Selection Guidelines

Empirical work recommends the following:

- **Swish/E-swish:** For shallow to moderate-depth architectures, $\beta \approx 1.25$–$1.5$; for deep (>30 layers), $\beta \approx 1.0$–$1.25$ [1801.07145].
- **SG-Blend/SSwish:** Use as a drop-in replacement for GELU or Swish in both vision and NLP, initializing $\alpha=0.5$, $\beta=1.0$, and $\gamma=0$ [2505.23942].
- **Swish-T$_A$ (non-parametric):** Suitable in low-resource or mobile inference settings.
- **ASH:** Trainable thresholding robustly outperforms fixed (non-learned) variants, and α (steepness) in [5, 20] works reliably [2210.11672].
- **Zorro:** Start with $m=0.7$, $a_1=1.3$, $a_2=0$, $b=1.8$ for Swish-like behavior, tuning $m$ upward to accelerate convergence if necessary [2409.19239].

Table: Empirical Accuracy Improvements (Sample Results)

| Task/Arch         | Baseline | Parametric Swish Variant       | Accuracy Δ (%)    | Source     |
|-------------------|----------|-------------------------------|-------------------|------------|
| CIFAR-10 WRN-10-2 | ReLU     | E-swish (β=1.5)               | +1.5              | [1801.07145]|
| CIFAR-10 ResNet-18| Swish    | SG-Blend                      | +0.36             | [2505.23942]|
| SVHN DNN-7        | ReLU     | PFTS                          | +71.83            | [2011.03155]|
| CIFAR-10 ConvNet  | Swish    | Zorro-Swish                   | +5.3              | [2409.19239]|

## 7. Implementation and Practical Integration

Integration is straightforward in modern DL frameworks, with minimal code modifications:

- **Swish/E-swish:** One-liner replacements are possible. For E-swish: `activation(x)=β*x*sigmoid(x)` or `activation(x)=β*tf.nn.swish(x)` in TensorFlow [1801.07145].
- **SG-Blend/SSwish:** Minimal PyTorch modules, registering parameters as `nn.Parameter`s and updating with network weights, clamping as needed [2505.23942].
- **ASH, PFTS, Swish-T:** Custom modules/layers, with in-line calculation of activation/statistics and simple per-layer parameter bookkeeping [2210.11672, 2011.03155, 2407.01012].
- **Zorro:** Explicit parameterization of the central slope/bounds allows both fixed and learnable regimes, adapting to dataset and performance requirements [2409.19239].

Standard optimization hyperparameters suffice; batch normalization and weight initialization practices remain unchanged except as noted for Swish (retaining batchnorm scale) [1710.05941].

---

Parametric Swish activation functions, by introducing trainable or context-adaptive parameters, provide an effective mechanism for learning data- and architecture-specific nonlinearities. Empirical and theoretical advantages include smoother optimization, improved accuracy and convergence, and increased functional flexibility, establishing these variants as robust, practical alternatives to both ReLU and fixed Swish/GELU baselines across a spectrum of deep learning applications [1710.05941, 1801.07145, 2210.11672, 2407.01012, 2011.03155, 2409.19239, 2505.23942].

Source: https://www.emergentmind.com/topics/parametric-swish-activation