---
title: 'Swoosh Activations: Theory & Applications'
url: https://www.emergentmind.com/topics/swoosh-activations
type: topic
---

# Swoosh Activations: Theory & Applications

The term "Swoosh activations" encompasses a family of activation functions characterized by a smooth, non-monotonic curve that resembles the "swish" or "swoosh" shape. These functions include canonical Swish ($x\cdot\sigma(\beta x)$), its parametric and thresholded variants such as Flatten-T Swish, and, more recently, the Swoosh Activation Function (SAF) designed for regularization in heatmap-based landmark detection. Their mathematical form typically combines a sigmoid gating with linear or modified branches, distinct from piecewise and monotonically-gated rectifiers. Swoosh-type activations are designed to boost expressivity, gradient flow, and trainability, especially in deep architectures or tasks with high-frequency and heterogeneous target structures.

## 1. Mathematical Definitions and Key Properties

**Canonical Swish** is defined as:
$$
\mathrm{Swish}_\beta(x) = x \cdot \sigma(\beta x) \quad \text{where} \quad \sigma(z) = \frac{1}{1+e^{-z}},
$$
with $\beta>0$ controlling the steepness of the gating. The unparameterized form ($\beta=1$) is common in practice [2110.07721], [1901.02671], [2010.09458], [1805.08266].

Key properties:
- *Smoothness*: $C^\infty$; derivatives exist everywhere, unlike ReLU.
- *Non-monotonicity*: For $x < 0$, the curve dips below zero, facilitating richer feature learning.
- *Non-zero derivatives*: $\frac{d}{dx}\mathrm{Swish}(x) = \sigma(x) + x \sigma(x)(1-\sigma(x))$, ensuring gradient propagation across all regions.
- *Limiting cases*: As $\beta \to \infty$, Swish approximates ReLU; as $\beta \to 0$, it becomes the identity.
- *Range*: For $\beta=1$, minimum is $\approx -0.2785$; unbounded above.

**Flatten-T Swish (FTS)** introduces a threshold:
$$
f_T(x) =
\begin{cases}
x \cdot \sigma(x) + T & x \geq 0 \\
T & x < 0
\end{cases}
$$
$T$ sets a negative floor; $T=-0.20$ performs best for MNIST deep FNNs, allowing negative value propagation while retaining sparse gradients on $x<0$ [1812.06247].

**Swoosh Activation Function (SAF)** is a regularization penalty function for optimizing heatmap MSEs in landmark detection:
$$
\mathrm{SAF}(x) = (a x + \frac{1}{b x})^c - \mathrm{Min}
$$
Optimal MSE $x^* = \sqrt{1/(a b)}$ aligns SAF's minimum with the desired regularization target, and $a$, $b$, $c$ tune its profile. SAF is not an activation in the standard forward-pass sense but enforces distributional sharpness/dispersion for outputs in heatmap regression [2412.11377].

## 2. Comparative Role in Deep Networks

Swoosh-family activations remedy limitations of classical nonlinearities:

| Activation         | Smoothness      | Gradient at $x<0$  | Monotonicity      | Negative Signal |
|--------------------|-----------------|---------------------|-------------------|-----------------|
| ReLU               | Not smooth      | Zero                | Yes               | None            |
| Tanh/Sigmoid       | Smooth          | Small (Saturating)  | Yes               | Yes (but saturating) |
| Swish/Swoosh       | Smooth          | Small, nonzero      | No                | Yes (non-monotonic) |
| Flatten-T Swish    | Piecewise smooth| Zero (x<0), $T<0$   | Partly            | Yes (via $T$)   |
| SAF (Loss reg.)    | N/A             | N/A                 | N/A               | N/A             |

Swish-type activations preserve gradient flow—especially for negative pre-activations—avoiding dead units ("dying ReLU"). Their non-monotonic bumps around zero enable networks to approximate functions with mixed-frequency content and sharp transitions more easily [2110.07721], [1901.02671], [1812.06247].

## 3. Empirical Performance in Key Domains

### Physics-Informed Neural Networks (PINNs), Helmholtz Equation

- Al-Safwan et al. [2110.07721] demonstrated that Swish activation markedly improves PINN convergence for 2D Helmholtz problems, particularly in models with sharp heterogeneities.
  - In an 8-layer, 20-neuron PINN, Swish accelerated initial loss decay by ~30% compared to tanh/ELU and reduced final $L_2$ error by ~15%.
  - Real and imaginary wavefield errors ("two-scatter test") in $L_2$ and $L_\infty$ were $10-20\%$ lower for Swish.
- Swish’s ability to represent high-wavenumber components—by leveraging non-monotonicity—overcomes the spectral bias toward low frequencies seen with tanh and ReLU.

### Natural Language Processing

- In "Is it Time to Swish?" [1901.02671], Swish achieved strong results across 8 NLP tasks (sentence/document classification, sequence tagging):
  - Best-case Swish was top-5 among 21 activations; on certain sequence tasks, Swish exceeded ReLU by 1–3 percentage points.
  - In deeply stacked networks, Swish outperformed saturated functions (cube, cosid) and matched the performance of penalized-tanh and Maxout variants.
  - Training stability and convergence were on par with ReLU; Swish did not exhibit dead neurons.

### Image Classification and Shallow CNNs

- On CIFAR-10 (2-conv CNN, 25 epochs), Swish (β=1) yielded 69.9% accuracy versus 71.8% for ReLU and 72.9% for Leaky ReLU [2010.09458], indicating that Swish excels primarily in deeper architectures.
- Flatten-T Swish (T=-0.20) improved MNIST accuracy by up to 1.15% and halved the epochs needed for convergence in 512–1568 neuron deep FNNs [1812.06247].

### Landmark Detection via Heatmap Regularization

- SAF, as introduced by Zhou et al. [2412.11377], enhanced the accuracy of fetal thalamus diameter and head circumference measurements in ultrasonography.
  - ICC improved from 0.684 (BiometryNet) to 0.737 with SAF ($a=1$), with similar gains for EfficientNet-based models.
  - The approach is architecture-agnostic and requires only a loss-term addition, not modification of forward activations.

## 4. Theoretical Insights: Initialization and Trainability

Hayou et al. [1805.08266] provided a theoretical foundation for Swish activations in deep, randomly initialized networks. Swish, owing to its smoothness and tunable nonlinearity, enables signal and gradient propagation "deeper" than ReLU:

- The edge-of-chaos criterion ($\chi_1=1$), preserving correlation and gradient magnitude across layers, is more tractable for Swish than for ReLU:
  - Swish admits a continuum of "edges" parameterized by $(\sigma_w, \sigma_b)$, allowing for nonzero bias and thus richer variance propagation.
  - On the edge, correlation collapse is polynomial ($O(\ell^{-p})$), not exponential; for ReLU, only one edge is available and prohibits nonzero bias.
- Swish’s depth-scale advantage explains its superior performance in very deep nets (often exceeding 100 layers).

## 5. Guidelines for Deployment and Tuning

- $\beta$-tuning: Default $\beta=1$ is effective in most settings. Grid-searching $\beta\in[0.5,2.0]$ or making it a trainable scalar can yield marginal performance gains [2110.07721], [1901.02671].
- Initialization: Xavier/Glorot normal weights with zero bias are recommended; avoid large initial weights that push activations into sigmoid tails.
- Optimizer regime: Swish’s rapid initial convergence supports early switching to second-order optimizers (e.g., L-BFGS) [2110.07721].
- SAF coefficients ($a,b,c$): For heatmap-based landmark detection, compute ground-truth MSE for initial calibration and sweep $a$ over small integer values.
- Depth and architecture: Swish/Swoosh activations are most beneficial in networks of moderate-to-greater depth; monitor gradient norms when stacking deeply.

## 6. Context: Limitations and Implementation Considerations

- *Computational overhead*: Swish/Swoosh activations incur a per-element sigmoid computation, which increases cost relative to ReLU (5–10$\times$) [2010.09458], [1901.02671].
- *Plug-and-play viability*: For tasks with minimal hyperparameter tuning, simple functions (sin, penalized-tanh) can exhibit greater stability and lower variance [1901.02671], [2010.09458].
- *Gradient sparsity*: Flatten-T Swish purposely zeroes gradients on $x<0$ to maintain sparsity, whereas canonical Swish retains them.
- *SAF specialization*: SAF is not used as a forward-pass activation but as a scalar penalty for regularized loss in heatmap distribution, applicable to any architecture outputting pairwise heatmaps [2412.11377].
- *Swoosh family distinctions*: Not all "swoosh" functions are activation nonlinearities; careful distinction is necessary between Swish/FTS (forward), SAF (regularization), and other variants.

## 7. Extensions and Outlook

Adaptive and spatially-varying Swish/Swoosh activations present promising extensions, as suggested in PINN literature [2110.07721]. Combining locally-learned gates $\beta(x)$ with the smooth, non-monotonic profile may further enhance representational power and compliance with heterogeneous or high-frequency domains. A plausible implication is the broader application of Swoosh-regularization frameworks like SAF beyond medical image analysis to general landmark detection or structured output learning, contingent on future empirical validation.

Source: https://www.emergentmind.com/topics/swoosh-activations