Swoosh Activations: Theory & Applications

• Swoosh activations are smooth, non-monotonic functions combining sigmoid gating with linear branches to enhance gradient propagation and feature learning.
• Variants like canonical Swish and Flatten-T Swish adjust parameters such as beta and threshold to optimize convergence and accuracy across diverse neural network architectures.
• The Swoosh Activation Function (SAF) employs a regularization penalty for heatmap-based landmark detection by tuning coefficients to align with optimal mean squared error targets.

The term "Swoosh activations" encompasses a family of activation functions characterized by a smooth, non-monotonic curve that resembles the "swish" or "swoosh" shape. These functions include canonical Swish ($x\cdot\sigma(\beta x)$), its parametric and thresholded variants such as Flatten-T Swish, and, more recently, the Swoosh Activation Function (SAF) designed for regularization in heatmap-based landmark detection. Their mathematical form typically combines a sigmoid gating with linear or modified branches, distinct from piecewise and monotonically-gated rectifiers. Swoosh-type activations are designed to boost expressivity, gradient flow, and trainability, especially in deep architectures or tasks with high-frequency and heterogeneous target structures.

1. Mathematical Definitions and Key Properties

Canonical Swish is defined as:

$$\mathrm{Swish}_\beta(x) = x \cdot \sigma(\beta x) \quad \text{where} \quad \sigma(z) = \frac{1}{1+e^{-z}},$$

with $\beta>0$ controlling the steepness of the gating. The unparameterized form ($\beta=1$) is common in practice (Al-Safwan et al., 2021, Eger et al., 2019, Szandała, 2020, Hayou et al., 2018).

Key properties:

• Smoothness: $C^\infty$; derivatives exist everywhere, unlike ReLU.
• Non-monotonicity: For $x < 0$, the curve dips below zero, facilitating richer feature learning.
• Non-zero derivatives: $$\frac{d}{dx}\mathrm{Swish}(x) = \sigma(x) + x \sigma(x)(1-\sigma(x))$$, ensuring gradient propagation across all regions.
• Limiting cases: As $\beta \to \infty$, Swish approximates ReLU; as $\beta \to 0$, it becomes the identity.
• Range: For $\beta=1$, minimum is $$\mathrm{Swish}_\beta(x) = x \cdot \sigma(\beta x) \quad \text{where} \quad \sigma(z) = \frac{1}{1+e^{-z}},$$0; unbounded above.

Flatten-T Swish (FTS) introduces a threshold:

$$\mathrm{Swish}_\beta(x) = x \cdot \sigma(\beta x) \quad \text{where} \quad \sigma(z) = \frac{1}{1+e^{-z}},$$1

$$\mathrm{Swish}_\beta(x) = x \cdot \sigma(\beta x) \quad \text{where} \quad \sigma(z) = \frac{1}{1+e^{-z}},$$2 sets a negative floor; $$\mathrm{Swish}_\beta(x) = x \cdot \sigma(\beta x) \quad \text{where} \quad \sigma(z) = \frac{1}{1+e^{-z}},$$3 performs best for MNIST deep FNNs, allowing negative value propagation while retaining sparse gradients on $$\mathrm{Swish}_\beta(x) = x \cdot \sigma(\beta x) \quad \text{where} \quad \sigma(z) = \frac{1}{1+e^{-z}},$$4 (Chieng et al., 2018).

Swoosh Activation Function (SAF) is a regularization penalty function for optimizing heatmap MSEs in landmark detection:

$$\mathrm{Swish}_\beta(x) = x \cdot \sigma(\beta x) \quad \text{where} \quad \sigma(z) = \frac{1}{1+e^{-z}},$$5

Optimal MSE $$\mathrm{Swish}_\beta(x) = x \cdot \sigma(\beta x) \quad \text{where} \quad \sigma(z) = \frac{1}{1+e^{-z}},$$6 aligns SAF's minimum with the desired regularization target, and $$\mathrm{Swish}_\beta(x) = x \cdot \sigma(\beta x) \quad \text{where} \quad \sigma(z) = \frac{1}{1+e^{-z}},$$7, $$\mathrm{Swish}_\beta(x) = x \cdot \sigma(\beta x) \quad \text{where} \quad \sigma(z) = \frac{1}{1+e^{-z}},$$8, $$\mathrm{Swish}_\beta(x) = x \cdot \sigma(\beta x) \quad \text{where} \quad \sigma(z) = \frac{1}{1+e^{-z}},$$9 tune its profile. SAF is not an activation in the standard forward-pass sense but enforces distributional sharpness/dispersion for outputs in heatmap regression (Zhou et al., 2024).

2. Comparative Role in Deep Networks

Swoosh-family activations remedy limitations of classical nonlinearities:

Activation	Smoothness	Gradient at $\beta>0$0	Monotonicity	Negative Signal
ReLU	Not smooth	Zero	Yes	None
Tanh/Sigmoid	Smooth	Small (Saturating)	Yes	Yes (but saturating)
Swish/Swoosh	Smooth	Small, nonzero	No	Yes (non-monotonic)
Flatten-T Swish	Piecewise smooth	Zero (x<0), $\beta>0$1	Partly	Yes (via $\beta>0$2)
SAF (Loss reg.)	N/A	N/A	N/A	N/A

Swish-type activations preserve gradient flow—especially for negative pre-activations—avoiding dead units ("dying ReLU"). Their non-monotonic bumps around zero enable networks to approximate functions with mixed-frequency content and sharp transitions more easily (Al-Safwan et al., 2021, Eger et al., 2019, Chieng et al., 2018).

3. Empirical Performance in Key Domains

Physics-Informed Neural Networks (PINNs), Helmholtz Equation

• Al-Safwan et al. (Al-Safwan et al., 2021) demonstrated that Swish activation markedly improves PINN convergence for 2D Helmholtz problems, particularly in models with sharp heterogeneities.
  • In an 8-layer, 20-neuron PINN, Swish accelerated initial loss decay by ~30% compared to tanh/ELU and reduced final $\beta>0$3 error by ~15%.
  • Real and imaginary wavefield errors ("two-scatter test") in $\beta>0$4 and $\beta>0$5 were $\beta>0$6 lower for Swish.
• Swish’s ability to represent high-wavenumber components—by leveraging non-monotonicity—overcomes the spectral bias toward low frequencies seen with tanh and ReLU.

Natural Language Processing

• In "Is it Time to Swish?" (Eger et al., 2019), Swish achieved strong results across 8 NLP tasks (sentence/document classification, sequence tagging):
  • Best-case Swish was top-5 among 21 activations; on certain sequence tasks, Swish exceeded ReLU by 1–3 percentage points.
  • In deeply stacked networks, Swish outperformed saturated functions (cube, cosid) and matched the performance of penalized-tanh and Maxout variants.
  • Training stability and convergence were on par with ReLU; Swish did not exhibit dead neurons.

Image Classification and Shallow CNNs

• On CIFAR-10 (2-conv CNN, 25 epochs), Swish (β=1) yielded 69.9% accuracy versus 71.8% for ReLU and 72.9% for Leaky ReLU (Szandała, 2020), indicating that Swish excels primarily in deeper architectures.
• Flatten-T Swish (T=-0.20) improved MNIST accuracy by up to 1.15% and halved the epochs needed for convergence in 512–1568 neuron deep FNNs (Chieng et al., 2018).

Landmark Detection via Heatmap Regularization

• SAF, as introduced by Zhou et al. (Zhou et al., 2024), enhanced the accuracy of fetal thalamus diameter and head circumference measurements in ultrasonography.
  • ICC improved from 0.684 (BiometryNet) to 0.737 with SAF ($\beta>0$7), with similar gains for EfficientNet-based models.
  • The approach is architecture-agnostic and requires only a loss-term addition, not modification of forward activations.

4. Theoretical Insights: Initialization and Trainability

Hayou et al. (Hayou et al., 2018) provided a theoretical foundation for Swish activations in deep, randomly initialized networks. Swish, owing to its smoothness and tunable nonlinearity, enables signal and gradient propagation "deeper" than ReLU:

• The edge-of-chaos criterion ($\beta>0$8), preserving correlation and gradient magnitude across layers, is more tractable for Swish than for ReLU:
  • Swish admits a continuum of "edges" parameterized by $\beta>0$9, allowing for nonzero bias and thus richer variance propagation.
  • On the edge, correlation collapse is polynomial ($\beta=1$0), not exponential; for ReLU, only one edge is available and prohibits nonzero bias.
• Swish’s depth-scale advantage explains its superior performance in very deep nets (often exceeding 100 layers).

5. Guidelines for Deployment and Tuning

• $\beta=1$1-tuning: Default $\beta=1$2 is effective in most settings. Grid-searching $\beta=1$3 or making it a trainable scalar can yield marginal performance gains (Al-Safwan et al., 2021, Eger et al., 2019).
• Initialization: Xavier/Glorot normal weights with zero bias are recommended; avoid large initial weights that push activations into sigmoid tails.
• Optimizer regime: Swish’s rapid initial convergence supports early switching to second-order optimizers (e.g., L-BFGS) (Al-Safwan et al., 2021).
• SAF coefficients ($\beta=1$4): For heatmap-based landmark detection, compute ground-truth MSE for initial calibration and sweep $\beta=1$5 over small integer values.
• Depth and architecture: Swish/Swoosh activations are most beneficial in networks of moderate-to-greater depth; monitor gradient norms when stacking deeply.

6. Context: Limitations and Implementation Considerations

• Computational overhead: Swish/Swoosh activations incur a per-element sigmoid computation, which increases cost relative to ReLU (5–10$\beta=1$6) (Szandała, 2020, Eger et al., 2019).
• Plug-and-play viability: For tasks with minimal hyperparameter tuning, simple functions (sin, penalized-tanh) can exhibit greater stability and lower variance (Eger et al., 2019, Szandała, 2020).
• Gradient sparsity: Flatten-T Swish purposely zeroes gradients on $\beta=1$7 to maintain sparsity, whereas canonical Swish retains them.
• SAF specialization: SAF is not used as a forward-pass activation but as a scalar penalty for regularized loss in heatmap distribution, applicable to any architecture outputting pairwise heatmaps (Zhou et al., 2024).
• Swoosh family distinctions: Not all "swoosh" functions are activation nonlinearities; careful distinction is necessary between Swish/FTS (forward), SAF (regularization), and other variants.

7. Extensions and Outlook

Adaptive and spatially-varying Swish/Swoosh activations present promising extensions, as suggested in PINN literature (Al-Safwan et al., 2021). Combining locally-learned gates $\beta=1$8 with the smooth, non-monotonic profile may further enhance representational power and compliance with heterogeneous or high-frequency domains. A plausible implication is the broader application of Swoosh-regularization frameworks like SAF beyond medical image analysis to general landmark detection or structured output learning, contingent on future empirical validation.