Adaptive Activation Functions (AAFs)

• Adaptive activation functions (AAFs) are nonlinear units with trainable parameters that adapt their shape during learning to optimize network performance and flexibility.
• They encompass various designs, including parametric-standard, piecewise, and ensemble methods, which tailor activations via adaptive slopes, affine warping, or basis expansions.
• AAFs improve convergence speed, accuracy, and robustness in applications ranging from image classification and regression to physics-informed and sparse-data neural networks.

Searching arXiv for recent and foundational papers on adaptive activation functions to ground the article in cited literature. Adaptive activation functions (AAFs) are activation functions whose exact shape is not fixed a priori but is governed by one or more trainable parameters that are optimized during network training (Hammad, 2024). In contrast to fixed-shape activation functions such as classic Sigmoid, Tanh, and ReLU, and to parametric standard functions whose overall shape remains closely tied to the original base function, AAFs allow richer functional adaptations, including piecewise linear segments, basis-function expansions, and learnable combinations of candidate nonlinearities (Hammad, 2024). Across the literature, AAFs have been used to tune slope, scale, bias, smoothness, threshold, or basis coefficients, with applications spanning regression, image classification, uncertainty-aware modeling with sparse data, physics-informed neural networks (PINNs), force fields, and signal-processing-inspired architectures (Jagtap et al., 2019, Hou et al., 2016, Pourkamali-Anaraki et al., 2024, Wang et al., 2023, Li, 2024, Martinez-Gost et al., 2023).

1. Taxonomy and conceptual scope

The survey taxonomy distinguishes fixed-shape activation functions, trainable activation functions, and trainable non-standard neurons (Hammad, 2024). Within trainable activation functions, two recurrent families are especially relevant. The first is the parametric-standard family, in which a classic activation is augmented by a small number of trainable parameters. Examples listed in the survey include PReLU, FReLU, SReLU, and PELU (Hammad, 2024). The second is the ensemble and mixture family, in which the learned nonlinearity is formed from multiple components, as in Adaptive Piecewise Linear (APL), Mexican ReLU, Look-up Table Units, and Bi-modal Derivative Adaptive Activation (Hammad, 2024).

A broader operational view appears across the primary papers. Some AAFs modify only the activation slope, as in the trainable scaling formulations used for deep networks and PINNs (Jagtap et al., 2019) and in layer-wise or neuron-wise locally adaptive activation functions (Jagtap et al., 2019). Some customize an existing activation by learning affine warping parameters, as in the adaptive Sigmoid, Tanh, and ReLU family defined by

$\phi_i(z)= b_i\,f(a_i z + c_i)+d_i$

or, for adaptive ReLU,

$\phi_i^{\rm AR}(z)=\max(a_i z + c_i,\;b_i z + d_i)$

with four learned parameters per layer (Hu et al., 2021). Others learn a new shape directly through hinge functions, polynomials, splines, basis expansions, or function mixtures (Agostinelli et al., 2014, Jang et al., 2018, Hou et al., 2016, Martinez-Gost et al., 2023, Wang et al., 2023, Li, 2024).

This suggests that “adaptive activation function” is best understood as a structural design principle rather than a single mechanism. The shared feature is joint optimization of activation-shape parameters with weights and biases; the distinguishing feature is the degree of functional freedom made available to the nonlinearity.

2. Core mathematical formulations

One of the simplest AAF constructions is the trainable slope or gain. In a standard feed-forward network, the fixed nonlinearity $\sigma(L_k(x_{k-1}))$ is replaced by $\sigma(a\,L_k(x_{k-1}))$ or more generally $\sigma(a_k\odot L_k(x_{k-1}))$, so that the slope of each layer’s activation becomes trainable (Jagtap et al., 2019). In the locally adaptive formulation, the same idea is expressed as layer-wise

$f_l(x)=\sigma(\alpha_l\cdot x)$

or neuron-wise

$f_{l,i}(x)=\sigma(\alpha_{l,i}\cdot x)$

with one learned scaling parameter per layer or per neuron (Jagtap et al., 2019).

A second major formulation is the adaptive affine warping of classical activations. The methodology that customizes Sigmoid, Tanh, and ReLU uses

$\phi_i(z)= b_i\,f(a_i z + c_i)+d_i$

where $a_i$ and $c_i$ warp the input to the base activation, while $\phi_i^{\rm AR}(z)=\max(a_i z + c_i,\;b_i z + d_i)$0 and $\phi_i^{\rm AR}(z)=\max(a_i z + c_i,\;b_i z + d_i)$1 scale and shift the output (Hu et al., 2021). The paper emphasizes that this adds only four parameters per layer (Hu et al., 2021).

A third formulation learns the activation as a piecewise or basis-expanded function. The APL unit defines, for neuron $\phi_i^{\rm AR}(z)=\max(a_i z + c_i,\;b_i z + d_i)$2,

$\phi_i^{\rm AR}(z)=\max(a_i z + c_i,\;b_i z + d_i)$3

where the hinge slopes and hinge locations are learned by gradient descent (Agostinelli et al., 2014). SAAF instead uses a piecewise polynomial form,

$\phi_i^{\rm AR}(z)=\max(a_i z + c_i,\;b_i z + d_i)$4

with trainable coefficients $\phi_i^{\rm AR}(z)=\max(a_i z + c_i,\;b_i z + d_i)$5 and $\phi_i^{\rm AR}(z)=\max(a_i z + c_i,\;b_i z + d_i)$6 and fixed breakpoints and degree (Hou et al., 2016).

Other architectures replace static activations with explicit basis-function models. ENN models the activation by a truncated DCT expansion with trainable DCT coefficients (Martinez-Gost et al., 2023). TAAFS replaces a fixed activation with a basis expansion

$\phi_i^{\rm AR}(z)=\max(a_i z + c_i,\;b_i z + d_i)$7

using B-splines, Fourier modes, Chebyshev polynomials, radial-basis functions, or related kernels, with a single $\phi_i^{\rm AR}(z)=\max(a_i z + c_i,\;b_i z + d_i)$8-vector per layer in the default configuration (Li, 2024). In PINNs, ABU-PINN writes the learned activation as

$\phi_i^{\rm AR}(z)=\max(a_i z + c_i,\;b_i z + d_i)$9

where $\sigma(L_k(x_{k-1}))$0 is chosen as softmax, the $\sigma(L_k(x_{k-1}))$1 are smooth candidate functions, and $\sigma(L_k(x_{k-1}))$2 are adaptive slopes (Wang et al., 2023).

A fourth formulation makes the activation context-aware. ASH gates a unit against an adaptive threshold based on feature-map statistics,

$\sigma(L_k(x_{k-1}))$3

where $\sigma(L_k(x_{k-1}))$4 is a trainable percentile parameter and the threshold depends on the mean and standard deviation of the current feature map (Lee et al., 2022). Activation Networks go further by using an auxiliary network to generate polynomial coefficients for each pixel, node, and layer, so that the activation of a feature depends on the features of neighboring pixels in a convolutional layer and other nodes in a dense layer (Jang et al., 2018).

3. Optimization mechanisms and theoretical interpretations

A common property of AAFs is that activation parameters are updated jointly with the ordinary network parameters by standard back-propagation or first-order optimization (Jagtap et al., 2019, Hou et al., 2016, Hu et al., 2021). In the slope-based PINN setting, one simply treats the adaptive parameter as another trainable variable and updates

$\sigma(L_k(x_{k-1}))$5

(Jagtap et al., 2019). In ABU-PINN, the network weights $\sigma(L_k(x_{k-1}))$6 and the activation parameters $\sigma(L_k(x_{k-1}))$7 are minimized jointly over the full PINN loss, typically with Adam and optionally L-BFGS, without alternating or block-coordinate updates (Wang et al., 2023).

Several papers interpret the resulting dynamics as more than a cosmetic rescaling of the nonlinearity. The 2019 PINN work states that introducing a scalable hyper-parameter dynamically changes the topology of the loss function involved in the optimization process (Jagtap et al., 2019). The locally adaptive formulation gives a more explicit optimization account: adaptive activation methods accelerate convergence by implicitly multiplying conditioning matrices to the gradient of the base method without any explicit computation of the conditioning matrix and the matrix-vector product, and the induced dynamics are not achievable by base methods with any adaptive learning rates (Jagtap et al., 2019). The same paper proves that, under practical conditions on initialization and learning rate, the gradient descent algorithms are not attracted to sub-optimal critical points or local minima (Jagtap et al., 2019).

Related arguments appear in the adaptive affine family. There, the input-scale parameter acts like a local scaling of the effective learning rate, with

$\sigma(L_k(x_{k-1}))$8

in a one-layer subnetwork analysis (Hu et al., 2021). In the regression-oriented SAAF framework, the emphasis shifts from conditioning to function-class control: an NN with SAAFs is Lipschitz continuous given a bounded magnitude of the NN parameters, and the paper proves an upper-bound for model complexity in terms of fat-shattering dimension for any Lipschitz continuous regression model (Hou et al., 2016). The APL paper instead stresses representation: any continuous piecewise-linear function that is asymptotically linear on the far left and far right can be written in the APL form for sufficiently large $\sigma(L_k(x_{k-1}))$9 (Agostinelli et al., 2014).

This suggests two complementary theoretical narratives. One views AAFs primarily as an optimization device that reshapes gradients, curvature, or conditioning (Jagtap et al., 2019, Jagtap et al., 2019). The other views them as a controlled enlargement of the hypothesis class, often with explicit regularity or approximation properties (Agostinelli et al., 2014, Hou et al., 2016).

4. Representative architectures and design patterns

The literature exhibits a small number of recurring design patterns. A first pattern is minimal-parameter adaptation, where only one or a few scalar parameters are introduced per layer or per neuron. Examples include trainable slopes in AAFs for PINNs (Jagtap et al., 2019), L-LAAF and N-LAAF (Jagtap et al., 2019), the four-parameter adaptive Sigmoid/Tanh/ReLU family (Hu et al., 2021), Adaptive Gumbel and smooth adaptive ReLU (Farhadi et al., 2019), and APALU, which uses two trainable positive parameters $\sigma(a\,L_k(x_{k-1}))$0 with a fixed sharpness constant $\sigma(a\,L_k(x_{k-1}))$1 (Subramanian et al., 2024). ErfReLU also belongs to this class, with a single trainable parameter $\sigma(a\,L_k(x_{k-1}))$2: $\sigma(a\,L_k(x_{k-1}))$3 (Rajanand et al., 2023).

A second pattern is piecewise and spline-like adaptation. APL learns hinge slopes and hinge positions independently for each neuron (Agostinelli et al., 2014). SAAF uses piecewise polynomials with trainable coefficients and fixed breakpoints, emphasizing smoothness and universal approximation on bounded intervals (Hou et al., 2016). APALU uses a piecewise positive branch and an ELU-style negative branch (Subramanian et al., 2024).

A third pattern is mixture and basis expansion. ABU-PINN learns convex combinations of smooth candidate activations using softmax gates and adaptive slopes (Wang et al., 2023). TAAFS learns coefficients over B-splines, Fourier modes, orthogonal polynomials, and related kernels, with per-layer adaptation identified as the default and per-neuron adaptation reported as too many parameters leading to gradient vanishing (Li, 2024). ENN employs DCT basis functions and links the learned activation to an interpretable “bump” in output space (Martinez-Gost et al., 2023).

A fourth pattern is context-aware or auxiliary-network adaptation. ASH computes an adaptive threshold from $\sigma(a\,L_k(x_{k-1}))$4 and $\sigma(a\,L_k(x_{k-1}))$5 and learns how sparse or dense each activation should be (Lee et al., 2022). Activation Networks assign each pixel, node, and layer a polynomial activation function whose coefficients are generated by a small auxiliary network conditioned on neighboring or peer activations (Jang et al., 2018).

A fifth pattern is task-guided activation design. In PINNs, candidate functions are selected to satisfy differentiability requirements and to reflect prior knowledge about the PDE. The ABU-PINN paper excludes piecewise-linear functions such as ReLU and ELU when higher-order PDE derivatives are required, retains smooth activations such as tanh, sigmoid, Softplus, GELU, and Swish, and may include $\sigma(a\,L_k(x_{k-1}))$6, $\sigma(a\,L_k(x_{k-1}))$7, or $\sigma(a\,L_k(x_{k-1}))$8 according to oscillatory or decaying solution structure (Wang et al., 2023). The 2026 wavelet-based PINN family extends this task-guided strategy with SoftMexTanh, SoftMorTanh, SoftGaussTanh, SoftGaborTanh, and SoftHerTanh (Murari, 18 Mar 2026).

5. Empirical behavior across domains

Empirical results reported in the cited papers are heterogeneous, but several recurring patterns emerge. In deep function approximation and PINNs, adaptive slopes substantially accelerate convergence. For smooth and discontinuous one-dimensional regression targets, a four-hidden-layer tanh network with adaptive activation and $\sigma(a\,L_k(x_{k-1}))$9 captures high modes in fewer iterations than fixed tanh, and final $\sigma(a_k\odot L_k(x_{k-1}))$0 errors reduce by $\sigma(a_k\odot L_k(x_{k-1}))$1–$\sigma(a_k\odot L_k(x_{k-1}))$2 (Jagtap et al., 2019). For PINNs, the same paper reports relative $\sigma(a_k\odot L_k(x_{k-1}))$3 versus $\sigma(a_k\odot L_k(x_{k-1}))$4 for viscous Burgers after 2000 Adam steps, $\sigma(a_k\odot L_k(x_{k-1}))$5 versus $\sigma(a_k\odot L_k(x_{k-1}))$6 for nonlinear Klein–Gordon after 1400 steps, and $\sigma(a_k\odot L_k(x_{k-1}))$7 versus $\sigma(a_k\odot L_k(x_{k-1}))$8 for Helmholtz in 3600 steps (Jagtap et al., 2019).

The locally adaptive framework reports strong improvements across both scientific and standard deep-learning settings. On the inverse Poisson problem, relative $\sigma(a_k\odot L_k(x_{k-1}))$9 errors at iteration 4000 were $f_l(x)=\sigma(\alpha_l\cdot x)$0 for fixed $f_l(x)=\sigma(\alpha_l\cdot x)$1, $f_l(x)=\sigma(\alpha_l\cdot x)$2 for GAAF, $f_l(x)=\sigma(\alpha_l\cdot x)$3 for L-LAAF, and $f_l(x)=\sigma(\alpha_l\cdot x)$4 for N-LAAF (Jagtap et al., 2019). On standard vision benchmarks, pre-activation ResNet-18 or a LeNet variant trained with slope-recovery and local adaptivity achieved a $f_l(x)=\sigma(\alpha_l\cdot x)$5–$f_l(x)=\sigma(\alpha_l\cdot x)$6 faster decay of the training cross-entropy loss than the fixed-$f_l(x)=\sigma(\alpha_l\cdot x)$7 baseline (Jagtap et al., 2019).

In regression, SAAF-equipped networks achieved state-of-the-art results on multiple datasets (Hou et al., 2016). Concrete examples reported include pose estimation on LSP, where average PCP improved from $f_l(x)=\sigma(\alpha_l\cdot x)$8 with ReLU to $f_l(x)=\sigma(\alpha_l\cdot x)$9 with R-SAAFc2, age estimation on Adience from $f_{l,i}(x)=\sigma(\alpha_{l,i}\cdot x)$0 with PReLU to $f_{l,i}(x)=\sigma(\alpha_{l,i}\cdot x)$1 with R-SAAFc2, attractiveness RMSE from $f_{l,i}(x)=\sigma(\alpha_{l,i}\cdot x)$2 with ReLU to $f_{l,i}(x)=\sigma(\alpha_{l,i}\cdot x)$3 with SAAFc2, and nuclear circularity RMSE from $f_{l,i}(x)=\sigma(\alpha_{l,i}\cdot x)$4 with ReLU to $f_{l,i}(x)=\sigma(\alpha_{l,i}\cdot x)$5 with R-SAAFc1 (Hou et al., 2016).

In image classification, several lightweight AAFs report modest but consistent gains. The ErfReLU study evaluates nine trainable activation functions plus the proposed one on MobileNetV1, ResNet-18, and VGG16 over CIFAR-10, MNIST, and Fashion-MNIST (Rajanand et al., 2023). On CIFAR-10, ErfReLU is reported at $f_{l,i}(x)=\sigma(\alpha_{l,i}\cdot x)$6 on MobileNet and $f_{l,i}(x)=\sigma(\alpha_{l,i}\cdot x)$7 on ResNet-18, while VGG16 is slightly better with ErfAct at $f_{l,i}(x)=\sigma(\alpha_{l,i}\cdot x)$8 versus $f_{l,i}(x)=\sigma(\alpha_{l,i}\cdot x)$9 for ErfReLU (Rajanand et al., 2023). APALU reports CIFAR-10 test accuracy improvements on MobileNet, GoogleNet, ResNet50, and SENet18, as well as gains in anomaly detection, sign-language recognition, and regression tasks (Subramanian et al., 2024). The adaptive-customization methodology also reports strong results across AlexNet, VGGNet, GoogLeNet, ResNet, and DenseNet on CIFAR, miniImageNet, PASCAL VOC, and COCO, with empirical loss-vs-epoch curves showing that the proposed AAFs converge in $\phi_i(z)= b_i\,f(a_i z + c_i)+d_i$0–$\phi_i(z)= b_i\,f(a_i z + c_i)+d_i$1 of the epochs needed by fixed activations (Hu et al., 2021).

In sparse-data scientific classification, per-neuron trainable parameters appear more useful than layer-shared ones. On additive-manufacturing datasets with fewer than one hundred samples, Model 3 with individual parameters per neuron consistently out-performs fixed-shape and shared-parameter variants, with ELU and Softplus more robust than Swish under uncertainty measures (Pourkamali-Anaraki et al., 2024). The same study reports that conformal-prediction metrics reveal cases where high point accuracy masks low confidence or calibration (Pourkamali-Anaraki et al., 2024).

In scientific computing, PINNs and force fields provide especially strong evidence for problem-specific activation adaptation. ABU-PINN reports up to $\phi_i(z)= b_i\,f(a_i z + c_i)+d_i$2 reduction in $\phi_i(z)= b_i\,f(a_i z + c_i)+d_i$3 error versus tanh or sin alone on one-dimensional Poisson problems, an average error drop from $\phi_i(z)= b_i\,f(a_i z + c_i)+d_i$4 to $\phi_i(z)= b_i\,f(a_i z + c_i)+d_i$5 on a set including convection, Burgers’, Allen–Cahn, KdV, and Cahn–Hilliard, and $\phi_i(z)= b_i\,f(a_i z + c_i)+d_i$6–$\phi_i(z)= b_i\,f(a_i z + c_i)+d_i$7 error reduction on two-dimensional Navier–Stokes inverse problems (Wang et al., 2023). TAAFS reports performance gains in neural network force fields with only dozens of additional parameters, including reduced energy RMSE by $\phi_i(z)= b_i\,f(a_i z + c_i)+d_i$8–$\phi_i(z)= b_i\,f(a_i z + c_i)+d_i$9 and force RMSE by $a_i$0–$a_i$1 across eight inorganic systems with the RLEKF optimizer, albeit at longer epoch times (Li, 2024). The wavelet-based PINN family reports substantially lower loss, rMAE, and rRMSE than tanh on representative reaction, wave, convection, and Navier–Stokes benchmarks, with SoftGaborTanh or SoftHerTanh variants frequently strongest (Murari, 18 Mar 2026).

6. Interpretability, practical guidance, and open issues

Many AAF papers emphasize that learned activations can be inspected rather than treated as opaque nuisance parameters. In ABU-PINN, the softmax gate weights identify which candidate basis dominates. Reported examples include a one-dimensional Poisson problem with sin-weight approximately $a_i$2–$a_i$3 and exp-weight approximately $a_i$4, a decaying Poisson problem with exp-weight approximately $a_i$5–$a_i$6, a convection problem whose learned activation sits roughly between $a_i$7 and $a_i$8, and a two-dimensional Navier–Stokes case that learns a smooth blend of GELU and sin in early layers and GELU–tanh mixtures in late layers (Wang et al., 2023). ENN recovers the concept of a “bump,” allowing one to plot each learned activation and associate it with local response regions in input space (Martinez-Gost et al., 2023). TAAFS likewise recommends monitoring the shape of the learned basis expansion during training and eliminating modes that produce high-frequency oscillations when stable dynamical behavior is required (Li, 2024).

Several practical rules recur. Initialization is usually chosen so that the network starts from a familiar activation or unbiased mixture: for slope-based AAFs one initializes $a_i$9 or $c_i$0 and $c_i$1 in ABU-PINN (Jagtap et al., 2019, Wang et al., 2023). Per-layer adaptation is often the default compromise between flexibility and cost; TAAFS identifies per-layer adaptation as the “sweet spot,” whereas per-neuron TAAFS caused rapid gradient vanishing (Li, 2024). In sparse-data tabular classification, however, per-neuron parameters were precisely the setting that produced the best trade-off of accuracy and calibrated confidence (Pourkamali-Anaraki et al., 2024). This suggests that the optimal granularity of adaptation depends strongly on task regime, architecture, and parameter budget.

Smoothness constraints are critical in scientific machine learning. ABU-PINN explicitly removes ReLU and ELU from the candidate set when second- or third-order PDE residuals are used, because PINN residuals require well-defined higher-order derivatives (Wang et al., 2023). The wavelet-based PINN paper similarly uses Softplus reparameterization to guarantee positivity of scale parameters and recommends fixing the tanh scale in unstable cases such as convection (Murari, 18 Mar 2026). More generally, the survey notes that piecewise methods must ensure derivative continuity or handle subgradient issues, and that excessively flexible activations may require stronger regularization or early stopping (Hammad, 2024).

The main open issues reported across the literature concern overfitting risk, hyperparameter selection, computational overhead, and scaling. Highly flexible activation families can over-adapt to noise or introduce many local minima, as noted for LuTU-like flexibility in the survey and for ENN when the number of DCT coefficients increases (Hammad, 2024, Martinez-Gost et al., 2023). Some methods add negligible overhead, such as single-parameter activations or four parameters per layer (Rajanand et al., 2023, Hu et al., 2021), while others incur substantial training-time increases, as in TAAFS with B-spline bases or some activation-network constructions (Li, 2024, Jang et al., 2018). Automated search over activation layouts, basis families, or adaptive structures remains an active direction, with Optuna-based “a la carte” activation selection, dynamic mixtures, and task-specific design all pointing toward increasingly learned nonlinearity architectures (Sipper, 2022, Wang et al., 2023).

Taken together, the literature presents AAFs as a broad family of mechanisms for learning nonlinear response laws rather than merely selecting them. The empirical record supports gains in convergence speed, precision, robustness, uncertainty behavior, and PDE-specific fit across many settings (Jagtap et al., 2019, Jagtap et al., 2019, Hou et al., 2016, Pourkamali-Anaraki et al., 2024, Wang et al., 2023, Li, 2024). A plausible implication is that future progress will depend less on proposing a single universally superior activation and more on matching activation adaptability, smoothness, and granularity to the structure of the task.