---
title: Learnable Activation Functions
url: https://www.emergentmind.com/topics/learnable-activation-functions
type: topic
---

# Learnable Activation Functions

A learnable activation function is a nonlinearity $f:\mathbb{R}\to\mathbb{R}$ whose shape is determined by trainable parameters and is learned jointly with the other weights of a neural network. Instead of statically fixing the activation at design time (e.g., ReLU, tanh, GELU), the network allocates additional parameter degrees of freedom to express a broad class of nonlinear functions, enabling adaptive, data-driven optimization of the activation profile. This paradigm is now found across multilayer perceptrons, convolutional architectures, random feature models, kernel machines, Kolmogorov–Arnold networks, and domain-specific systems such as physics-informed neural networks (PINNs). The section below synthesizes the methodologies and analysis from contemporary research.

## 1. Mathematical Formulation and Basis Families

A learnable activation $f(x;\theta)$ is generally written as an expansion over fixed basis functions:
\[
f(x;\theta) = \sum_{i=1}^m \alpha_i\,g_i(x;\phi_i)
\]
where $g_i$ are fixed or parameterized basis functions (e.g., monomials, polynomials, radial basis functions (RBFs), trigonometric functions, splines, or piecewise-linear elements), $\alpha_i$ are trainable linear weights, and $\phi_i$ are further internal parameters if the basis is itself parameterized (e.g., RBF centers/widths, spline knots). In the case of piecewise polynomials,
\[
f(x;\theta) = \sum_{k=0}^{K-1} c_k\,B_k(x)
\]
with $B_k$ as B-spline basis functions and $c_k$ learnable coefficients [2411.15111].

The following basis families are central:

| Family        | Example Formula                                 | Trainable Parameters                     |
|---------------|-------------------------------------------------|------------------------------------------|
| Gaussian RBF  | $\sum_{i=1}^k\alpha_i \exp(-\|x-c_i\|^2/\sigma^2)$ | $\alpha_i, c_i, \sigma$                  |
| Fourier       | $\sum_{k=1}^K [ a_k\cos(kx) + b_k\sin(kx) ]$   | $a_k, b_k$                               |
| Chebyshev     | $\sum_{d=0}^D \alpha_d T_d(x)$                  | $\alpha_d$                               |
| Jacobi        | $\sum_{d=0}^D \alpha_d P_d^{(\alpha,\beta)}(x)$ | $\alpha_d$                               |
| B-spline      | $f(x)=\lambda_0\,\mathrm{SiLU}(x)+\lambda_1\sum c_i B_{i,d}(x)$ | $\lambda_0,\lambda_1,c_i$                |

These families allow precise tailoring of the activation's locality or globality, smoothness, and spectral properties [2411.15111, 2502.01247]. The dual-parameter forms (e.g., Tangma, Dual Parametric ReLU) introduce explicit learned shifts and slopes [2507.10560, 1912.10752].

## 2. Training Protocols and Architectural Integration

Learnable activations are inserted wherever static nonlinearities would occur:

- **MLPs and CNNs:** Each layer can be assigned its own activation $f^\ell(\cdot;\theta^\ell)$. The full forward pass augments the network parameter set $\Theta$ to include both weight/bias matrices and the activation parameters. Standard optimization steps (SGD/Adam) propagate through both sets [2411.15111, 1902.03306, 1912.12187].
- **Piecewise Linear / Spline Approaches:** In APL, SPLASH, or TV-regularized splines, per-neuron or per-layer activation parameters are updated via gradient descent, sometimes with additional sparsity or total-variation regularizers to control complexity [1412.6830, 2006.08947, 2210.16222].
- **Basis Expansion Methods:** Polynomial, Fourier, or RBF coefficients are parameterized at a chosen degree/order. Proper variance-preserving initialization is critical to stable learning [2502.01247, 2411.15111].
- **Random Feature Models:** Learnable activation functions enter as basis expansions on random projections, with weights adapted by ridge regression or similar schemes [2510.15327, 2411.19468].
- **Transformers:** Substituting fixed GELU with rational function activations (RAF) per feed-forward block, adding only a small parametric overhead and yielding marked empirical gains [2208.14111].

Kolmogorov–Arnold Networks (KANs) replace fixed post-summation nonlinearities with learnable edge (input-variable) univariate nonlinearities, allowing universal function approximation with much smaller models [2411.15111, 2407.04149].

## 3. Expressivity, Training Dynamics, and Trade-offs

Learnable activation functions offer quantifiable increases in expressivity, often characterized by comparisons of Neural Tangent Kernel (NTK) spectra, Hessian eigenvalues, and approximation-theoretic coverage:

- **Spectral Bias:** Studies in PINNs reveal that families such as B-splines and RBFs produce flatter NTK eigenvalue spectra (reduced spectral bias), enabling rapid acquisition of high-frequency target components [2411.15111]. However, excessively flat spectra can cause convergence instability due to large Hessian curvature (large $\lambda_{\max}(H)$), while steeply biased bases delay the acquisition of high-frequency information but favor stable, monotonic learning.
- **Approximation Power:** Hermite, Fourier, and tropical polynomial activations can be initialized and tuned to approximate classical activations (ReLU, GELU) to arbitrary accuracy, enabling transfer or fine-tuning of pre-trained models [2502.01247, 2411.15111].
- **Model Capacity:** Layer- or channel-wise degree allocation in polynomial or spline-based activations increases the total degree or piecewise granularity of the network, enhancing the effective representational power relative to fixed activations [1906.09529].
- **Stability and Scalability:** Local-support bases (e.g., B-splines, RBFs) are preferred in regimes with sharp gradients or non-periodic boundary conditions (e.g., stiff classic PDEs), mitigating the Gibbs phenomenon and promoting stable curvature [2411.15111].

The figure below summarizes NTK and Hessian trade-offs for representative bases in neural PDE solvers [2411.15111]:

| Activation Base   | NTK Spectral Bias | Hessian $\lambda_{max}$ | Generalization |
|-------------------|------------------|-------------------------|----------------|
| B-spline, RBF     | Low              | Moderate/Low            | Stable, high   |
| Fourier, Chebyshev| Lowest           | Highest                 | Unstable, problem-dependent |
| Tanh (fixed)      | Moderate         | Moderate                | Most robust on smooth PDEs |

## 4. Specialized Constructions and Algorithmic Innovations

Several strategies have emerged for constructing learnable activation functions:

- **Parametric Combinations:** Linear and affine convex combinations of a small basis set (e.g., ReLU, tanh, id) with simplex or affine constraints on coefficients, yielding monotonicity (convex-hull) or non-monotonic richness (affine-hull) [1801.09403].
- **Discrete Mixture Selection:** Gumbel-Softmax and straight-through estimator for stochastic selection among a finite dictionary of candidate functions during training (FlexAct) [2601.06441]. Regularization prevents scale bias toward unbounded activations.
- **Series Expansions:** Learnable Series Linear Units (LSLU) employ several shifted base activations summed with learnable amplitude and residual linear terms to increase per-layer nonlinearity and improve generalization in shallower networks [2409.08283].
- **Rational and Polynomial Parameterization:** Rational function activations (as ratio of learnable polynomials) can be plugged directly into transformers, while learnable polynomial activations are now subject to global polynomial optimization (e.g., via Moment-SOS hierarchy) for certified solution recovery, especially in low-dimension/dataset settings [2510.03682, 2208.14111].
- **Functional/Optimization Theoretic:** For Lipschitz-constrained nets, the theoretically optimal 1-Lipschitz activation under second-order total variation regularization is a linear spline, and optimal parameters can be recovered with a representer theorem [2210.16222].

## 5. Applications, Empirical Results, and Task-Specific Guidelines

Empirical evaluations consistently report that well-constructed learnable activation functions improve test accuracy, convergence, and sometimes adversarial robustness across domains:

- **Physics-Informed Neural Networks (PINNs):** Proper basis selection is task-dependent. For globally oscillatory PDEs, tanh or low-degree Chebyshev suffice; for multi-scale or sharp-boundary PDEs, local-support bases (B-splines, RBFs) are critical. Overly global bases risk instability unless stringent boundary handling is implemented [2411.15111].
- **Transformer Language Models:** RAFs yield lower perplexity and higher accuracy (up to +5.7 points on GLUE in low-data regimes) than GELU, with each pre-trained layer specializing its solution to the target task [2208.14111].
- **Vision Networks:** Empirical gains of 0.5–3 percentage points accuracy are typical for series-based (LSLU), spline-based, or basis-combination activations, with typically small per-layer parameter and runtime overhead [2409.08283, 2006.08947, 1412.6830].
- **Random Feature Models:** Integrating learnable activation functions with effective weighted-sampling (e.g., via leverage scores) achieves low excess risk with an order of magnitude fewer features than plain random sampling; see RFLAF and its empirical validation [2510.15327, 2411.19468].

Guidance:

- Select basis family and support according to the anticipated task structure (e.g., support sharp gradients or global oscillations).
- Monitor NTK spectrum and Hessian eigenvalues to ensure neither excessive bias nor unstable curvature (e.g., $\lambda_{\max}(H)<10^3$ at learning rate $10^{-3}$ for Adam) [2411.15111].
- Use shallow models with moderate parameterized activations as a starting point, increasing model complexity only if performance and convergence remain stable.
- For activation selection problems, discrete approaches (e.g., FlexAct) offer modularity and interpretability [2601.06441].

## 6. Limitations, Open Problems, and Theoretical Foundations

While learnable activation functions have demonstrated broad utility, certain constraints and limitations are observed:

- **Scalability:** For very high-dimensional or deep models, local minima, optimization sensitivity, and parameter blow-up can become problematic, particularly with global supports or high-degree polynomials [2411.15111, 2502.01247].
- **Initialization:** Proper coefficient normalization and variance-preserving schemes are pivotal, especially for polynomial, Fourier, or spline-based parameterizations [2502.01247].
- **Generalization and Overfitting:** Overly rich activations without regularization can induce overfitting. Penalties such as $L_1$ on expansion coefficients or second-order total variation for splines are recommended [2210.16222].
- **Implementation Overhead:** Parameter and computational costs, as well as the need for careful scheduling (e.g., activation hyperparameters, temperature annealing), can increase relative to static activations [2409.08283, 2006.08947].
- **Nonconvexity:** For polynomial activations, global optimization (Moment-SOS) is possible for small models, but the approach is computationally prohibitive for large-scale networks [2510.03682].

Theoretical results guarantee universal approximation power, explicit polynomial representability, and (in regularized spline cases) existence and finite support of optimal activation forms under broad functional-analytic constraints [1906.09529, 2210.16222].

## 7. Representative Implementations and Practical Recipes

Several open-source implementations and recipes are available:

- **torchortho** provides Hermite, Fourier, and tropical polynomial activations with variance-preserving initialization for PyTorch [2502.01247].
- **Spline-based activations** and projection layers for 1-Lipschitz functions are available for plug-and-play with standard autodiff frameworks [2210.16222].
- **Random Feature Models with Learnable Activations** are implementable in PyTorch by broadcasting activation coefficient vectors over random projections [2411.19468].
- **Transformers:** RAFs, replacing static GELU in standard transformer architectures, increase parameter count negligibly but enable on-the-fly adaptation [2208.14111].
- **SineKAN**: Learnable sinusoidal edge activations in Kolmogorov–Arnold Networks, initialized with phase grids and amplitude/frequency parameters, implemented with standard stochastic optimization [2407.04149].

In conclusion, learnable activation functions offer a flexible and theoretically grounded means of enhancing neural model expressivity, are accompanied by mature methodologies for basis construction and optimization, and are proven to confer measurable gains across a variety of settings, provided their parameterization and regularization are adapted to the structure of the specific learning problem.

Source: https://www.emergentmind.com/topics/learnable-activation-functions