---
title: Learnable Activation Selection Mechanism
url: https://www.emergentmind.com/topics/learnable-activation-selection-mechanism
type: topic
---

# Learnable Activation Selection Mechanism

A learnable activation selection mechanism is a method that treats the choice of nonlinearity as a trainable architectural variable rather than a fixed hyperparameter. In the literature, this can mean selecting one activation from a predefined library, learning a parametric activation family, searching over activation configurations by discrete optimization, or fitting an activation through a basis expansion whose coefficients are learned jointly with the model. Recent formulations include discrete-yet-differentiable routing with Gumbel-Softmax, layer-wise search with early evaluation, evolutionary discovery of activation forms, rational activations in Transformers, and basis-function models in random features and radial-basis networks [2601.06441], [1811.03980], [1808.00783], [2208.14111], [1911.09257], [2411.19468].

## 1. Concept and scope

In the standard feedforward formulation,
$$
F(x)=g_N\circ g_{N-1}\circ \dots \circ g_1(x), \qquad g_j(z)=\sigma_j(W_j z+b_j),
$$
the activation is usually fixed before training. Learnable activation selection changes this by elevating the activation to a learned design degree of freedom. In FlexAct, for example, a layer chooses from a finite set \(\{\sigma^{(1)},\dots,\sigma^{(p)}\}\), while in other work the activation is treated as a layer-wise hyperparameter, a global function evolved by a genetic algorithm, or a learnable basis expansion [2601.06441], [1811.03980], [1808.00783].

The literature also differs in granularity. FlexAct is layer-wise and, in the reported experiments, applied primarily to the penultimate layer; SmartMixed is per-neuron; RAFT uses one rational activation per Transformer layer; DeepLABNet supports channel-wise, layer-wise, or global sharing; and S-KAN performs per-node activation-space selection in Kolmogorov-Arnold networks [2601.06441], [2510.22450], [2208.14111], [1911.09257], [2408.08338]. This distribution of granularities indicates that “activation selection” is not a single mechanism but a family of mechanisms defined by what is being selected, at what scale, and by what optimizer.

## 2. Discrete differentiable selection

FlexAct is a canonical discrete selection mechanism. At layer \(i\), it computes a pre-activation \(h_i=W_iX_{i-1}+b_i\) and then forms
$$
X_i=\sum_{j=1}^{p} p_i^{(j)}\sigma^{(j)}(h_i),
$$
where the selection probabilities \(p_i^{(j)}\) are obtained from trainable logits through a Gumbel-Softmax relaxation. As \(\tau\) is annealed, the distribution becomes sharply peaked, and the model transitions from exploring blends to committing to specific activations. FlexAct also allows a straight-through estimator, so that the forward pass uses a one-hot argmax selection while gradients flow through the relaxed probabilities. In the reported synthetic regression experiments, the candidate set contains ReLU, Sigmoid, Tanh, Leaky ReLU, and Identity, and a gradient-norm-based KL regularizer is introduced to counter a scale-induced bias toward unbounded activations [2601.06441].

SmartMixed implements a related but finer-grained mechanism. Each neuron has logits over six candidates—ReLU, Sigmoid, Tanh, Leaky ReLU, ELU, and SELU—and Phase 1 uses a hard Gumbel-Softmax with a straight-through estimator so that each neuron effectively samples one activation in the forward pass. Phase 2 freezes each neuron’s activation to the argmax choice and continues training with a fixed mixed-activation network. On the representative MNIST architecture \([784,768,512,512,256,256,128,10]\), the reported accuracy rises from \(95.73\%\) test accuracy in Phase 1 to \(98.03\%\) in Phase 2, while the learned distributions show that early layers prefer ReLU and Leaky ReLU and deeper layers prefer ELU and SELU; Sigmoid is consistently avoided across layers [2510.22450].

These discrete approaches preserve a strong form of interpretability. The final network uses identifiable standard functions rather than an opaque learned nonlinearity, which is precisely the design rationale emphasized by FlexAct’s “Why learn when you can pick?” framing [2601.06441].

## 3. Search-based and evolutionary selection

Not all learnable activation selection is differentiable. A 2018 methodology for hybrid deep neural networks treats layer-wise activation choice and dropout rate as a structured search problem over \(\{\mathrm{ReLU},\mathrm{ELU},\mathrm{SELU}\}\). Its central device is the Evaluation Point, defined as the first epoch at which the Accuracy Gradient falls below \(0.1\%\). This allows early termination of candidate trials and yields reported Training Time Reduction of \(4\times\)–\(7\times\) during exploration, while producing \(7\%\) to \(15\%\) Relative Error Reduction on MNIST, CIFAR-10, and CIFAR-100 [1811.03980].

“The Quest for the Golden Activation Function” takes a different route and treats the activation itself as an evolutionary object. Candidate activations are piecewise functions with separate left and right branches, built from primitives such as ELiSH, HardELiSH, Swish, ReLU, ELU, SeLU, Softplus, HardSigmoid, Sigmoid, \(\sin\), and Linear, together with operations such as \(+\), \(-\), \(\times\), \(/\), \(\min\), \(\max\), composition, and exponentiation. The reported genetic algorithm uses population size 40 and 8 generations, and it shows that different datasets and architectures favor different activation shapes: HardELiSH and Swish-like forms are strong on CIFAR-10, while CIFAR-100 and Tiny ImageNet often favor different composites and ELiSH variants [1808.00783].

These search-based mechanisms are learnable in a data-driven sense, but the optimization variable is architectural rather than gradient-embedded. This suggests that activation selection spans a continuum from end-to-end differentiable routing to outer-loop architecture search.

## 4. Learnable activation families and basis expansions

A second major line of work does not choose among a finite menu but instead learns the activation family itself. In RAFT, each Transformer feed-forward layer replaces GELU with a Rational Activation Function
$$
F(x)=\frac{\sum_{j=0}^{m} a_j x^j}{1+\left|\sum_{k=0}^{n} b_k x^k\right|},
$$
with \(m=5\) and \(n=4\). The resulting model adds only 108 parameters to a 12-layer BERT-base-style encoder, lowers validation perplexity from \(5.18\) to \(5.00\), improves GLUE by \(5.71\) points on average in the 100-example regime, and improves SQuAD by \(2.05\) points in the full-data setting. The learned rational functions differ substantially across layers and often look very different from conventional activations [2208.14111].

DeepLABNet pushes this further by replacing static activations with per-feature radial-basis expansions,
$$
f(x)=\sum_{i=1}^{s}\lambda_i \phi(|x-c_i|)+v_0x+v_1,
$$
using polyharmonic spline bases with \(s=3\) and a hockey-stick initialization. The reported results show \(98.9\%\) on MNIST, \(91.5\%\) on CIFAR-10, and \(66.3\%\) on CIFAR-100, compared with ReLU baselines of \(97.9\%\), \(90.6\%\), and \(65.4\%\), respectively. The method demonstrates that fully learnable activation basis functions can be made tractable if the basis is scalar per feature rather than multivariate over the entire input [1911.09257].

Other basis-driven formulations generalize the same idea. RFLAF parameterizes the activation as \(\sigma_\theta(z)=\sum_i a_i B_i(z)\) using Gaussian basis functions and proves that the model can represent a broad class of random feature models whose activations lie in \(C_c(\mathbb{R})\), while requiring only about twice the parameter number of a conventional random feature model [2411.19468]. A separate orthogonal-basis line develops Hermite, Fourier, and tropical activations with variance-preserving initialization, and reports that such activations can train GPT-2 on OpenWebText and ConvNeXt on ImageNet without additional clamping mechanisms [2502.01247].

S-KAN extends learnable selection to activation spaces rather than single functions. Each KAN node initially computes
$$
\mathrm{Output}_{\langle \mathrm{node}\rangle}(x_i)=\sum_{p=1}^{m} w_p\,\phi_i^{(K_p)}(x_i),
$$
with candidate activation families drawn from a pool that includes B-splines, RBFs, Chebyshev polynomials, wavelets, and related KAN variants. Training alternates between soft weighting and pruning until one activation space remains at each node. The reported S-KAN and S-ConvKAN results show improved function fitting and image classification relative to fixed-space KAN baselines [2408.08338].

## 5. Empirical patterns and interpretability

Across these papers, one recurrent finding is that activation preference is not uniform across depth, architecture, or task. SmartMixed finds rectifier-heavy early layers and ELU/SELU-heavy deeper layers on MNIST [2510.22450]. RAFT finds that learned rational activations vary markedly by layer, and in some top layers the RAF output is close to zero across much of the input range, which the authors relate to the possibility that the residual path dominates there [2208.14111]. DeepLABNet visualizes control-point motion and progressive deformation away from the initial hockey-stick shape [1911.09257]. S-KAN reports different activation-space choices at different nodes for different function-fitting tasks [2408.08338].

A parallel empirical pattern is that activation learning can improve not only accuracy but also robustness. SPLASH, a continuous grounded piecewise-linear activation with symmetric hinges and data-derived hinge locations, reports that commonly-used architectures can be up to \(31\%\) more robust to adversarial attacks by simply using SPLASH units instead of ReLUs, while also improving accuracy across MNIST, CIFAR-10, and CIFAR-100 [2006.08947]. Tangma, defined as
$$
\mathrm{Tangma}(x)=x\tanh(x+\alpha)+\gamma x,
$$
learns \(\alpha\) and \(\gamma\) jointly with the network and reaches \(99.09\%\) validation accuracy on MNIST and \(78.15\%\) on CIFAR-10, with the learned parameters converging to approximately \(\alpha\approx0.28,\gamma\approx0.135\) on MNIST and \(\alpha\approx0.40,\gamma\approx0.38\) on CIFAR-10 [2507.10560]. RFLAF adds an interpretable dimension by allowing the learned activation to be read directly from the basis coefficients \(\{a_i\}\) [2411.19468].

These results collectively weaken the assumption that one globally fixed activation is structurally adequate. The papers do not claim a universal replacement rule; rather, they repeatedly show task- and layer-dependent nonlinear preferences.

## 6. Limitations, misconceptions, and future directions

A common misconception is that learnable activation selection must be end-to-end differentiable. The literature shows otherwise: differentiable routing via Gumbel-Softmax is one option, but layer-wise discrete search with early stopping and evolutionary search over compositional activation grammars are also effective automated mechanisms [2601.06441], [1811.03980], [1808.00783]. A second misconception is that activation learning always means inventing a new opaque function. FlexAct, SmartMixed, and S-KAN emphasize discrete or pruned selection from known families, precisely to preserve interpretability [2601.06441], [2510.22450], [2408.08338].

The technical limitations are also consistent across papers. Discrete-selection methods depend on a predefined candidate set and can miss an optimal nonlinearity outside that set; FlexAct explicitly notes this dependence and also reports sensitivity to temperature and to heuristic regularization [2601.06441]. Search-based methods can reduce exploration cost, but they still require many truncated training runs and may not scale cleanly to very deep or very large models [1811.03980]. Activation-space approaches such as S-KAN and orthogonal-basis activations improve flexibility at the price of extra compute and implementation complexity [2408.08338], [2502.01247]. RAFT and DeepLABNet show that learnable families can remain lightweight, but even there stability depends on careful initialization and regularization [2208.14111], [1911.09257].

Future directions stated in the papers are likewise convergent: multi-layer activation selection beyond a single penultimate layer, more principled regularizers grounded in information theory or smoothness, per-channel or per-neuron extensions in larger architectures, and broader evaluation in CNNs, Transformers, and other large-scale settings [2601.06441], [1808.00783], [2510.22450], [2208.14111]. A plausible implication is that activation selection will increasingly be treated as part of neural architecture search, conditional computation, and interpretable model design rather than as an isolated activation-engineering problem.

Source: https://www.emergentmind.com/topics/learnable-activation-selection-mechanism