---
title: Trainable Rational Activations
url: https://www.emergentmind.com/topics/trainable-rational-activations
type: topic
---

# Trainable Rational Activations

Trainable rational activations are learnable nonlinearities in which the scalar activation is parameterized as a ratio of numerator and denominator expansions and optimized jointly with the rest of a neural network. In the standard formulation, a scalar input \(x\) is mapped by a rational function such as
\[
r(x)=\frac{P(x)}{Q(x)}=\frac{\sum_{i=0}^{r_P} a_i x^i}{\sum_{j=0}^{r_Q} b_j x^j},
\]
with trainable coefficients in both numerator and denominator; practical implementations differ mainly in how they avoid poles, how they initialize the coefficients, and whether the same rational is shared across units, layers, groups, or higher-order modules [2004.01902]. Within this literature, trainable rationals are used both as drop-in replacements for fixed activations in ordinary MLP, CNN, RL, and Transformer pipelines and as structured nonlinear primitives in architectures such as joint-rational Atari agents, RAF-based Transformers, Rational-ANOVA Networks, and parameter-efficient activation adapters [2102.09407][2208.14111][2602.04006][2509.13240].

## 1. Formal definitions and taxonomic boundaries

A central formulation, used explicitly in deep reinforcement learning, defines a rational activation of order \((m,n)\) by
\[
R(x)=\frac{\sum_{j=0}^{m} a_j x^j}{1+\sum_{k=1}^{n} b_k x^k},
\]
so that the activation contributes \(m+1+n\) learned coefficients; in the Atari experiments of "Adaptive Rational Activations to Boost Deep Reinforcement Learning," the choice \((m,n)=(5,4)\) gives \(10\) learned coefficients per layer [2102.09407]. "Rational neural networks" adopts the closely related type-\((r_P,r_Q)\) notation and treats the numerator and denominator coefficients as trainable parameters optimized together with weights and biases; in its experiments the common choice is a type \((3,2)\) rational, adding \(7\) activation coefficients per layer when one activation is shared within a layer [2004.01902].

The literature also contains orthogonal-basis variants. "Orthogonal-Padé Activation Functions" replaces the monomial basis by orthogonal polynomial families and uses the safe orthogonal-Padé form
\[
G(x)=\frac{\sum_{i=0}^{k} c_i f_i(x)}{1+\sum_{j=1}^{l}|d_j|\,|f_j(x)|},
\]
with learnable coefficients \(c_i,d_j\) and basis functions \(f_i\) drawn from Chebyshev, Laguerre, Legendre, or Hermite systems; the paper’s recommended activations, HP-1 and HP-2, are safe Hermite-Padé variants [2106.09693].

Not every paper employing rational nonlinearities studies end-to-end trainable activation coefficients. "Rational activation functions in neural networks with uniform based loss functions and its application in classification" uses the fixed degree-\((1,1)\) activation
\[
R(x)=\frac{a_0+a_1x}{b_0+b_1x},
\]
but the coefficients \(a_0,a_1,b_0,b_1\) are chosen beforehand from the best uniform rational approximation to ReLU and are not optimized during network training; only the network weights and bias are trained [2111.02602]. This fixed-coefficient setting is therefore distinct from the main trainable-rational literature.

A second boundary concerns broader ratio-based activation families. "REAct: Rational Exponential Activation for Better Learning and Generalization in PINNs" introduces
\[
REAct(x)=\frac{1-\exp(ax+b)}{1+\exp(cx+d)},
\]
with four learnable shape parameters. This is a trainable parametric activation and a ratio involving exponentials, but it is not a rational function in the strict polynomial-over-polynomial sense [2503.02267]. The distinction matters because most approximation-theoretic and stability results in the rational-activation literature are stated specifically for polynomial ratios.

## 2. Stability mechanisms, parameterization choices, and optimization

A persistent practical issue is pole avoidance. In Atari RL, the raw denominator is not used directly; instead the paper adopts a safe rational form,
\[
R_{\mathrm{safe}}(x)=\frac{\sum_{j=0}^{m} a_j x^j}{\left|\,1+\sum_{k=1}^{n} b_k x^k\,\right|},
\]
which the authors state “avoids poles,” stabilizes training, and makes the function continuous. No extra normalization beyond the fixed constant term \(b_0=1\) is introduced, and no explicit coefficient constraints or projection steps are reported beyond the safe-denominator trick [2102.09407]. RAFT in Transformers and NoRA in PEFT adopt closely related safe PAU-style parameterizations with denominator \(1+\left|\sum b_k x^k\right|\), again making denominator collapse structurally difficult [2208.14111][2509.13240].

Other works impose stronger positivity by construction. Rational-ANOVA Networks define the scalar denominator as
\[
d_i(x)=1+\mathrm{softplus}\!\left(\sum_{b=1}^{n}\beta_{i,b}x^b\right)+\varepsilon,
\]
so that \(d_i(x)\ge 1+\varepsilon\) and the unit is pole-free on \(\mathbb{R}\). The same paper adds residual-style gating,
\[
r(x)=x+\alpha\cdot(\tilde r(x)-x),
\]
and analyzes the Jacobian
\[
\frac{\partial r}{\partial x}=(1-\alpha)+\alpha\,\frac{\partial \tilde r}{\partial x},
\]
using small initial \(\alpha\) and identity-like initialization as a safe-start mechanism for deep composition [2602.04006]. By contrast, "Rational neural networks" explicitly does not use a safe denominator in its own experiments and imposes no hard denominator positivity constraint during learning; it relies mainly on initialization near the best rational approximation of ReLU on \([-1,1]\) and notes that bad initialization may create poles on the real axis [2004.01902].

Optimization is usually standard joint gradient-based learning rather than a specialized outer loop. In Atari DQN and Rainbow, rational coefficients are “optimised together with the rest of the weights” under Adam with learning rate \(2.5\times 10^{-4}\) [2102.09407]. RAFT pretraining uses AdamW with separate learning rates \(7\times 10^{-4}\) for model parameters and \(5\times 10^{-3}\) for RAF coefficients, warms both for the first \(1\%\) of steps, then linearly decays only the model rate while keeping the RAF rate constant; it also uses DeepNorm initialization and initializes RAFs to approximate GELU [2208.14111]. Orthogonal-Padé activations are initialized by least-squares fitting to Leaky ReLU with \(\alpha=0.01\), using \(k=5\) and \(l=4\), so that each activation-bearing layer adds \(9\) learnable parameters [2106.09693].

Recent work increasingly treats stabilization as a structural design problem rather than as mere coefficient tuning. "Balancing Expressivity and Robustness: Constrained Rational Activations for Reinforcement Learning" identifies activation explosion in unconstrained rationals and proposes
\[
f(x)=\frac{a_nx^n + a_{n-1}x^{n-1} + \dots + a_1x}{|b_mx^m| + |b_{m-1}x^{m-1}| + \dots + |b_1x| + 1 + \left|\frac{x}{c}\right|^d},
\qquad d>n,
\]
with the constant numerator term removed so that \(f(0)=0\). In experiments the paper uses \((n,m)=(3,2)\) and \(d=n+1\), interpreting the added denominator tail as an internal regularizer that forces asymptotic decay and suppresses excessive output scaling [2507.14736]. NoRA pursues a different structural route: it freezes the base rational activation and learns low-rank perturbations to numerator and denominator coefficients,
\[
\Delta P_g=A_g^PB_g^P,\qquad \Delta Q_g=A_g^QB_g^Q,
\]
with zero-start initialization \(A_g\sim\mathcal N(0,0.02)\), \(B_g=0\), so that the initial activation exactly matches the pretrained one [2509.13240].

## 3. Expressivity results and theoretical interpretations

The strongest approximation-theoretic arguments come from "Rational neural networks" and "Rational Neural Networks have Expressivity Advantages." The former proves a bidirectional simulation theorem: if \(R:[-1,1]^d\to[-1,1]\) is a rational network with \(M\) layers and at most \(k\) nodes per layer, then there exists a ReLU network of size
\[
\mathcal{O}\left(kM\log(ML^M/\epsilon)^3\right)
\]
approximating it to accuracy \(\epsilon\); conversely, if \(f:[-1,1]^d\to[-1,1]\) is a ReLU network with \(M\) layers and at most \(k\) nodes per layer, then there exists a rational network of size
\[
\mathcal{O}(kM\log(\log(M/\epsilon)))
\]
with the same approximation tolerance [2004.01902]. The same paper further proves that for the Sobolev class
\[
F_{d,n}:=\{f\in \mathcal{W}^{n,\infty}([0,1]^d),\ \|f\|_{\mathcal{W}^{n,\infty}([0,1]^d)}\leq 1\},
\]
there exists a rational neural network of size
\[
\mathcal{O}(\epsilon^{-d/n}\log(\log(1/\epsilon)))
\]
and maximum depth
\[
\mathcal{O}(\log(\log(1/\epsilon)))
\]
achieving uniform error \(\epsilon\), compared with ReLU constructions whose size and depth include \(\log(1/\epsilon)\) rather than \(\log\log(1/\epsilon)\) factors [2004.01902].

The 2026 expressivity paper extends this line to modern activation families and transformer-style nonlinearities. For GELU on \([-1,1]\), it states that there exists a rational neural network \(R\) of size
\[
\mathcal{O}\bigl(\log^3\log(1/\varepsilon)\bigr)
\]
such that \(\|R-G\|_{L^\infty([-1,1])}\le \varepsilon\), while no rational neural network of size \(o(\log\log(1/\varepsilon))\) can achieve this accuracy. In the reverse direction, it constructs rational targets for which any bounded-parameter GELU network achieving uniform error \(\varepsilon\) must have size at least
\[
\Omega(\log(1/\varepsilon)).
\]
At network level, it gives
\[
\mathcal{O}\bigl(kM\log^3\log(1/\varepsilon)\bigr)
\]
for rational approximation of \(k\)-by-\(M\) GELU networks, and an \(\Omega(kM\log(1/\varepsilon))\) lower bound in the worst-case converse direction [2602.12390]. Within the scope of the paper, this is presented as an exponential gap in approximation efficiency between low-degree trainable rationals and a broad class of fixed smooth or piecewise-linear activations.

A separate theoretical theme is residual representability. In Atari RL, the paper proves:
\[
\textbf{Theorem: } \text{A rational function }R\text{ of order }(m,n)\text{ embeds a residual connection } \Longleftrightarrow m>n.
\]
Equivalently, \(R(X)+X\) is again rational, with transformed order \((m',n')\) satisfying \(m'=\max(m,n+1)\), \(n'=n\). The authors interpret this as an implicit identity-plus-transform capacity and motivate the experimental choice \((m,n)=(5,4)\) precisely because \(m>n\) gives this residual-closure property [2102.09407].

Function-space arguments also appear outside conventional supervised learning. In the BNN–GP inverse problem, the induced covariance of a wide BNN layer is
\[
\mathrm{Cov}(f^l(x),f^l(x'))={\sigma_b^l}^2+{\sigma_w^l}^2\mathbb E[\phi(f_j^0(x))\phi(f_j^0(x'))],
\]
so the activation function directly shapes the kernel. That paper explicitly explores Rational (Padé) activations as one learnable family among several and finds that learning activations jointly with prior variances improves fidelity to target GP priors, whereas learning activations alone is generally not sufficient for good fits [2410.15777]. This suggests that trainable rationals can act not only as local nonlinearities but also as function-space prior parameters.

## 4. Architectural roles: from pointwise nonlinearities to structured modules

The simplest deployment is layerwise substitution: wherever a conventional nonlinearity would be placed, a rational unit is inserted instead. In Atari DQN-style networks, the activations separate conv1, conv2, conv3, and the hidden fully connected layer of the classic Nature CNN, and the standard setting gives one order-\((5,4)\) rational per activation layer [2102.09407]. "Rational neural networks" similarly uses one shared type-\((3,2)\) rational activation per layer rather than one per neuron to reduce trainable-parameter count and computational expense [2004.01902]. Orthogonal-Padé activations also follow this layer-shared regime [2106.09693].

A more regularized variant is coefficient sharing across layers. In the RL paper, the ordinary per-layer setting is contrasted with the joint-rational construction, in which multiple layers are “always activated by the same learnable rational activation function.” The regularization here is entirely architectural: there is no extra penalty such as \(\lambda\|a\|^2+\mu\|b\|^2\), only parameter tying across layers [2102.09407]. This paper also explicitly corrects a terminological misunderstanding: the regularized variant is called joint-rational, not recurrent-rational, and there is no temporal recurrence or hidden-state update in the activation itself [2102.09407].

Transformer work introduces additional sharing granularities. RAFT replaces the FFN nonlinearity in each BERT layer by a layer-specific rational function and keeps one RAF shared within that layer; in a 12-layer model this amounts to \(108\) additional parameters total, with one RAF per encoder layer and an additional RAF-like component in the pooling layer for the “only tune RAFs” setting [2208.14111]. NoRA moves from layerwise sharing to group-wise sharing over the hidden dimension: each transformer MLP partitions channels into \(G\) disjoint groups, all neurons within a group share one rational activation, and low-rank coefficient updates are learned separately for numerator and denominator in each group [2509.13240].

The architecture can also be organized around rational modules rather than around pointwise activations. Rational-ANOVA Networks write
\[
f_\theta(x)=\sum_{i=1}^d r_i(x_i)+\sum_{(i,j)\in S} r_{ij}(x_i,x_j),
\]
with learnable 1D rational main effects and sparse 2D rational interaction modules; only in the RAN-FFN variant,
\[
\mathrm{RAN\text{-}FFN}(h)=W_2\,r(W_1h)+b,
\]
do the rational units behave directly like conventional learnable activations [2602.04006]. In certified control, the departure from standard activation placement is even stronger: rational activations such as
\[
\mathrm{Rtanh}(x)=\frac{4x}{x^2+4},\qquad \mathrm{Rsig}(x)=\frac{(x+4)^2}{2(x^2+16)}
\]
are mainly valuable because their input–output relations admit exact polynomial equalities for Sum of Squares reasoning, and the paper ultimately advocates a general rational-controller parameterization that is convex in the controller coefficients rather than a backprop-trained activation layer [2307.06287].

## 5. Empirical record across reinforcement learning, language modeling, vision, PINNs, and PEFT

In deep reinforcement learning, the original Atari study reports unusually consistent gains. On its 15-game DQN benchmark, standard rational DQN achieves the best mean score on \(6/15\) games, joint-rational DQN on \(9/15\), both rational variants reach \(11/15\) superhuman scores, and all rigid baselines—Leaky ReLU DQN, SiLU DQN, dSiLU+SiLU DQN, and Leaky ReLU DDQN—achieve \(0/15\) wins in best mean-score count. The paper further states that DQN with rational plasticity outperforms rigid DDQN on every considered Atari game, that tempered rationals frozen to a previously learned shape underperform fully trainable rationals in dynamic environments, and that rational plasticity can reduce Q-learning overestimation “as much or further than” rigid DDQN on several games [2102.09407]. Later continuous-control work complicates this picture: on DeepMind Control Suite and MetaWorld, unconstrained rationals can suffer activation explosion, overestimation, and instability under high UTD, whereas constrained rationals improve stability and performance; the paper also reports that preliminary Atari 100k experiments did not show the same instability, suggesting that the expressivity–robustness trade-off is especially acute in continuous control [2507.14736].

In Transformers, RAFT shows that activation learning is beneficial even when the rest of the model is unchanged. During BERT-style pretraining, RAFT improves validation MLM loss from \(1.645\) to \(1.611\) and perplexity from \(5.18\) to \(5.00\). On GLUE, the 100-example setting rises from \(48.07\) for the fixed-activation baseline to \(53.38\) for RAFT\(^\text{full}\) and \(53.78\) for RAFT\(^\text{fixed}\); on full-data SQuAD, F1 improves from \(72.33\) to \(74.45\) for RAFT\(^\text{full}\) [2208.14111]. The same paper observes that learned RAF shapes vary substantially across layers and tasks, often differ markedly from GELU, and sometimes nearly suppress the FFN branch in upper layers. In parameter-efficient tuning, NoRA extends the activation-learning idea to frozen-weight transformers: on ViT-Tiny, pure NoRA updates only \(0.02\)M parameters, or \(0.4\%\) of the model, and still improves over full fine-tuning by \(+0.17\%\) on CIFAR-10 and \(+0.27\%\) on CIFAR-100; NoRA++ further edges out LoRA and DoRA under matched budgets. On LLaMA3-8B instruction tuning, NoRA++ yields average MMLU gains of \(+0.3\%\) to \(+0.8\%\), including \(+1.6\%\) on STEM for Alpaca and \(+1.3\%\) on OpenOrca [2509.13240].

In supervised approximation and scientific learning, trainable rationals have repeatedly shown strong low-parameter gains. "Rational neural networks" reports on KdV interpolation that a trainable rational \((3,2)\) network reaches \(\mathrm{MSE}=1.2\times 10^{-7}\), compared with \(1.9\times 10^{-4}\) for ReLU, \(3.3\times 10^{-6}\) for sinusoid, and \(3.6\times 10^{-5}\) for a trainable cubic polynomial, with only \(35\) extra parameters beyond an \(8000\)-parameter ReLU baseline [2004.01902]. The same paper also shows that rational activations are feasible in GANs but notes mode collapse for digit \(1\), presenting this as a caution that adversarial training may require additional stabilization [2004.01902]. In PINNs, REAct is not a polynomial rational activation, but it provides adjacent evidence that ratio-based learnable nonlinearities can materially improve PDE training: on the heat equation, REAct attains \(\mathrm{MSE}=2.09\times10^{-7}\) versus \(0.0001\) for tanh and \(0.0001\) for STan, and it also improves inverse-problem noise robustness across varying noise levels [2503.02267].

In image classification and related vision tasks, safe Hermite-Padé activations show large and consistent improvements. The OPAU paper reports that HP-1 and HP-2 improve top-1 accuracy over ReLU by \(5.06\%\) and \(4.63\%\) respectively in PreActResNet-34 on CIFAR100, by \(3.02\%\) and \(2.75\%\) respectively in MobileNet V2 on CIFAR100, and by \(2.02\%\) and \(1.78\%\) in PreActResNet-34 on CIFAR10; the summary table states that HP-1 and HP-2 outperform ReLU, Leaky ReLU, ELU, Softplus, and PAU in \(22/22\) tested networks and Swish in \(21/22\) [2106.09693]. More structured rational architectures preserve this empirical competitiveness. Rational-ANOVA Networks report, for example, CIFAR-10 at \(1.0\)M parameters with KAF \(56.95\), MLP \(56.45\), KAN \(43.32\), and RAN \(59.05\), and on ViT-Tiny / ImageNet-1K they report \(74.2\%\) top-1 for RAN-FFN versus \(72.3\%\) for GELU and \(73.2\%\) for KAF under matched parameters and FLOPs [2602.04006].

## 6. Misconceptions, limitations, and unresolved design questions

A first recurring misconception is taxonomic. Some papers on “rational activations” do not in fact learn activation coefficients during network training. The classification paper based on uniform loss is explicit that \(a_0,a_1,b_0,b_1\) are fixed beforehand from best uniform rational approximation and that making them learnable would destroy the quasiconvex structure on which its bisection and differential-correction solvers rely [2111.02602]. A second misconception is terminological: in the Atari RL paper, the regularized shared-coefficient construction is joint-rational, not recurrent-rational, and no temporal recurrence should be inferred from the activation itself [2102.09407].

A second limitation concerns incomplete optimization detail. The Atari paper does not provide explicit gradient formulas, exact RL rational initialization coefficients, or more detailed stabilization than the safe denominator; reproduction depends partly on released code [2102.09407]. RAFT likewise notes that rational approximation is reliable only on a bounded interval and warns that swapping RAF into an already pretrained model only at fine-tuning time may diverge because inputs may fall outside the approximation range [2208.14111]. "Rational neural networks" states that exact rational activations are tricky to train because poles can appear during learning, and its experiments rely heavily on careful ReLU-like initialization rather than on explicit admissibility constraints [2004.01902].

A third issue is domain-dependent instability. The continuous-control study shows that original rationals can exacerbate overestimation in high-UTD RL and that constrained rationals may trade away some long-horizon plasticity in continual learning; LayerNorm does not solve the problem and can even induce instability where none previously existed [2507.14736]. A related, but differently framed, observation appears in the 2026 expressivity paper: normalization layers may be specifically harmful for trainable rationals because affine normalization and rational coefficients are not separately identifiable, producing flat directions and poor conditioning; the paper backs this claim empirically with no-normalization ViT results in which rational activations succeed while fixed activations fail [2602.12390]. This suggests that trainable rationals interact with standard optimization scaffolding in ways that are not yet fully normalized across architectures.

Finally, activation learning does not automatically substitute for broader modeling choices. In the BNN–GP setting, learning activations alone is generally not sufficient for good prior matching; the best results come from learning activations jointly with prior variances [2410.15777]. In RL, the Atari paper explicitly identifies degree selection as future work, since its empirical study fixes \((m,n)=(5,4)\) throughout [2102.09407]. In PEFT, NoRA shows that direct coefficient tuning, zero-initialized tuning, GELU-initialized tuning, and constant-only tuning all underperform structured low-rank coefficient updates, implying that how rational coefficients are parameterized can matter as much as whether they are trainable at all [2509.13240]. A plausible implication is that future work will center less on the abstract question of whether rational activations are expressive and more on which constrained, shared, grouped, or gated parameterizations deliver the best trade-off among stability, plasticity, compute overhead, and architectural fit.

Source: https://www.emergentmind.com/topics/trainable-rational-activations