Trainable Rational Activations

• Trainable rational activations are learnable nonlinear functions defined as a ratio of polynomial expansions, offering enhanced expressivity and stability in neural networks.
• They incorporate stabilization techniques like safe denominators, careful initialization, and coefficient sharing to avoid poles and improve convergence.
• Empirical studies show that these activations boost performance across domains including reinforcement learning, Transformers, vision tasks, and scientific computing.

Trainable rational activations are learnable nonlinearities in which the scalar activation is parameterized as a ratio of numerator and denominator expansions and optimized jointly with the rest of a neural network. In the standard formulation, a scalar input $x$ is mapped by a rational function such as

$$r(x)=\frac{P(x)}{Q(x)}=\frac{\sum_{i=0}^{r_P} a_i x^i}{\sum_{j=0}^{r_Q} b_j x^j},$$

with trainable coefficients in both numerator and denominator; practical implementations differ mainly in how they avoid poles, how they initialize the coefficients, and whether the same rational is shared across units, layers, groups, or higher-order modules (Boullé et al., 2020). Within this literature, trainable rationals are used both as drop-in replacements for fixed activations in ordinary MLP, CNN, RL, and Transformer pipelines and as structured nonlinear primitives in architectures such as joint-rational Atari agents, RAF-based Transformers, Rational-ANOVA Networks, and parameter-efficient activation adapters (Delfosse et al., 2021, Fang et al., 2022, Zhang et al., 3 Feb 2026, Yin et al., 16 Sep 2025).

1. Formal definitions and taxonomic boundaries

A central formulation, used explicitly in deep reinforcement learning, defines a rational activation of order $(m,n)$ by

$$R(x)=\frac{\sum_{j=0}^{m} a_j x^j}{1+\sum_{k=1}^{n} b_k x^k},$$

so that the activation contributes $m+1+n$ learned coefficients; in the Atari experiments of "Adaptive Rational Activations to Boost Deep Reinforcement Learning," the choice $(m,n)=(5,4)$ gives $10$ learned coefficients per layer (Delfosse et al., 2021). "Rational neural networks" adopts the closely related type-$(r_P,r_Q)$ notation and treats the numerator and denominator coefficients as trainable parameters optimized together with weights and biases; in its experiments the common choice is a type $(3,2)$ rational, adding $7$ activation coefficients per layer when one activation is shared within a layer (Boullé et al., 2020).

The literature also contains orthogonal-basis variants. "Orthogonal-Padé Activation Functions" replaces the monomial basis by orthogonal polynomial families and uses the safe orthogonal-Padé form

$$r(x)=\frac{P(x)}{Q(x)}=\frac{\sum_{i=0}^{r_P} a_i x^i}{\sum_{j=0}^{r_Q} b_j x^j},$$0

with learnable coefficients $$r(x)=\frac{P(x)}{Q(x)}=\frac{\sum_{i=0}^{r_P} a_i x^i}{\sum_{j=0}^{r_Q} b_j x^j},$$1 and basis functions $$r(x)=\frac{P(x)}{Q(x)}=\frac{\sum_{i=0}^{r_P} a_i x^i}{\sum_{j=0}^{r_Q} b_j x^j},$$2 drawn from Chebyshev, Laguerre, Legendre, or Hermite systems; the paper’s recommended activations, HP-1 and HP-2, are safe Hermite-Padé variants (Biswas et al., 2021).

Not every paper employing rational nonlinearities studies end-to-end trainable activation coefficients. "Rational activation functions in neural networks with uniform based loss functions and its application in classification" uses the fixed degree-$$r(x)=\frac{P(x)}{Q(x)}=\frac{\sum_{i=0}^{r_P} a_i x^i}{\sum_{j=0}^{r_Q} b_j x^j},$$3 activation

$$r(x)=\frac{P(x)}{Q(x)}=\frac{\sum_{i=0}^{r_P} a_i x^i}{\sum_{j=0}^{r_Q} b_j x^j},$$4

but the coefficients $$r(x)=\frac{P(x)}{Q(x)}=\frac{\sum_{i=0}^{r_P} a_i x^i}{\sum_{j=0}^{r_Q} b_j x^j},$$5 are chosen beforehand from the best uniform rational approximation to ReLU and are not optimized during network training; only the network weights and bias are trained (Peiris, 2021). This fixed-coefficient setting is therefore distinct from the main trainable-rational literature.

A second boundary concerns broader ratio-based activation families. "REAct: Rational Exponential Activation for Better Learning and Generalization in PINNs" introduces

$$r(x)=\frac{P(x)}{Q(x)}=\frac{\sum_{i=0}^{r_P} a_i x^i}{\sum_{j=0}^{r_Q} b_j x^j},$$6

with four learnable shape parameters. This is a trainable parametric activation and a ratio involving exponentials, but it is not a rational function in the strict polynomial-over-polynomial sense (Mishra et al., 4 Mar 2025). The distinction matters because most approximation-theoretic and stability results in the rational-activation literature are stated specifically for polynomial ratios.

2. Stability mechanisms, parameterization choices, and optimization

A persistent practical issue is pole avoidance. In Atari RL, the raw denominator is not used directly; instead the paper adopts a safe rational form,

$$r(x)=\frac{P(x)}{Q(x)}=\frac{\sum_{i=0}^{r_P} a_i x^i}{\sum_{j=0}^{r_Q} b_j x^j},$$7

which the authors state “avoids poles,” stabilizes training, and makes the function continuous. No extra normalization beyond the fixed constant term $$r(x)=\frac{P(x)}{Q(x)}=\frac{\sum_{i=0}^{r_P} a_i x^i}{\sum_{j=0}^{r_Q} b_j x^j},$$8 is introduced, and no explicit coefficient constraints or projection steps are reported beyond the safe-denominator trick (Delfosse et al., 2021). RAFT in Transformers and NoRA in PEFT adopt closely related safe PAU-style parameterizations with denominator $$r(x)=\frac{P(x)}{Q(x)}=\frac{\sum_{i=0}^{r_P} a_i x^i}{\sum_{j=0}^{r_Q} b_j x^j},$$9, again making denominator collapse structurally difficult (Fang et al., 2022, Yin et al., 16 Sep 2025).

Other works impose stronger positivity by construction. Rational-ANOVA Networks define the scalar denominator as

$(m,n)$0

so that $(m,n)$1 and the unit is pole-free on $(m,n)$2. The same paper adds residual-style gating,

$(m,n)$3

and analyzes the Jacobian

$(m,n)$4

using small initial $(m,n)$5 and identity-like initialization as a safe-start mechanism for deep composition (Zhang et al., 3 Feb 2026). By contrast, "Rational neural networks" explicitly does not use a safe denominator in its own experiments and imposes no hard denominator positivity constraint during learning; it relies mainly on initialization near the best rational approximation of ReLU on $(m,n)$6 and notes that bad initialization may create poles on the real axis (Boullé et al., 2020).

Optimization is usually standard joint gradient-based learning rather than a specialized outer loop. In Atari DQN and Rainbow, rational coefficients are “optimised together with the rest of the weights” under Adam with learning rate $(m,n)$7 (Delfosse et al., 2021). RAFT pretraining uses AdamW with separate learning rates $(m,n)$8 for model parameters and $(m,n)$9 for RAF coefficients, warms both for the first $$R(x)=\frac{\sum_{j=0}^{m} a_j x^j}{1+\sum_{k=1}^{n} b_k x^k},$$0 of steps, then linearly decays only the model rate while keeping the RAF rate constant; it also uses DeepNorm initialization and initializes RAFs to approximate GELU (Fang et al., 2022). Orthogonal-Padé activations are initialized by least-squares fitting to Leaky ReLU with $$R(x)=\frac{\sum_{j=0}^{m} a_j x^j}{1+\sum_{k=1}^{n} b_k x^k},$$1, using $$R(x)=\frac{\sum_{j=0}^{m} a_j x^j}{1+\sum_{k=1}^{n} b_k x^k},$$2 and $$R(x)=\frac{\sum_{j=0}^{m} a_j x^j}{1+\sum_{k=1}^{n} b_k x^k},$$3, so that each activation-bearing layer adds $$R(x)=\frac{\sum_{j=0}^{m} a_j x^j}{1+\sum_{k=1}^{n} b_k x^k},$$4 learnable parameters (Biswas et al., 2021).

Recent work increasingly treats stabilization as a structural design problem rather than as mere coefficient tuning. "Balancing Expressivity and Robustness: Constrained Rational Activations for Reinforcement Learning" identifies activation explosion in unconstrained rationals and proposes

$$R(x)=\frac{\sum_{j=0}^{m} a_j x^j}{1+\sum_{k=1}^{n} b_k x^k},$$5

with the constant numerator term removed so that $$R(x)=\frac{\sum_{j=0}^{m} a_j x^j}{1+\sum_{k=1}^{n} b_k x^k},$$6. In experiments the paper uses $$R(x)=\frac{\sum_{j=0}^{m} a_j x^j}{1+\sum_{k=1}^{n} b_k x^k},$$7 and $$R(x)=\frac{\sum_{j=0}^{m} a_j x^j}{1+\sum_{k=1}^{n} b_k x^k},$$8, interpreting the added denominator tail as an internal regularizer that forces asymptotic decay and suppresses excessive output scaling (Surdej et al., 19 Jul 2025). NoRA pursues a different structural route: it freezes the base rational activation and learns low-rank perturbations to numerator and denominator coefficients,

$$R(x)=\frac{\sum_{j=0}^{m} a_j x^j}{1+\sum_{k=1}^{n} b_k x^k},$$9

with zero-start initialization $m+1+n$0, $m+1+n$1, so that the initial activation exactly matches the pretrained one (Yin et al., 16 Sep 2025).

3. Expressivity results and theoretical interpretations

The strongest approximation-theoretic arguments come from "Rational neural networks" and "Rational Neural Networks have Expressivity Advantages." The former proves a bidirectional simulation theorem: if $m+1+n$2 is a rational network with $m+1+n$3 layers and at most $m+1+n$4 nodes per layer, then there exists a ReLU network of size

$m+1+n$5

approximating it to accuracy $m+1+n$6; conversely, if $m+1+n$7 is a ReLU network with $m+1+n$8 layers and at most $m+1+n$9 nodes per layer, then there exists a rational network of size

$(m,n)=(5,4)$0

with the same approximation tolerance (Boullé et al., 2020). The same paper further proves that for the Sobolev class

$(m,n)=(5,4)$1

there exists a rational neural network of size

$(m,n)=(5,4)$2

and maximum depth

$(m,n)=(5,4)$3

achieving uniform error $(m,n)=(5,4)$4, compared with ReLU constructions whose size and depth include $(m,n)=(5,4)$5 rather than $(m,n)=(5,4)$6 factors (Boullé et al., 2020).

The 2026 expressivity paper extends this line to modern activation families and transformer-style nonlinearities. For GELU on $(m,n)=(5,4)$7, it states that there exists a rational neural network $(m,n)=(5,4)$8 of size

$(m,n)=(5,4)$9

such that $10$0, while no rational neural network of size $10$1 can achieve this accuracy. In the reverse direction, it constructs rational targets for which any bounded-parameter GELU network achieving uniform error $10$2 must have size at least

$10$3

At network level, it gives

$10$4

for rational approximation of $10$5-by-$10$6 GELU networks, and an $10$7 lower bound in the worst-case converse direction (Tang et al., 12 Feb 2026). Within the scope of the paper, this is presented as an exponential gap in approximation efficiency between low-degree trainable rationals and a broad class of fixed smooth or piecewise-linear activations.

A separate theoretical theme is residual representability. In Atari RL, the paper proves: $10$8 Equivalently, $10$9 is again rational, with transformed order $(r_P,r_Q)$0 satisfying $(r_P,r_Q)$1, $(r_P,r_Q)$2. The authors interpret this as an implicit identity-plus-transform capacity and motivate the experimental choice $(r_P,r_Q)$3 precisely because $(r_P,r_Q)$4 gives this residual-closure property (Delfosse et al., 2021).

Function-space arguments also appear outside conventional supervised learning. In the BNN–GP inverse problem, the induced covariance of a wide BNN layer is

$(r_P,r_Q)$5

so the activation function directly shapes the kernel. That paper explicitly explores Rational (Padé) activations as one learnable family among several and finds that learning activations jointly with prior variances improves fidelity to target GP priors, whereas learning activations alone is generally not sufficient for good fits (Sendera et al., 2024). This suggests that trainable rationals can act not only as local nonlinearities but also as function-space prior parameters.

4. Architectural roles: from pointwise nonlinearities to structured modules

The simplest deployment is layerwise substitution: wherever a conventional nonlinearity would be placed, a rational unit is inserted instead. In Atari DQN-style networks, the activations separate conv1, conv2, conv3, and the hidden fully connected layer of the classic Nature CNN, and the standard setting gives one order-$(r_P,r_Q)$6 rational per activation layer (Delfosse et al., 2021). "Rational neural networks" similarly uses one shared type-$(r_P,r_Q)$7 rational activation per layer rather than one per neuron to reduce trainable-parameter count and computational expense (Boullé et al., 2020). Orthogonal-Padé activations also follow this layer-shared regime (Biswas et al., 2021).

A more regularized variant is coefficient sharing across layers. In the RL paper, the ordinary per-layer setting is contrasted with the joint-rational construction, in which multiple layers are “always activated by the same learnable rational activation function.” The regularization here is entirely architectural: there is no extra penalty such as $(r_P,r_Q)$8, only parameter tying across layers (Delfosse et al., 2021). This paper also explicitly corrects a terminological misunderstanding: the regularized variant is called joint-rational, not recurrent-rational, and there is no temporal recurrence or hidden-state update in the activation itself (Delfosse et al., 2021).

Transformer work introduces additional sharing granularities. RAFT replaces the FFN nonlinearity in each BERT layer by a layer-specific rational function and keeps one RAF shared within that layer; in a 12-layer model this amounts to $(r_P,r_Q)$9 additional parameters total, with one RAF per encoder layer and an additional RAF-like component in the pooling layer for the “only tune RAFs” setting (Fang et al., 2022). NoRA moves from layerwise sharing to group-wise sharing over the hidden dimension: each transformer MLP partitions channels into $(3,2)$0 disjoint groups, all neurons within a group share one rational activation, and low-rank coefficient updates are learned separately for numerator and denominator in each group (Yin et al., 16 Sep 2025).

The architecture can also be organized around rational modules rather than around pointwise activations. Rational-ANOVA Networks write

$(3,2)$1

with learnable 1D rational main effects and sparse 2D rational interaction modules; only in the RAN-FFN variant,

$(3,2)$2

do the rational units behave directly like conventional learnable activations (Zhang et al., 3 Feb 2026). In certified control, the departure from standard activation placement is even stronger: rational activations such as

$(3,2)$3

are mainly valuable because their input–output relations admit exact polynomial equalities for Sum of Squares reasoning, and the paper ultimately advocates a general rational-controller parameterization that is convex in the controller coefficients rather than a backprop-trained activation layer (Newton et al., 2023).

5. Empirical record across reinforcement learning, language modeling, vision, PINNs, and PEFT

In deep reinforcement learning, the original Atari study reports unusually consistent gains. On its 15-game DQN benchmark, standard rational DQN achieves the best mean score on $(3,2)$4 games, joint-rational DQN on $(3,2)$5, both rational variants reach $(3,2)$6 superhuman scores, and all rigid baselines—Leaky ReLU DQN, SiLU DQN, dSiLU+SiLU DQN, and Leaky ReLU DDQN—achieve $(3,2)$7 wins in best mean-score count. The paper further states that DQN with rational plasticity outperforms rigid DDQN on every considered Atari game, that tempered rationals frozen to a previously learned shape underperform fully trainable rationals in dynamic environments, and that rational plasticity can reduce Q-learning overestimation “as much or further than” rigid DDQN on several games (Delfosse et al., 2021). Later continuous-control work complicates this picture: on DeepMind Control Suite and MetaWorld, unconstrained rationals can suffer activation explosion, overestimation, and instability under high UTD, whereas constrained rationals improve stability and performance; the paper also reports that preliminary Atari 100k experiments did not show the same instability, suggesting that the expressivity–robustness trade-off is especially acute in continuous control (Surdej et al., 19 Jul 2025).

In Transformers, RAFT shows that activation learning is beneficial even when the rest of the model is unchanged. During BERT-style pretraining, RAFT improves validation MLM loss from $(3,2)$8 to $(3,2)$9 and perplexity from $7$0 to $7$1. On GLUE, the 100-example setting rises from $7$2 for the fixed-activation baseline to $7$3 for RAFT$7$4 and $7$5 for RAFT$7$6; on full-data SQuAD, F1 improves from $7$7 to $7$8 for RAFT$7$9 (Fang et al., 2022). The same paper observes that learned RAF shapes vary substantially across layers and tasks, often differ markedly from GELU, and sometimes nearly suppress the FFN branch in upper layers. In parameter-efficient tuning, NoRA extends the activation-learning idea to frozen-weight transformers: on ViT-Tiny, pure NoRA updates only $$r(x)=\frac{P(x)}{Q(x)}=\frac{\sum_{i=0}^{r_P} a_i x^i}{\sum_{j=0}^{r_Q} b_j x^j},$$00M parameters, or $$r(x)=\frac{P(x)}{Q(x)}=\frac{\sum_{i=0}^{r_P} a_i x^i}{\sum_{j=0}^{r_Q} b_j x^j},$$01 of the model, and still improves over full fine-tuning by $$r(x)=\frac{P(x)}{Q(x)}=\frac{\sum_{i=0}^{r_P} a_i x^i}{\sum_{j=0}^{r_Q} b_j x^j},$$02 on CIFAR-10 and $$r(x)=\frac{P(x)}{Q(x)}=\frac{\sum_{i=0}^{r_P} a_i x^i}{\sum_{j=0}^{r_Q} b_j x^j},$$03 on CIFAR-100; NoRA++ further edges out LoRA and DoRA under matched budgets. On LLaMA3-8B instruction tuning, NoRA++ yields average MMLU gains of $$r(x)=\frac{P(x)}{Q(x)}=\frac{\sum_{i=0}^{r_P} a_i x^i}{\sum_{j=0}^{r_Q} b_j x^j},$$04 to $$r(x)=\frac{P(x)}{Q(x)}=\frac{\sum_{i=0}^{r_P} a_i x^i}{\sum_{j=0}^{r_Q} b_j x^j},$$05, including $$r(x)=\frac{P(x)}{Q(x)}=\frac{\sum_{i=0}^{r_P} a_i x^i}{\sum_{j=0}^{r_Q} b_j x^j},$$06 on STEM for Alpaca and $$r(x)=\frac{P(x)}{Q(x)}=\frac{\sum_{i=0}^{r_P} a_i x^i}{\sum_{j=0}^{r_Q} b_j x^j},$$07 on OpenOrca (Yin et al., 16 Sep 2025).

In supervised approximation and scientific learning, trainable rationals have repeatedly shown strong low-parameter gains. "Rational neural networks" reports on KdV interpolation that a trainable rational $$r(x)=\frac{P(x)}{Q(x)}=\frac{\sum_{i=0}^{r_P} a_i x^i}{\sum_{j=0}^{r_Q} b_j x^j},$$08 network reaches $$r(x)=\frac{P(x)}{Q(x)}=\frac{\sum_{i=0}^{r_P} a_i x^i}{\sum_{j=0}^{r_Q} b_j x^j},$$09, compared with $$r(x)=\frac{P(x)}{Q(x)}=\frac{\sum_{i=0}^{r_P} a_i x^i}{\sum_{j=0}^{r_Q} b_j x^j},$$10 for ReLU, $$r(x)=\frac{P(x)}{Q(x)}=\frac{\sum_{i=0}^{r_P} a_i x^i}{\sum_{j=0}^{r_Q} b_j x^j},$$11 for sinusoid, and $$r(x)=\frac{P(x)}{Q(x)}=\frac{\sum_{i=0}^{r_P} a_i x^i}{\sum_{j=0}^{r_Q} b_j x^j},$$12 for a trainable cubic polynomial, with only $$r(x)=\frac{P(x)}{Q(x)}=\frac{\sum_{i=0}^{r_P} a_i x^i}{\sum_{j=0}^{r_Q} b_j x^j},$$13 extra parameters beyond an $$r(x)=\frac{P(x)}{Q(x)}=\frac{\sum_{i=0}^{r_P} a_i x^i}{\sum_{j=0}^{r_Q} b_j x^j},$$14-parameter ReLU baseline (Boullé et al., 2020). The same paper also shows that rational activations are feasible in GANs but notes mode collapse for digit $$r(x)=\frac{P(x)}{Q(x)}=\frac{\sum_{i=0}^{r_P} a_i x^i}{\sum_{j=0}^{r_Q} b_j x^j},$$15, presenting this as a caution that adversarial training may require additional stabilization (Boullé et al., 2020). In PINNs, REAct is not a polynomial rational activation, but it provides adjacent evidence that ratio-based learnable nonlinearities can materially improve PDE training: on the heat equation, REAct attains $$r(x)=\frac{P(x)}{Q(x)}=\frac{\sum_{i=0}^{r_P} a_i x^i}{\sum_{j=0}^{r_Q} b_j x^j},$$16 versus $$r(x)=\frac{P(x)}{Q(x)}=\frac{\sum_{i=0}^{r_P} a_i x^i}{\sum_{j=0}^{r_Q} b_j x^j},$$17 for tanh and $$r(x)=\frac{P(x)}{Q(x)}=\frac{\sum_{i=0}^{r_P} a_i x^i}{\sum_{j=0}^{r_Q} b_j x^j},$$18 for STan, and it also improves inverse-problem noise robustness across varying noise levels (Mishra et al., 4 Mar 2025).

In image classification and related vision tasks, safe Hermite-Padé activations show large and consistent improvements. The OPAU paper reports that HP-1 and HP-2 improve top-1 accuracy over ReLU by $$r(x)=\frac{P(x)}{Q(x)}=\frac{\sum_{i=0}^{r_P} a_i x^i}{\sum_{j=0}^{r_Q} b_j x^j},$$19 and $$r(x)=\frac{P(x)}{Q(x)}=\frac{\sum_{i=0}^{r_P} a_i x^i}{\sum_{j=0}^{r_Q} b_j x^j},$$20 respectively in PreActResNet-34 on CIFAR100, by $$r(x)=\frac{P(x)}{Q(x)}=\frac{\sum_{i=0}^{r_P} a_i x^i}{\sum_{j=0}^{r_Q} b_j x^j},$$21 and $$r(x)=\frac{P(x)}{Q(x)}=\frac{\sum_{i=0}^{r_P} a_i x^i}{\sum_{j=0}^{r_Q} b_j x^j},$$22 respectively in MobileNet V2 on CIFAR100, and by $$r(x)=\frac{P(x)}{Q(x)}=\frac{\sum_{i=0}^{r_P} a_i x^i}{\sum_{j=0}^{r_Q} b_j x^j},$$23 and $$r(x)=\frac{P(x)}{Q(x)}=\frac{\sum_{i=0}^{r_P} a_i x^i}{\sum_{j=0}^{r_Q} b_j x^j},$$24 in PreActResNet-34 on CIFAR10; the summary table states that HP-1 and HP-2 outperform ReLU, Leaky ReLU, ELU, Softplus, and PAU in $$r(x)=\frac{P(x)}{Q(x)}=\frac{\sum_{i=0}^{r_P} a_i x^i}{\sum_{j=0}^{r_Q} b_j x^j},$$25 tested networks and Swish in $$r(x)=\frac{P(x)}{Q(x)}=\frac{\sum_{i=0}^{r_P} a_i x^i}{\sum_{j=0}^{r_Q} b_j x^j},$$26 (Biswas et al., 2021). More structured rational architectures preserve this empirical competitiveness. Rational-ANOVA Networks report, for example, CIFAR-10 at $$r(x)=\frac{P(x)}{Q(x)}=\frac{\sum_{i=0}^{r_P} a_i x^i}{\sum_{j=0}^{r_Q} b_j x^j},$$27M parameters with KAF $$r(x)=\frac{P(x)}{Q(x)}=\frac{\sum_{i=0}^{r_P} a_i x^i}{\sum_{j=0}^{r_Q} b_j x^j},$$28, MLP $$r(x)=\frac{P(x)}{Q(x)}=\frac{\sum_{i=0}^{r_P} a_i x^i}{\sum_{j=0}^{r_Q} b_j x^j},$$29, KAN $$r(x)=\frac{P(x)}{Q(x)}=\frac{\sum_{i=0}^{r_P} a_i x^i}{\sum_{j=0}^{r_Q} b_j x^j},$$30, and RAN $$r(x)=\frac{P(x)}{Q(x)}=\frac{\sum_{i=0}^{r_P} a_i x^i}{\sum_{j=0}^{r_Q} b_j x^j},$$31, and on ViT-Tiny / ImageNet-1K they report $$r(x)=\frac{P(x)}{Q(x)}=\frac{\sum_{i=0}^{r_P} a_i x^i}{\sum_{j=0}^{r_Q} b_j x^j},$$32 top-1 for RAN-FFN versus $$r(x)=\frac{P(x)}{Q(x)}=\frac{\sum_{i=0}^{r_P} a_i x^i}{\sum_{j=0}^{r_Q} b_j x^j},$$33 for GELU and $$r(x)=\frac{P(x)}{Q(x)}=\frac{\sum_{i=0}^{r_P} a_i x^i}{\sum_{j=0}^{r_Q} b_j x^j},$$34 for KAF under matched parameters and FLOPs (Zhang et al., 3 Feb 2026).

6. Misconceptions, limitations, and unresolved design questions

A first recurring misconception is taxonomic. Some papers on “rational activations” do not in fact learn activation coefficients during network training. The classification paper based on uniform loss is explicit that $$r(x)=\frac{P(x)}{Q(x)}=\frac{\sum_{i=0}^{r_P} a_i x^i}{\sum_{j=0}^{r_Q} b_j x^j},$$35 are fixed beforehand from best uniform rational approximation and that making them learnable would destroy the quasiconvex structure on which its bisection and differential-correction solvers rely (Peiris, 2021). A second misconception is terminological: in the Atari RL paper, the regularized shared-coefficient construction is joint-rational, not recurrent-rational, and no temporal recurrence should be inferred from the activation itself (Delfosse et al., 2021).

A second limitation concerns incomplete optimization detail. The Atari paper does not provide explicit gradient formulas, exact RL rational initialization coefficients, or more detailed stabilization than the safe denominator; reproduction depends partly on released code (Delfosse et al., 2021). RAFT likewise notes that rational approximation is reliable only on a bounded interval and warns that swapping RAF into an already pretrained model only at fine-tuning time may diverge because inputs may fall outside the approximation range (Fang et al., 2022). "Rational neural networks" states that exact rational activations are tricky to train because poles can appear during learning, and its experiments rely heavily on careful ReLU-like initialization rather than on explicit admissibility constraints (Boullé et al., 2020).

A third issue is domain-dependent instability. The continuous-control study shows that original rationals can exacerbate overestimation in high-UTD RL and that constrained rationals may trade away some long-horizon plasticity in continual learning; LayerNorm does not solve the problem and can even induce instability where none previously existed (Surdej et al., 19 Jul 2025). A related, but differently framed, observation appears in the 2026 expressivity paper: normalization layers may be specifically harmful for trainable rationals because affine normalization and rational coefficients are not separately identifiable, producing flat directions and poor conditioning; the paper backs this claim empirically with no-normalization ViT results in which rational activations succeed while fixed activations fail (Tang et al., 12 Feb 2026). This suggests that trainable rationals interact with standard optimization scaffolding in ways that are not yet fully normalized across architectures.

Finally, activation learning does not automatically substitute for broader modeling choices. In the BNN–GP setting, learning activations alone is generally not sufficient for good prior matching; the best results come from learning activations jointly with prior variances (Sendera et al., 2024). In RL, the Atari paper explicitly identifies degree selection as future work, since its empirical study fixes $$r(x)=\frac{P(x)}{Q(x)}=\frac{\sum_{i=0}^{r_P} a_i x^i}{\sum_{j=0}^{r_Q} b_j x^j},$$36 throughout (Delfosse et al., 2021). In PEFT, NoRA shows that direct coefficient tuning, zero-initialized tuning, GELU-initialized tuning, and constant-only tuning all underperform structured low-rank coefficient updates, implying that how rational coefficients are parameterized can matter as much as whether they are trainable at all (Yin et al., 16 Sep 2025). A plausible implication is that future work will center less on the abstract question of whether rational activations are expressive and more on which constrained, shared, grouped, or gated parameterizations deliver the best trade-off among stability, plasticity, compute overhead, and architectural fit.