Rational Neural Networks

• Rational neural networks are models that incorporate trainable rational functions as activations or layer parameterizations to adapt the nonlinearity during optimization.
• They achieve efficient approximation of non-smooth functions, offering exponential depth savings and improved accuracy compared to traditional ReLU networks.
• Different formulations—from symbolic regression and graph spectral filtering to control synthesis—demonstrate broad applicability and exact representability of these algebraically structured models.

Searching arXiv for papers on rational neural networks and closely related formulations. Rational neural networks are neural-network models in which rational functions—ratios of polynomials—enter either as trainable activation functions, as the explicit parameterization of layers and filters, or as the algebraic objects used to characterize representable maps. In the literature, the term covers feedforward networks with activations of the form $R(x)=P(x)/Q(x)$, controller and symbolic-regression architectures whose units are directly given by polynomial numerators and denominators, graph networks with rational spectral filters, rational Gaussian wavelet layers, and rational-weight ReLU networks studied through exact depth lower bounds (Boullé et al., 2020, Newton et al., 2023, Chen et al., 2018, Averkov et al., 10 Feb 2025). This suggests that rational neural networks are best viewed as a family of algebraically structured models rather than a single canonical architecture.

1. Terminological scope and basic forms

A common definition makes the nonlinearity itself rational. In this formulation, each activation is a learnable function

$$\operatorname{R}(x)=\frac{\operatorname{P}(x)}{\operatorname{Q}(x)}=\frac{\sum_{j=0}^{m} a_{j} x^{j}}{1 + \sum_{k=1}^{n} b_{k} x^{k}},$$

or, equivalently, a low-degree rational function

$$F(x)=\frac{\sum_{i=0}^{r_P} a_i x^i}{\sum_{j=0}^{r_Q} b_j x^j},$$

with trainable numerator and denominator coefficients (Delfosse et al., 2021, Boullé et al., 2020). In this line of work, the attraction of rationals is their ability to let each neuron adapt its nonlinear shape during optimization, rather than fixing the activation to ReLU, sigmoid, or tanh.

A second formulation makes the entire layer rational. In the control-oriented parameterization, one writes

$$x_{i}^{k+1}=\frac{p_i(x^k)}{q_i(x^k)}, \qquad \pi_i(u)=\frac{p_i(x^\ell)}{q_i(x^\ell)},$$

so that the trainable objects are polynomial coefficients inside numerators and denominators rather than affine weights followed by a separate activation (Newton et al., 2023). In symbolic-regression-oriented models, the output itself is directly constrained to be rational, for example

$\hat y=\frac{E_N C}{E_D d},$

with polynomial basis terms explicitly enumerated in the input layer and sparsity imposed on the coefficient vectors (Sun, 2021).

A third usage is architectural rather than neuron-wise. Rational Gaussian wavelet models define a mother wavelet by

$$\psi^{\boldsymbol\eta}(t)=C(\boldsymbol\eta)\,P^{\boldsymbol\eta}(t)\,v^{\boldsymbol\eta}(t)\,e^{-t^2/2},$$

where the zeros of the polynomial factor and the poles of the rational factor become trainable parameters that shape the feature extractor (Ámon et al., 3 Feb 2025). Spectral graph models similarly replace polynomial graph filters by rational ones of the form $P(\Lambda)/Q(\Lambda)$ (Chen et al., 2018).

A fourth usage concerns weights rather than activations. In the theory of rational ReLU networks, the activation remains ReLU, but weights are restricted to rational forms such as decimal fractions or, more generally, $N$-ary fractions (Averkov et al., 10 Feb 2025). This version is central when exact representability and depth lower bounds are the main object of study.

Across these variants, pole avoidance is a recurring technical constraint. Safe rational functions with absolute values in the denominator, positivity-enforcing parameterizations, and softplus-based positive denominators are repeatedly introduced to prevent real poles and training instability (Delfosse et al., 2021, Boullé et al., 2020, Zhang et al., 3 Feb 2026).

2. Approximation-theoretic foundations

The modern theory begins from the observation that rational functions and ReLU networks approximate each other surprisingly efficiently. For any ReLU network, there exists a rational function of degree $O(\mathrm{polylog}(1/\epsilon))$ that is $\epsilon$-close, and for suitable rational functions there exists a ReLU network of size $$\operatorname{R}(x)=\frac{\operatorname{P}(x)}{\operatorname{Q}(x)}=\frac{\sum_{j=0}^{m} a_{j} x^{j}}{1 + \sum_{k=1}^{n} b_{k} x^{k}},$$0 that is $$\operatorname{R}(x)=\frac{\operatorname{P}(x)}{\operatorname{Q}(x)}=\frac{\sum_{j=0}^{m} a_{j} x^{j}}{1 + \sum_{k=1}^{n} b_{k} x^{k}},$$1-close (Telgarsky, 2017). The key classical ingredient is Newman’s approximation of $$\operatorname{R}(x)=\frac{\operatorname{P}(x)}{\operatorname{Q}(x)}=\frac{\sum_{j=0}^{m} a_{j} x^{j}}{1 + \sum_{k=1}^{n} b_{k} x^{k}},$$2, from which

$$\operatorname{R}(x)=\frac{\operatorname{P}(x)}{\operatorname{Q}(x)}=\frac{\sum_{j=0}^{m} a_{j} x^{j}}{1 + \sum_{k=1}^{n} b_{k} x^{k}},$$3

yields efficient rational approximants to ReLU. By contrast, polynomials require degree $$\operatorname{R}(x)=\frac{\operatorname{P}(x)}{\operatorname{Q}(x)}=\frac{\sum_{j=0}^{m} a_{j} x^{j}}{1 + \sum_{k=1}^{n} b_{k} x^{k}},$$4 to approximate even a single ReLU (Telgarsky, 2017).

This bidirectional relationship was sharpened by work that treated rational activations as a native network primitive rather than as an external surrogate. Composing $$\operatorname{R}(x)=\frac{\operatorname{P}(x)}{\operatorname{Q}(x)}=\frac{\sum_{j=0}^{m} a_{j} x^{j}}{1 + \sum_{k=1}^{n} b_{k} x^{k}},$$5 rational functions of type $$\operatorname{R}(x)=\frac{\operatorname{P}(x)}{\operatorname{Q}(x)}=\frac{\sum_{j=0}^{m} a_{j} x^{j}}{1 + \sum_{k=1}^{n} b_{k} x^{k}},$$6 yields a rational function of type $$\operatorname{R}(x)=\frac{\operatorname{P}(x)}{\operatorname{Q}(x)}=\frac{\sum_{j=0}^{m} a_{j} x^{j}}{1 + \sum_{k=1}^{n} b_{k} x^{k}},$$7 while using only $$\operatorname{R}(x)=\frac{\operatorname{P}(x)}{\operatorname{Q}(x)}=\frac{\sum_{j=0}^{m} a_{j} x^{j}}{1 + \sum_{k=1}^{n} b_{k} x^{k}},$$8 parameters, so the represented degree grows exponentially with depth although the number of trainable coefficients grows only linearly (Boullé et al., 2020). The same paper proves that smooth Sobolev functions can be approximated by rational networks with size

$$\operatorname{R}(x)=\frac{\operatorname{P}(x)}{\operatorname{Q}(x)}=\frac{\sum_{j=0}^{m} a_{j} x^{j}}{1 + \sum_{k=1}^{n} b_{k} x^{k}},$$9

and depth

$$F(x)=\frac{\sum_{i=0}^{r_P} a_i x^i}{\sum_{j=0}^{r_Q} b_j x^j},$$0

which is presented as an exponential depth saving relative to comparable ReLU constructions (Boullé et al., 2020). Empirically, the same study reported mean squared error $$F(x)=\frac{\sum_{i=0}^{r_P} a_i x^i}{\sum_{j=0}^{r_Q} b_j x^j},$$1 for a rational network on a Korteweg–de Vries approximation task, compared with $$F(x)=\frac{\sum_{i=0}^{r_P} a_i x^i}{\sum_{j=0}^{r_Q} b_j x^j},$$2 for ReLU (Boullé et al., 2020).

A later approximation-theoretic program extends the comparison from ReLU to a broad family of modern fixed activations. It shows that any network built from standard fixed activations can be uniformly approximated on compact domains by a rational-activation network with only $$F(x)=\frac{\sum_{i=0}^{r_P} a_i x^i}{\sum_{j=0}^{r_Q} b_j x^j},$$3 overhead in size, while the converse provably requires $$F(x)=\frac{\sum_{i=0}^{r_P} a_i x^i}{\sum_{j=0}^{r_Q} b_j x^j},$$4 parameters in the worst case (Tang et al., 12 Feb 2026). The separation is stated not only for scalar activations such as GELU, SiLU, Mish, ELU, SELU, CELU, Softplus, Sigmoid, Tanh, Softmin, Softmax, and LogSoftmax, but also for gated activations and transformer-style nonlinearities (Tang et al., 12 Feb 2026).

Rational approximation has also been pushed into derivative-sensitive regimes. For $$F(x)=\frac{\sum_{i=0}^{r_P} a_i x^i}{\sum_{j=0}^{r_Q} b_j x^j},$$5, there exist rational functions of type $$F(x)=\frac{\sum_{i=0}^{r_P} a_i x^i}{\sum_{j=0}^{r_Q} b_j x^j},$$6 with

$$F(x)=\frac{\sum_{i=0}^{r_P} a_i x^i}{\sum_{j=0}^{r_Q} b_j x^j},$$7

and

$$F(x)=\frac{\sum_{i=0}^{r_P} a_i x^i}{\sum_{j=0}^{r_Q} b_j x^j},$$8

for every $$F(x)=\frac{\sum_{i=0}^{r_P} a_i x^i}{\sum_{j=0}^{r_Q} b_j x^j},$$9 (Morina et al., 27 Aug 2025). Using these approximants, suitable Hölder-smooth functions $$x_{i}^{k+1}=\frac{p_i(x^k)}{q_i(x^k)}, \qquad \pi_i(u)=\frac{p_i(x^\ell)}{q_i(x^\ell)},$$0 can be approximated in $$x_{i}^{k+1}=\frac{p_i(x^k)}{q_i(x^k)}, \qquad \pi_i(u)=\frac{p_i(x^\ell)}{q_i(x^\ell)},$$1 by rational neural networks of width of order $$x_{i}^{k+1}=\frac{p_i(x^k)}{q_i(x^k)}, \qquad \pi_i(u)=\frac{p_i(x^\ell)}{q_i(x^\ell)},$$2, constant depth, and maximal rational degree of order $$x_{i}^{k+1}=\frac{p_i(x^k)}{q_i(x^k)}, \qquad \pi_i(u)=\frac{p_i(x^\ell)}{q_i(x^\ell)},$$3, with

$$x_{i}^{k+1}=\frac{p_i(x^k)}{q_i(x^k)}, \qquad \pi_i(u)=\frac{p_i(x^\ell)}{q_i(x^\ell)},$$4

The same framework yields $$x_{i}^{k+1}=\frac{p_i(x^k)}{q_i(x^k)}, \qquad \pi_i(u)=\frac{p_i(x^\ell)}{q_i(x^\ell)},$$5-approximation results for the $$x_{i}^{k+1}=\frac{p_i(x^k)}{q_i(x^k)}, \qquad \pi_i(u)=\frac{p_i(x^\ell)}{q_i(x^\ell)},$$6 and ParFam architectures used in symbolic regression and physical law learning (Morina et al., 27 Aug 2025).

3. Exact representability and depth lower bounds

A distinct branch of the literature studies exact representation rather than approximation, and here the test function

$$x_{i}^{k+1}=\frac{p_i(x^k)}{q_i(x^k)}, \qquad \pi_i(u)=\frac{p_i(x^\ell)}{q_i(x^\ell)},$$7

is central because it is the support function of the standard simplex $$x_{i}^{k+1}=\frac{p_i(x^k)}{q_i(x^k)}, \qquad \pi_i(u)=\frac{p_i(x^\ell)}{q_i(x^\ell)},$$8 (Averkov et al., 10 Feb 2025). The underlying conjecture, due to Hertrich, Basu, Di Summa, and Skutella, predicts that any ReLU network exactly representing $$x_{i}^{k+1}=\frac{p_i(x^k)}{q_i(x^k)}, \qquad \pi_i(u)=\frac{p_i(x^\ell)}{q_i(x^\ell)},$$9 should need at least $\hat y=\frac{E_N C}{E_D d},$0 hidden layers.

For rationally representable weights of arithmetic relevance, the strongest general theorem currently available states that if $\hat y=\frac{E_N C}{E_D d},$1 is a prime not dividing $\hat y=\frac{E_N C}{E_D d},$2, then every ReLU network whose weights are $\hat y=\frac{E_N C}{E_D d},$3-ary fractions needs at least

$\hat y=\frac{E_N C}{E_D d},$4

hidden layers to exactly represent $\hat y=\frac{E_N C}{E_D d},$5 (Averkov et al., 10 Feb 2025). For decimal fractions, choosing $\hat y=\frac{E_N C}{E_D d},$6 yields the concrete lower bound

$\hat y=\frac{E_N C}{E_D d},$7

The same paper also proves a denominator-sensitive asymptotic lower bound: there exists a constant $\hat y=\frac{E_N C}{E_D d},$8 such that for all $\hat y=\frac{E_N C}{E_D d},$9, every ReLU network with $$\psi^{\boldsymbol\eta}(t)=C(\boldsymbol\eta)\,P^{\boldsymbol\eta}(t)\,v^{\boldsymbol\eta}(t)\,e^{-t^2/2},$$0-ary fraction weights that exactly represents $$\psi^{\boldsymbol\eta}(t)=C(\boldsymbol\eta)\,P^{\boldsymbol\eta}(t)\,v^{\boldsymbol\eta}(t)\,e^{-t^2/2},$$1 has depth at least

$$\psi^{\boldsymbol\eta}(t)=C(\boldsymbol\eta)\,P^{\boldsymbol\eta}(t)\,v^{\boldsymbol\eta}(t)\,e^{-t^2/2},$$2

These are presented as the first non-constant lower bounds on the depth of practically relevant rational-weight ReLU networks (Averkov et al., 10 Feb 2025).

The proof combines a polyhedral characterization of positively homogeneous ReLU representability with a modular volume obstruction. Representability is expressed through the sum-union closure $$\psi^{\boldsymbol\eta}(t)=C(\boldsymbol\eta)\,P^{\boldsymbol\eta}(t)\,v^{\boldsymbol\eta}(t)\,e^{-t^2/2},$$3 of point polytopes, together with the equivalence

$$\psi^{\boldsymbol\eta}(t)=C(\boldsymbol\eta)\,P^{\boldsymbol\eta}(t)\,v^{\boldsymbol\eta}(t)\,e^{-t^2/2},$$4

A denominator-clearing lemma then reduces rational weights to integer weights: $$\psi^{\boldsymbol\eta}(t)=C(\boldsymbol\eta)\,P^{\boldsymbol\eta}(t)\,v^{\boldsymbol\eta}(t)\,e^{-t^2/2},$$5 whenever $$\psi^{\boldsymbol\eta}(t)=C(\boldsymbol\eta)\,P^{\boldsymbol\eta}(t)\,v^{\boldsymbol\eta}(t)\,e^{-t^2/2},$$6 is representable with $$\psi^{\boldsymbol\eta}(t)=C(\boldsymbol\eta)\,P^{\boldsymbol\eta}(t)\,v^{\boldsymbol\eta}(t)\,e^{-t^2/2},$$7 hidden layers and all rational weights have common denominator $$\psi^{\boldsymbol\eta}(t)=C(\boldsymbol\eta)\,P^{\boldsymbol\eta}(t)\,v^{\boldsymbol\eta}(t)\,e^{-t^2/2},$$8 (Averkov et al., 10 Feb 2025). The contradiction is obtained by showing that certain normalized face volumes must be divisible by a prime $$\psi^{\boldsymbol\eta}(t)=C(\boldsymbol\eta)\,P^{\boldsymbol\eta}(t)\,v^{\boldsymbol\eta}(t)\,e^{-t^2/2},$$9, whereas the simplex has normalized volume $P(\Lambda)/Q(\Lambda)$0.

These results are explicitly described as a partial confirmation of the original conjecture. They establish that depth must grow with $P(\Lambda)/Q(\Lambda)$1 for exact max computation in rational settings, but they do not settle the $P(\Lambda)/Q(\Lambda)$2 bound for arbitrary real weights (Averkov et al., 10 Feb 2025).

4. Architecture families and empirical domains

In graph learning, RationalNet replaces polynomial spectral filters by rational ones,

$P(\Lambda)/Q(\Lambda)$3

in order to better approximate jump discontinuities and non-smooth high-pass behavior (Chen et al., 2018). The motivation is classical Gibbs-type failure of Chebyshev approximants at discontinuities. The paper gives a rational convergence rate

$P(\Lambda)/Q(\Lambda)$4

contrasted with a polynomial rate

$P(\Lambda)/Q(\Lambda)$5

and uses a relaxed Remez algorithm to initialize the rational coefficients (Chen et al., 2018). On a 1000-node synthetic graph, RationalNet achieved spectral MSE around $P(\Lambda)/Q(\Lambda)$6 on $P(\Lambda)/Q(\Lambda)$7, and Remez initialization improved spectral MSE by 56.26% for $P(\Lambda)/Q(\Lambda)$8 and 81.39% for $P(\Lambda)/Q(\Lambda)$9 (Chen et al., 2018).

Wavelet-based rational neural networks form another important family. Rational Gaussian wavelets define an adaptive mother wavelet

$N$0

where the zeros of $N$1 and the poles of $N$2 directly control wavelet morphology (Ámon et al., 3 Feb 2025). A variable-projection layer

$N$3

then produces interpretable wavelet coefficients, while the reconstruction map

$N$4

supports an additional regularizer (Ámon et al., 3 Feb 2025). On ventricular ectopic beat detection from MIT-BIH ECG data, the reported setup used $N$5 wavelet coefficients, $N$6 zeros, and $N$7 poles, and achieved 98.51% total accuracy (Ámon et al., 3 Feb 2025).

The same rational Gaussian wavelet idea was later transferred from variable projection to discrete convolution for nonperiodic acoustic segments in UAV recognition (Gergő et al., 25 May 2026). The RGW convolution layer maps a sampled signal to multiple scale-dependent responses, after which top-$N$8 pooling keeps the dominant coefficients. In indoor swarm detection, indoor drone classification, and noisy outdoor detection, the reported mean accuracies were 92.52%, 99.20%, and 90.55%, respectively; in the outdoor scenario, the proposed model outperformed CNN, with CNN at 89.60% (Gergő et al., 25 May 2026).

In scientific model discovery, rational function neural networks are used as symbolic-form estimators rather than purely predictive models. RafNN constrains the output to a rational expression with explicitly enumerated numerator and denominator basis terms, uses sparsity penalties

$N$9

and applies thresholding to prune small coefficients (Sun, 2021). On synthetic rock-physics data with 1% random noise, the method reconstructed Gassmann’s equation; after 35 independent training runs, the active term counts converged to $O(\mathrm{polylog}(1/\epsilon))$0 and $O(\mathrm{polylog}(1/\epsilon))$1, and the reported recovered coefficients differed from the target values by below 0.03%, with best-case differences of 0.01% and 0.004% (Sun, 2021).

A recent architectural synthesis is the Rational-ANOVA Network, which combines Padé-style rational units with a functional-ANOVA interaction topology,

$O(\mathrm{polylog}(1/\epsilon))$2

and enforces a strictly positive denominator

$O(\mathrm{polylog}(1/\epsilon))$3

to avoid poles (Zhang et al., 3 Feb 2026). Under matched budgets, the paper reports roughly 59.05% on CIFAR-10 at around 1.0M parameters, compared with 56.95% for MLP and 56.45% for KAN, and a ViT-Tiny variant whose top-1 accuracy rises from 72.3% to 74.2% when the FFN is replaced by RAN (Zhang et al., 3 Feb 2026).

5. Optimization and training regimes

One influential theme treats rationality as adaptive activation plasticity. In deep reinforcement learning, rational activations of order $O(\mathrm{polylog}(1/\epsilon))$4 are proposed as trainable replacements for static nonlinearities, and a central theorem states that a rational function embeds a residual connection if and only if $O(\mathrm{polylog}(1/\epsilon))$5 (Delfosse et al., 2021). The same work introduces a regularized parameter-sharing version, called joint-rational, in which one rational activation is shared across layers. In Atari experiments, replacing fixed activations by rational ones led to consistent improvements for DQN, with joint-rational often performing especially well on stationary games and making simple DQN competitive with DDQN and Rainbow (Delfosse et al., 2021).

Another training line fixes the rational activation coefficients and changes the loss. For the one-degree activation

$O(\mathrm{polylog}(1/\epsilon))$6

with coefficients chosen as the best rational $O(\mathrm{polylog}(1/\epsilon))$7 approximation to ReLU on $O(\mathrm{polylog}(1/\epsilon))$8, network training under the uniform loss

$O(\mathrm{polylog}(1/\epsilon))$9

becomes a quasiconvex generalized rational uniform approximation problem (Peiris, 2021). This permits bisection and differential-correction methods instead of least-squares optimization. On TwoLeadECG, the reported test accuracies were 87.71% for bisection, 70.2% for the MATLAB toolbox, and 55.31% for differential correction; on an imbalanced SonyAIBORobotSurface1 split with class 1 underrepresented, the corresponding values were 73.54%, 52.2%, and 65.06% (Peiris, 2021).

In control, rational parameterization is used to make neural feedback loops compatible with Sum of Squares programming. Standard rational activations such as

$\epsilon$0

satisfy exact polynomial equalities after clearing denominators, which is advantageous for Positivstellensatz-based stability certificates (Newton et al., 2023). The same work proposes a more general rational neural network structure that is convex in the network parameters and a refined architecture with state-dependent denominators to avoid numerical recovery issues in SOS synthesis. The resulting method recovers stabilizing rational neural network controllers for unstable and nonlinear plants with saturation, noise, and parametric uncertainty (Newton et al., 2023).

6. Algebraic, logical, and geometric theories

The rational perspective has also produced a substantial formal theory. In continuous-time systems, a broad class of recurrent neural networks can be embedded into rational or polynomial systems under mild assumptions on the activation function (Defourneau et al., 2019). The standing hypothesis is that the activation is analytic and satisfies a differential-algebraic closure property, equivalently a nontrivial polynomial relation among $\epsilon$1. This includes $\epsilon$2 and the sigmoid $\epsilon$3, since $\epsilon$4 and $\epsilon$5, respectively (Defourneau et al., 2019). The embedding transfers questions of realizability, minimality, reachability, and observability from RNNs to the established realization theory of rational systems.

In graph logic, however, rationality does not simply enlarge expressivity. The logic of rational graph neural networks studies message-passing GNNs whose combination networks use rational activations $\epsilon$6 with no real pole (Khalife, 2023). The main negative result is that some depth-3 $\epsilon$7 queries cannot be expressed by any rational GNN, even though ReLU GNNs are known to capture the full logical fragment $\epsilon$8 uniformly (Khalife, 2023). To delimit the positive fragment, the paper defines $\epsilon$9 and proves that rational GNNs can express every query in that fragment uniformly over all graphs (Khalife, 2023).

A complementary logic-theoretic result shows that rational-weight ReLU networks admit fuzzy-logic characterizations. Using a scaling map

$$\operatorname{R}(x)=\frac{\operatorname{P}(x)}{\operatorname{Q}(x)}=\frac{\sum_{j=0}^{m} a_{j} x^{j}}{1 + \sum_{k=1}^{n} b_{k} x^{k}},$$00

the paper proves that rational-weight feedforward ReLU networks have the same expressive power, with respect to scaling, as Rational Pavelka Logic $$\operatorname{R}(x)=\frac{\operatorname{P}(x)}{\operatorname{Q}(x)}=\frac{\sum_{j=0}^{m} a_{j} x^{j}}{1 + \sum_{k=1}^{n} b_{k} x^{k}},$$01, fragments of $$\operatorname{R}(x)=\frac{\operatorname{P}(x)}{\operatorname{Q}(x)}=\frac{\sum_{j=0}^{m} a_{j} x^{j}}{1 + \sum_{k=1}^{n} b_{k} x^{k}},$$02, and a generalized polynomial ring over $$\operatorname{R}(x)=\frac{\operatorname{P}(x)}{\operatorname{Q}(x)}=\frac{\sum_{j=0}^{m} a_{j} x^{j}}{1 + \sum_{k=1}^{n} b_{k} x^{k}},$$03 in countably many variables with $$\operatorname{R}(x)=\frac{\operatorname{P}(x)}{\operatorname{Q}(x)}=\frac{\sum_{j=0}^{m} a_{j} x^{j}}{1 + \sum_{k=1}^{n} b_{k} x^{k}},$$04 permitted (Heiman et al., 4 May 2026). The translation is constructive in both directions and identifies proto-neurons with degree-$$\operatorname{R}(x)=\frac{\operatorname{P}(x)}{\operatorname{Q}(x)}=\frac{\sum_{j=0}^{m} a_{j} x^{j}}{1 + \sum_{k=1}^{n} b_{k} x^{k}},$$05 terms in the generalized polynomial structure (Heiman et al., 4 May 2026).

Algebraic geometry supplies another viewpoint. For RationalNets with activation $$\operatorname{R}(x)=\frac{\operatorname{P}(x)}{\operatorname{Q}(x)}=\frac{\sum_{j=0}^{m} a_{j} x^{j}}{1 + \sum_{k=1}^{n} b_{k} x^{k}},$$06, the output of a fixed architecture can be written as a tuple of homogeneous polynomial fractions with common denominator, and the set of all such outputs is called the neuromanifold; its Zariski closure is the neurovariety (Grosdos et al., 14 Sep 2025). For one-hidden-layer architectures $$\operatorname{R}(x)=\frac{\operatorname{P}(x)}{\operatorname{Q}(x)}=\frac{\sum_{j=0}^{m} a_{j} x^{j}}{1 + \sum_{k=1}^{n} b_{k} x^{k}},$$07, the paper shows that the Zariski closure is filling but the actual neuromanifold is not; for deep binary RationalNets, it classifies when the Zariski closure fills the ambient space and proves that this happens only for architectures of the form $$\operatorname{R}(x)=\frac{\operatorname{P}(x)}{\operatorname{Q}(x)}=\frac{\sum_{j=0}^{m} a_{j} x^{j}}{1 + \sum_{k=1}^{n} b_{k} x^{k}},$$08 (Grosdos et al., 14 Sep 2025). Membership algorithms are also given for deciding whether a prescribed rational function belongs to the neuromanifold (Grosdos et al., 14 Sep 2025).

A related algebraic-combinatorial analysis links rational neural networks to VC-theory. Using the Erzeugungsgrad of Boolean algebras of constructible sets and degree notions for constructible families, one obtains bounds in which VC-dimension and Krull dimension are linearly related up to logarithmic factors (Pardo et al., 15 Apr 2025). These results are then applied to parameterized families of neural networks with rational activation function, yielding bounds on growth functions, VC-dimension, and densities of correct test sequences (Pardo et al., 15 Apr 2025).

7. Limitations and open problems

Despite their approximation-theoretic strength, rational neural networks are not uniformly dominant across all formulations. A direct comparison between classical rational approximation and neural networks with rational activations reports that direct rational approximation is consistently more accurate than neural-network-based approximation when both are given the same number of decision variables, including on nonsmooth and non-Lipschitz targets (Peiris et al., 2023). This suggests that, in some regimes, the compositional neural parameterization is a restriction rather than an advantage.

Training stability remains a persistent concern. Unconstrained rational functions can develop poles and unstable gradients, which is why many implementations adopt absolute values in denominators, positivity constraints, or softplus-based denominator parameterizations (Boullé et al., 2020, Delfosse et al., 2021, Zhang et al., 3 Feb 2026). Recent work on adaptive rational activations also argues that normalization layers can interfere with adaptive rationals by introducing non-identifiability and stochasticity that interact badly with coefficient sensitivity (Tang et al., 12 Feb 2026).

Expressivity is also architecture-dependent. Rational activations do not make GNNs maximally expressive in the uniform logical sense, since some $$\operatorname{R}(x)=\frac{\operatorname{P}(x)}{\operatorname{Q}(x)}=\frac{\sum_{j=0}^{m} a_{j} x^{j}}{1 + \sum_{k=1}^{n} b_{k} x^{k}},$$09 queries remain unattainable (Khalife, 2023). In exact-representation theory, current depth lower bounds for rational-weight ReLU networks stop short of the conjectured $$\operatorname{R}(x)=\frac{\operatorname{P}(x)}{\operatorname{Q}(x)}=\frac{\sum_{j=0}^{m} a_{j} x^{j}}{1 + \sum_{k=1}^{n} b_{k} x^{k}},$$10 barrier for arbitrary real weights, so the general real-weight case remains open (Averkov et al., 10 Feb 2025). In $$\operatorname{R}(x)=\frac{\operatorname{P}(x)}{\operatorname{Q}(x)}=\frac{\sum_{j=0}^{m} a_{j} x^{j}}{1 + \sum_{k=1}^{n} b_{k} x^{k}},$$11-approximation, the extension from $$\operatorname{R}(x)=\frac{\operatorname{P}(x)}{\operatorname{Q}(x)}=\frac{\sum_{j=0}^{m} a_{j} x^{j}}{1 + \sum_{k=1}^{n} b_{k} x^{k}},$$12 to $$\operatorname{R}(x)=\frac{\operatorname{P}(x)}{\operatorname{Q}(x)}=\frac{\sum_{j=0}^{m} a_{j} x^{j}}{1 + \sum_{k=1}^{n} b_{k} x^{k}},$$13 is conjectured for $$\operatorname{R}(x)=\frac{\operatorname{P}(x)}{\operatorname{Q}(x)}=\frac{\sum_{j=0}^{m} a_{j} x^{j}}{1 + \sum_{k=1}^{n} b_{k} x^{k}},$$14 but not proved (Morina et al., 27 Aug 2025).

The cumulative picture is therefore differentiated rather than uniform. Rational neural networks offer unusually strong tools for approximation of non-smooth structure, exact algebraic modeling, symbolic regression, and control-compatible synthesis; yet they also introduce specific numerical and structural issues, and in several subfields their expressive envelope is now known to differ sharply from that of ReLU-based models (Tang et al., 12 Feb 2026, Khalife, 2023).