Deep Binary RationalNets: Algebra & Expressivity

• Deep Binary RationalNets are specialized neural architectures that blend binary constraints with rational activation functions to enhance both efficiency and expressivity.
• They employ an algebraic-geometric framework to rigorously characterize network expressivity via closed-form and recursive formulations of numerator and denominator functions.
• Parameter recovery and membership testing algorithms enable model inversion and interpretability, while dimensional analysis exposes symmetry-induced redundancy.

Deep Binary RationalNets are a class of neural network architectures characterized by the use of rational functions as activation functions combined with binary constraints—typically on weights, activations, or both. The term refers to architectures in which all hidden layers are binary (in various senses), while the expressive capacity is enhanced by rational parameterization. This design seeks to combine the substantial efficiency of binarized representations with the augmented expressivity of rational neural units. Recent works analyze the functional, computational, and algebraic properties of these networks, providing rigorous expressivity results and constructive algorithms for both training and functional inversion.

1. Algebraic Characterization of Expressivity

The expressive power of Deep Binary RationalNets is rigorously formulated through the lens of algebraic geometry, analyzing the set of output functions as points in a structured function space. Each output coordinate of such a network represents a rational function of the form

$f_w(x) = \frac{P_w(x)}{Q_w(x)}$

where both $P_w$ and $Q_w$ are multivariate homogeneous polynomials with degrees determined by the network architecture and the number and width of its layers. The neuromanifold is defined as the image of a combinatorial parameter map

$$\Psi_{d,\sigma} : \mathbb{R}^N \to S^{n(d)} \times \cdots \times S^{n(d)} \times S^{m(d)}$$

assigning to each parameter vector $w$ the tuple of polynomials arising from the recursive composition of rational activations throughout the network. The Zariski closure of the neuromanifold, called the neurovariety, is the ambient space of all tuples of the relevant degrees, modulo algebraic constraints imposed by the network’s structure (Grosdos et al., 14 Sep 2025).

A central result for binary-width architectures (i.e., all hidden layers of width 2) is that in single-output cases, the neurovariety fills the entire space of functions of the appropriate numerator and denominator degree. In the multi-output case, additional algebraic relations constrain the set of expressible functions.

2. Closed-Form and Recursive Structure in Binary Architectures

For architectures $d=(2,2,\dots,2,d_L)$ with $L$ layers and fixed rational activation $\sigma(x) = 1/x$, explicit closed-form expressions for the numerator and denominator functions can be given recursively:

$$\begin{aligned} P_{i,w}(x) &= p_{i,L}(x) \cdot q_{L-1}(x)\, q_{L-3}(x)\, \cdots\, q_{\delta(L+1)}(x) \ Q_{w}(x) &= q_{L}(x)\, q_{L-2}(x) \cdots\, q_{\delta(L)}(x) \end{aligned}$$

with $\delta(L)=0$ (even $L$) or $1$ (odd $L$), and $q_{k}(x)$ referencing the denominator polynomials induced at each layer. The degrees of these polynomials can be computed inductively, yielding

$\deg(P)=L+\delta(L)-1, \qquad \deg(Q)=L-\delta(L)$

demonstrating rapid increase in expressive complexity with depth. Notably, the output denominator is shared across all coordinates, implying significant symmetry between the representations.

3. Algorithms for Parameter Recovery and Membership Testing

Given a rational function with known numerator and denominator degrees, the problem of deciding whether it lies within the function space of a Deep Binary RationalNet reduces to an algebraic-geometric membership test. The steps are:

1. Factor Test: Determine whether the denominator $Q$ splits completely into $m$ linear forms (when $m$ neurons in the hidden layer). In practice, this is achieved via Brill’s equations defining the Chow variety of products of linear forms.
2. Weight Recovery: Once the linear factors are identified, reconstruct the first weight matrix $W_1$.
3. Linear System Resolution: Solve for the parameters of subsequent layers by linear inversion, attributing the appropriate coefficients to the network connections. For special architectures, much of this inversion can be reduced to univariate polynomial root-finding followed by backsolving linear equations (Grosdos et al., 14 Sep 2025).

Such algorithms allow not only parameter identification (given an explicit rational function) but also have potential utility in model inversion, interpretability, and learning settings where analytic forms are relevant.

4. Dimension, Filling Properties, and Fiber Structure

The dimension of the neuromanifold is bounded above by

$$\dim(\mathcal{V}_{d,\sigma}) \leq \left(\sum_{i=0}^L d_i d_{i+1}\right) - \left(\sum_{i=1}^{L-1} d_i\right) + 1$$

where $d_i$ are the layer widths. In the binary-width case ($d_i=2$), dimension growth is linear in depth. Proposition 9 establishes, for $d = (2, ..., 2, 1)$, that the neuromanifold is Zariski dense in the ambient coefficient space for $(P, Q)$. With $d_L>1$, generic parameter choices populate only a proper subvariety, as symmetries and polynomial dependencies restrict function expressivity.

The fibers of the combinatorial parameter map may have dimension $\sum_{i=1}^{L-1} d_i - 1$, reflecting parameter redundancy and symmetry (notably, invariance under diagonal rescaling and permutations).

5. Theoretical and Applied Implications

By encoding network output as structured tuples of rational functions, the analysis provides clear characterization of expressivity:

• Deep binary architectures, even with highly constrained widths, can approximate broad classes of rational functions, and in single-output cases this class is maximal given the degree constraints.
• Algebraic constraints governing parameterization elucidate the boundaries of representable functions, informing both theoretical approximation limits and practical design (e.g., in constructing architectures for system identification or physically-informed regression).
• The explicit description of parameter symmetry groups enables a deeper understanding of the network loss landscape, especially for issues such as nonidentifiability, local minima, and regularity of solutions.

Additionally, constructive parameter recovery opens pathways for reverse engineering models (relevant in scientific settings) or certifying the learnability of target function classes.

6. Connections to Broader Literature and Future Directions

The algebraic-geometric approach complements functional approximation perspectives (Telgarsky, 2017, Boullé et al., 2020), which show that rational neural networks achieve efficient (often exponentially more so than ReLU networks) approximation of functions with singularities using a small number of parameters and layers. In practical terms, this suggests that the blending of binarization (for memory/computation efficiency) with rational function parameterization (for broadened expressivity) is a potent combination for applications requiring both compactness and advanced modeling capacity.

As a direction for further paper, these results motivate:

• The exploration of rational activation functions beyond $\sigma(x)=1/x$, including learnable forms amenable to such algebraic analysis.
• The investigation of landscapes for training algorithms within parameter-redundant or symmetry-constrained function spaces.
• The extension of parameter recovery algorithms to cases involving noise, overparameterization, or partial observations.

The unified algebraic-geometric and functional analysis of deep binary RationalNets offers a powerful framework for both modeling complex function classes and understanding the theoretical underpinnings of compact neural architectures.