Toric geometry of ReLU neural networks (2509.05894v1)

Abstract: Given a continuous finitely piecewise linear function $f:\mathbb{R}^{n_0} \to \mathbb{R}$ and a fixed architecture $(n_0,\ldots,n_k;1)$ of feedforward ReLU neural networks, the exact function realization problem is to determine when some network with the given architecture realizes $f$. To develop a systematic way to answer these questions, we establish a connection between toric geometry and ReLU neural networks. This approach enables us to utilize numerous structures and tools from algebraic geometry to study ReLU neural networks. Starting with an unbiased ReLU neural network with rational weights, we define the ReLU fan, the ReLU toric variety, and the ReLU Cartier divisor associated with the network. This work also reveals the connection between the tropical geometry and the toric geometry of ReLU neural networks. As an application of the toric geometry framework, we prove a necessary and sufficient criterion of functions realizable by unbiased shallow ReLU neural networks by computing intersection numbers of the ReLU Cartier divisor and torus-invariant curves.

• The paper establishes a rigorous correspondence between ReLU network outputs and toric divisors, enabling exact classification of piecewise linear functions.
• It leverages tropical geometry by linking Newton polytopes with toric polytopes to clarify network expressivity and convexity properties.
• The framework provides necessary and sufficient conditions for function realization in shallow ReLU networks, guiding future architectural designs.

Toric Geometry Framework for ReLU Neural Networks

Introduction

This paper establishes a rigorous correspondence between feedforward ReLU neural networks and toric geometry, providing a geometric and algebraic framework for analyzing the exact function realization problem: determining which finitely piecewise linear functions can be exactly represented by a given ReLU network architecture. The approach leverages the machinery of toric varieties, fans, and divisors from algebraic geometry, extending previous connections to tropical geometry and enabling new structural insights into the expressivity and limitations of ReLU networks.

ReLU Networks as Toric Objects

The central construction interprets the output of an unbiased feedforward ReLU network (with rational weights and no biases) as the support function of a $\mathbb{Q}$-Cartier divisor on a rational polyhedral fan. The canonical polyhedral complex induced by the network's bent hyperplane arrangement is identified as the "ReLU fan," and the associated toric variety is termed the "ReLU toric variety." The output function, up to affine shift, corresponds to a ReLU Cartier divisor supported on this fan.

Formally, for a fixed architecture $(n_0, n_1, \ldots, n_k; 1)$, the toric encoding map $\mathcal{T}$ sends network parameters $\theta$ to the ReLU fan $\Sigma_{f_\theta}$, which encodes the combinatorial structure of the network's piecewise linear regions. The ReLU Cartier divisor $D_f$ captures the output function's slopes on each maximal cone, and the equivalence class of functions modulo affine shifts is in bijection with the set of $\mathbb{Q}$-Cartier divisors on the fan.

Tropical and Toric Geometry Linkage

The paper rigorously connects the tropical geometry perspective—where ReLU network outputs with integral weights are tropical rational functions—to the toric framework. For tropical polynomials (convex max-of-affine functions), the Newton polytope coincides (up to sign) with the polytope $P_{-D}$ associated to the ReLU Cartier divisor $-D$. The mixed volume of the Newton polytope equals the volume of the corresponding line bundle on the ReLU toric variety, establishing a precise geometric correspondence.

When the output is a tropical rational function, the toric polytope $P_D$ generalizes the Newton polytope, providing a well-defined object even when the tropical perspective lacks a canonical polytope. This generalization is significant for analyzing the expressivity of ReLU networks beyond convex functions.

Exact Realization and Architectural Expressivity

A key theoretical result is that the set of functions realizable by a ReLU network of fixed depth $k$ is invariant under affine shifts. That is, if $f$ is realizable and $f' = f + g$ for affine $g$, then $f'$ is also realizable by a network of the same depth. This reduces the classification problem to the paper of ReLU Cartier divisors modulo affine functions.

The paper provides a complete characterization of functions exactly realizable by unbiased shallow (one-hidden-layer) ReLU networks with rational weights. The necessary and sufficient condition is formulated in terms of intersection numbers of the ReLU Cartier divisor with torus-invariant curves (walls of the fan): for any two walls lying in the same hyperplane, the intersection numbers must be equal. This criterion is both algebraic and geometric, reflecting the symmetry and combinatorial constraints imposed by the network architecture.

The reduced ReLU representation algorithm ensures that the analysis is performed on canonical forms, eliminating redundancies due to parallel or zero rows in the weight matrices.

Implications and Future Directions

The toric geometry framework provides a powerful toolkit for analyzing the structural properties of ReLU networks, including exact function realization, architectural limitations, and expressivity. The explicit connection to tropical geometry clarifies the role of convexity and combinatorial structure in network outputs, and the use of divisors and intersection theory enables precise classification results.

Practically, these results inform the design of network architectures for exact representation of piecewise linear functions, and the geometric perspective may guide the development of new architectures with desired expressivity properties. The framework also suggests potential generalizations to networks with biases, deeper architectures, and other activation functions, as well as connections to algebraic statistics and neuroalgebraic geometry.

Theoretically, the identification of network outputs with toric divisors opens avenues for applying advanced techniques from algebraic geometry, such as cohomological methods, to neural network analysis. The generalization of the Newton polytope via the toric polytope $P_D$ may have further implications for understanding the combinatorial complexity of network function spaces.

Conclusion

This work establishes a comprehensive dictionary between feedforward ReLU neural networks and toric geometry, enabling the use of algebraic and geometric tools to address the exact function realization problem. The framework unifies and extends previous tropical geometry approaches, provides complete classification results for shallow architectures, and lays the foundation for future research at the intersection of algebraic geometry and neural network theory.