ReLU Toric Variety Analysis

• ReLU toric variety is the toric variety constructed from the canonical ReLU fan that encodes the activation regions of a feedforward ReLU network with rational weights.
• It translates the network’s piecewise linear output into a support function on the toric variety, enabling analysis through Cartier divisors and intersection theory.
• The framework bridges polyhedral, tropical, and toric geometry to offer computational and algebraic tools for understanding network expressivity and function realizability.

A ReLU toric variety is a toric variety explicitly associated to the canonical fan arising from the regions of linearity of a continuous finitely piecewise linear function realized by an (unbiased) feedforward ReLU neural network with rational weights. This construction translates the geometric partition induced by network activations into the combinatorial data of a fan, enabling the application of algebraic and tropical geometry to analyze neural network expressivity, realizability, and function capacity. The associated toric variety encodes the piecewise linear structure of the network, and the output function acts as a support function (Q–Cartier divisor) furnishing powerful geometric invariants.

1. Canonical Construction of ReLU Toric Varieties

For a ReLU network with architecture $(n_0, \ldots, n_k; 1)$ and rational weights, the input space $\mathbb{R}^{n_0}$ is partitioned by the arrangement of hyperplanes defined by the network’s weights. Each region where a particular subset of hidden units are active is a polyhedral cone, and the collection of these cones forms a canonical polyhedral complex $\mathcal{C}_{f_{\theta}}$, called the ReLU fan $\Sigma_{f_{\theta}}$ [(Fu, 7 Sep 2025), Definition 4.1].

The ReLU toric variety $X_{\Sigma_{f_{\theta}}}$ is then the toric variety determined by this fan [(Fu, 7 Sep 2025), Definition 4.2]. This procedure consists of:

• Determining the full hyperplane arrangement from the weights.
• Constructing the polyhedral complex encoding activation patterns.
• Forming the fan $\Sigma_{f_{\theta}}$, gluing maximal cones corresponding to these regions.
• Defining the associated toric variety $X_{\Sigma_{f_{\theta}}}$.

This approach provides a canonical embedding of the network’s piecewise linear function into the language of toric geometry, generalizing the passage from combinatorial polyhedral data to algebraic varieties.

2. Toric Geometry and Neural Network Function Realization

A key insight is that the ReLU network’s output, a continuous piecewise linear function $f$, is linear on each maximal cone of the fan $\Sigma_{f_{\theta}}$. In toric geometry, such a function is interpreted as a support function for a (Q–)Cartier divisor $D_f$ on $X_{\Sigma_{f_{\theta}}}$ [(Fu, 7 Sep 2025), Definition 4.4].

This translation enables the full machinery of toric intersection theory, divisor theory, and polytope operations to be applied to the network realization problem. Specifically:

• Each maximal cone $\sigma$ corresponds to a region where $f(x) = a_1 x_1 + \ldots + a_{n_0} x_{n_0}$, and the associated slope data gives the Cartier datum $m_{\sigma}$ [(Fu, 7 Sep 2025), Lemma 4.3].
• Two piecewise linear functions differing by an affine term yield the same divisor class (up to linear equivalence), reducing realizability to checking divisor support functions modulo affine ambiguity [(Fu, 7 Sep 2025), Theorem 4.6].

This framework recasts the “exact function realization problem” as classifying which support functions (or equivalently, Cartier divisors) can occur for a fixed fan, yielding geometric criteria for network expressivity.

3. The ReLU Fan, Cartier Divisors, and Polyhedral Geometry

The ReLU fan $\Sigma_{f_{\theta}}$ arises from the regions of linearity induced by the layerwise hyperplane arrangement of the network. Each maximal cone represents a distinct activation pattern, and the collection of cones encodes all possible combinatorial outcomes of the activation function, namely $\operatorname{ReLU}(x) = \max(0,x)$ (Fu, 7 Sep 2025).

The associated ReLU Cartier divisor $D_f$ is a Q–Cartier divisor supported on the fan, with local data extracted from the slopes of $f$ in each region. The divisor encodes the “bending” of the function across cell boundaries, formalizing the geometry of the network output in the same manner that a divisor on a toric variety encodes the geometry of an algebraic hypersurface [(Fu, 7 Sep 2025), Definition 4.4].

This formalism is mirrored in classical toric geometry, where cones encode monomial supports and polytopal data drives solution behavior for polynomial systems (Telen, 2022). The stratification by cones matches the partitioning of the input space into regions of activation.

4. Criteria for Neural Network Function Realizability

The toric geometry framework yields both necessary and sufficient conditions for a piecewise linear function $f$ to be exactly realizable by a given architecture:

• The network output $f$ must yield constant intersection numbers between its Cartier divisor $D_f$ and torus-invariant curves corresponding to the codimension-one walls (hyperplanes) of the fan.
• In particular, for unbiased shallow (one-hidden-layer) ReLU networks, realizability is characterized by these intersection numbers being constant along all “full” hyperplanes—the combinatorial symmetry among such hyperplanes must be preserved [(Fu, 7 Sep 2025), Theorems 5.1, 5.2].

The approach generalizes the classical Newton polytope analysis (via Kushnirenko’s theorem (Telen, 2022)) to the ReLU setting: the count and arrangement of regions, together with divisor and intersection theory, govern exact expressivity and realizability criteria.

5. Tropical Geometry and Newton Polytopes

A substantial connection exists between tropical geometry and the toric description of ReLU networks:

• When the output $f$ of the network is a tropical polynomial, $$f(x) = \max\{c_1 + \langle u_1, x \rangle, \ldots, c_r + \langle u_r, x \rangle\}$$, the Newton polytope $\operatorname{Newt}(f)$ coincides (up to sign) with the toric polytope $P_{-D_f}$ assigned to the divisor $-D_f$ on the toric variety [(Fu, 7 Sep 2025), Theorem 6.1].
• The mixed volume of the Newton polytope governs counts of solutions to associated systems, and this volume matches precisely with that of the toric polytope in the tropical case [(Fu, 7 Sep 2025), Theorem 6.2].

This establishes that the toric perspective subsumes tropical methods, generalizing geometric analysis to network functions that are tropical rational functions, not merely polynomials.

6. Computational and Algebraic Implications

The ReLU toric variety framework provides a concrete combinatorial model for analyzing the partitioning of the input, solution counts, and expressivity of neural networks:

• Algorithms leveraging polyhedral homotopies and Cox’s homogeneous coordinate approach become available (see (Telen, 2022)), with computational efficiency governed by the fixed support of the network-induced fan.
• Geometric invariants (degree, Hilbert polynomial, intersection numbers) of the ReLU toric variety translate into explicit bounds or formulae for the number and type of functions realizable by networks, echoing computations for classical toric varieties.
• The encoding of redundancy and symmetry by the class group grading and GIT quotient structure in Cox’s construction methodically captures network parameter space symmetries and activation pattern combinatorics (Telen, 2022).

A plausible implication is that network optimization and learning dynamics may be further informed and constrained by these algebraic invariants, particularly when computationally tractable descriptions exist.

7. Connections to Toric Variety Classification and Homotopy Theory

Complete smooth toric varieties whose rational homotopy is of elliptic type feature a cohomology algebra that is a complete intersection concentrated in even degrees (Biswas et al., 2019):

• $$H^*(X, \mathbb{Q}) \simeq \mathbb{Q}[x_1, \ldots, x_n]/(p_1,\ldots,p_n)$$ with $\deg(x_i) = 2$.
• The Poincaré polynomial matches that of a product of complex projective spaces.
• The intrinsic formality (cohomology ring fully determines rational homotopy type) simplifies computation and classification of global network invariants.

This suggests deeper topological invariants emerging from the ReLU toric variety construction: cohomological simplicity may translate to reductions in the complexity of invariant extraction from network-induced varieties and provide benchmarks or test cases in computational and applied algebraic topology.

---

The synthesis of ReLU activation-induced polyhedral partitioning and toric algebraic geometry establishes a precise dictionary for studying neural network function realization and expressivity. The use of divisors, intersection theory, combinatorial fan structures, and homotopical invariants offers a powerful geometric lens for both theoretical investigation and computational application in deep learning and beyond.