ReLU Fan: Polyhedral Geometry in Neural Networks

• ReLU Fan is a polyhedral complex that partitions the input space into affine regions based on constant ReLU activation patterns.
• It translates neural network function realization into toric and tropical geometry, linking combinatorial structures with algebraic invariants.
• The geometric framework provides practical criteria for exact function realizability and guides analyses of network expressivity and identifiability.

A ReLU fan is the canonical polyhedral complex—termed a "fan" in toric geometry—arising from the piecewise linear structure of a feedforward ReLU neural network acting on $\mathbb{R}^{n_0}$. The ReLU fan is constructed from the regions of the input space on which the composite function of the network behaves affinely due to the gating effect of the ReLU activations. By recasting neural network function realization in terms of the combinatorics and geometry of the associated fan, the theory exploits algebraic and tropical geometry to provide necessary and sufficient criteria for the exact representability of piecewise linear functions by ReLU networks of a fixed architecture (Fu, 7 Sep 2025).

1. Definition and Geometry of the ReLU Fan

The ReLU fan $\Sigma_f$ associated with a network $f_\theta$ is a rational polyhedral complex in $\mathbb{R}^{n_0}$ corresponding to the partitioning of the input space into regions where the network is affine linear. Formally, each region (cell, cone) of $\Sigma_f$ is defined by the set of input points where the activation pattern of all ReLU neurons is constant. In each such cell, the network’s output admits the representation: $$f_\theta(x) = \langle m_\sigma, x \rangle + b_\sigma, \quad x \in \sigma$$ where $m_\sigma$ is the slope vector and $b_\sigma$ the offset associated with the region $\sigma$.

This structure encodes all non-linearities induced by the ReLU activation: each bent hyperplane corresponding to a neuron is reflected as a wall (codimension-1 face) of the fan, and the intersection patterns of these hyperplanes determine the combinatorics of the fan.

2. Toric Variety and the ReLU Cartier Divisor

Each ReLU fan $\Sigma_f$ defines a toric variety $X_{\Sigma_f}$ (complete and proper, as the fan covers all of $\mathbb{R}^{n_0}$). The output function $f$ of the network naturally serves as a support function for a (rational) Cartier divisor $D_f$ on $X_{\Sigma_f}$, with slope data $(m_\sigma)_{\sigma \in \Sigma_f}$, uniquely determined up to addition of a global affine function.

The divisor $D_f$ encodes the bending at the walls of $\Sigma_f$ and is central to the expressivity and realization problem. Specifically, the divisor’s intersection numbers with torus-invariant curves (corresponding to walls between regions) quantify how $f$ “bends” as one transitions across hyperplane boundaries.

3. Tropical Geometry Connection

ReLU functions are max-affine and, thus, are tropical polynomials when expressed as: $f(x) = \max_{i}\{c_i + \langle u_i, x \rangle\}$ in tropical arithmetic ($\oplus = \max$, $\odot = +$). The Newton polytope $\operatorname{Newt}(f)$ is the convex hull of the exponent vectors $\{u_i\}$.

A central result establishes that the polytope $P_{-D}$ associated to the negative of the ReLU Cartier divisor equals minus the Newton polytope: $P_{-D} = -\operatorname{Newt}(f)$ Thus, tropical convexity of $f$ aligns with convexity conditions on the divisor, allowing algebraic and combinatorial tools to analyze network expressivity.

4. Exact Function Realization Problem

The criterion for a piecewise linear function $f$ to be realizable exactly by an unbiased shallow ReLU network (one hidden layer, rational weights) can be stated in terms of intersection numbers:

• Extend the walls (codimension-1 loci of nonlinearity of $f$) to full hyperplanes, yielding a fan $\Sigma$.
• The output function’s divisor $D_f$ must satisfy: $D_f \cdot V(\tau_1) = D_f \cdot V(\tau_2)$ for any two walls $\tau_1, \tau_2$ lying on the same full hyperplane, where $V(\tau)$ denotes the torus-invariant curve crossing the wall $\tau$.

If these conditions are met, then $f$ is realizable (up to an affine shift) by the network.

5. Intersection Theory and Algebraic Invariants

Intersection numbers $\langle m_\sigma - m_{\sigma'}, u \rangle$—where $u$ is the generator of the shared face between adjacent cones $\sigma, \sigma'$—measure the “bending intensity” across each wall. The criterion above requires uniform intersection numbers for all walls arising from the same neuron (hyperplane), reflecting consistent activation behavior. The volume of the associated line bundle $\mathcal{O}_X(-D)$ is equal to the mixed volume of the Newton polytope: $$\operatorname{Vol}(\operatorname{Newt}(f)) = \operatorname{Vol}(\mathcal{O}_X(-D))$$ giving a direct geometric measure of network capacity.

6. Algorithmic and Combinatorial Structure

From a combinatorial geometry perspective, the ReLU fan provides complete information about the partitioning of input space, the arrangement of bent hyperplanes, and the possible regions of affine behavior. Enumerating activation patterns (as studied in learning-theoretic analyses of “ReLU fan” combinatorics (Bakshi et al., 2018)) allows algebraic geometric reasoning to be directly applied to understanding network identifiability, expressivity, and learning hardness.

In analysis and learning of network weights, the correspondence between fan combinatorics, divisor data, and activation patterns underpins both theoretical results (e.g., hardness criteria, exact recovery) and practical algorithms for recovery, clustering, and interpretability.

7. Significance for Neural Network Theory

The toric geometry perspective via the ReLU fan synthesizes algebraic geometry, tropical geometry, and deep learning. It translates the function realization problem to geometric invariants, facilitates classification of realizable functions, and emphasizes the foundational role of polyhedral combinatorics in neural network expressivity. Furthermore, it unites tropical and algebraic viewpoints for piecewise linear models, allowing both rigorous mathematical analysis and practical application in architecture certification, clustering-based interpretation, and function recovery.

In summary, the ReLU fan is the polyhedral complex governing the affine regions of a ReLU neural network, serving as the bridge between network architecture and toric geometry. It encodes all necessary combinatorial, algebraic, and geometric data to determine exact function realizability, supports the translation of tropical polynomials to toric Cartier divisors, and underlies both theoretical and algorithmic results on the expressive capacity and identifiability of ReLU networks (Fu, 7 Sep 2025).