ReLU Transition Graph

• ReLU Transition Graph is a combinatorial representation where nodes correspond to linear regions defined by fixed ReLU activation patterns and edges denote transitions by a single activation flip.
• It captures the geometric and combinatorial structure of ReLU networks, linking facet-sharing polyhedral regions with graph theoretic properties like connectivity and spectral gaps.
• RTGs support both exact and sampling-based constructions, providing practical insights into network expressivity, generalization, robustness, and potential compression strategies.

Searching arXiv for the cited RTG papers to ground the article in the current literature. arXiv search query: "ReLU Transition Graph" The ReLU Transition Graph (RTG) is a combinatorial representation of the piecewise-linear geometry induced by a ReLU network. In the now-standard formulation, each node corresponds to a linear region determined by a fixed activation pattern, and edges connect regions that are adjacent across a common facet, equivalently regions whose activation patterns differ by exactly one ReLU flip (Dhayalkar, 16 May 2025). This viewpoint shifts analysis away from counting linear regions in isolation and toward the structure of transitions among them, thereby linking polyhedral geometry, graph theory, expressivity, generalization, robustness, and compression within a single object (Dhayalkar, 3 Sep 2025). Closely related antecedents include the dual graph of the ReLU-induced polyhedral decomposition (Liu et al., 2023) and earlier graph-based optimization formalisms for ReLU training that use factor graphs and slack variables rather than region adjacency (Zhang et al., 2017).

1. Formal definition and geometric semantics

Let $f:\mathbb{R}^d \to \mathbb{R}^k$ be a fully-connected feedforward ReLU network with total number of ReLU units $m$. For an input $x$, its activation pattern is the binary vector

$$a(x)\in\{0,1\}^m,\qquad a_i(x)=\mathbf{1}[z_i(x)>0],$$

where $z_i(x)$ is the pre-activation of neuron $i$ (Dhayalkar, 16 May 2025). Each pattern determines a linear region

$R_a=\{x\in\mathbb{R}^d \mid a(x)=a\},$

and on each such region the network reduces to a single affine map. The input space is therefore partitioned into convex polyhedral cells indexed by activation patterns (Dhayalkar, 16 May 2025).

The RTG is defined as

$G=(V,E),$

with one vertex for each linear region $R_a$. Two vertices are adjacent when the corresponding regions share a common $(d-1)$-dimensional facet,

$m$0

which the literature also states as the equivalent Hamming-one condition

$m$1

Thus, traversing an edge corresponds to crossing exactly one ReLU hyperplane while all other neuron states remain unchanged (Dhayalkar, 16 May 2025).

A closely aligned formulation appears in work on polyhedral decompositions of ReLU networks. There, the network induces a finite decomposition of input space into convex polyhedra labeled by binary activation vectors, and the associated dual graph has vertices corresponding to maximal-dimensional polyhedra and edges corresponding to shared facets (Liu et al., 2023). That work explicitly states that two polyhedra share a facet if and only if their associated binary vectors differ in exactly one bit, which is precisely the RTG adjacency rule (Liu et al., 2023).

An extended graph-theoretic interpretation treats the RTG as a subgraph of the $m$2-dimensional hypercube, where $m$3 is the total number of hidden ReLU units and only realized activation patterns are retained as vertices (Dhayalkar, 3 Sep 2025). This emphasizes that RTG adjacency is simultaneously geometric, via polyhedral facet-sharing, and combinatorial, via one-bit flips in activation space.

2. Polyhedral structure and exact region encoding

The polyhedral character of RTGs follows from the fact that a ReLU network is piecewise affine. For an $m$4-layer feedforward ReLU network,

$m$5

the activation pattern $m$6 is formed by concatenating the binary states of all hidden units (Liu et al., 2023). For a fixed pattern, the corresponding region can be written as a system of linear inequalities

$m$7

so each realized pattern determines a convex polyhedron, possibly empty or unbounded (Liu et al., 2023).

On each cell $m$8, the network is affine: $m$9 This makes the RTG the adjacency graph of the affine pieces of the network (Liu et al., 2023). In the dual-graph language, active inequalities correspond to facets of the polyhedron, and the number of active bits equals the number of nearest neighbors of that polyhedron in the transition graph (Liu et al., 2023).

A useful consequence is that RTG geometry is more informative than region count alone. Two networks may realize the same number of regions but differ in how those regions touch, cluster, or fragment. This suggests that RTG structure captures the connective geometry of the partition rather than only its cardinality, a distinction made explicitly in the RTG literature (Dhayalkar, 16 May 2025).

3. Construction methods and computational realization

Two broad construction paradigms appear in the literature. The first is exact and exploits the polyhedral description. One may brute-force all $x$0 candidate bit vectors, build the corresponding inequalities $x$1, and test feasibility; feasible patterns correspond to realized regions (Liu et al., 2023). A more efficient traversal-and-pruning method starts from a random input, computes its activation vector, identifies the active bits of its region by LP-based redundancy testing, flips each active bit one at a time, and recursively explores neighboring regions. This is explicitly a graph walk on the dual graph/RTG (Liu et al., 2023).

The second construction paradigm is sampling-based. One samples inputs $x$2, computes activation patterns $x$3, collects all unique patterns, makes each unique pattern a node, and adds an undirected edge whenever two patterns have Hamming distance $x$4 (Dhayalkar, 16 May 2025). In one set of experiments this was done on dense $x$5-D grids using synthetic inputs on a uniform grid in $x$6, with 100,000 input points and 5 random seeds, making approximation of the full region structure feasible (Dhayalkar, 16 May 2025). Another study used a $x$7 grid over $x$8, grouped points by activation pattern, and connected nodes when the patterns differed in exactly one bit (Dhayalkar, 3 Sep 2025).

The exact and sampled constructions play different roles. Exact enumeration gives the genuine polyhedral adjacency graph for small networks, while sampled RTGs approximate the region structure operationally through observed activation patterns. In the sampled setting, Hamming-one adjacency is the operational rule; in the exact geometric setting, adjacency is facet-sharing (Dhayalkar, 16 May 2025). The papers treat these as equivalent under the stated genericity assumptions for realized full-dimensional regions (Liu et al., 2023).

The literature also uses Hamming distance more generally as a coarse metric. Because neighboring regions differ by one bit, Hamming distance between activation patterns approximates shortest-path distance in the dual graph, thereby inducing a network-driven distance on data points that is distinct from Euclidean distance (Liu et al., 2023).

4. Theoretical properties of RTGs

A central result is a size bound on the number of RTG nodes. For a fully connected ReLU network with $x$9 hidden layers, each of width at most $$a(x)\in\{0,1\}^m,\qquad a_i(x)=\mathbf{1}[z_i(x)>0],$$0, and input dimension $$a(x)\in\{0,1\}^m,\qquad a_i(x)=\mathbf{1}[z_i(x)>0],$$1,

$$a(x)\in\{0,1\}^m,\qquad a_i(x)=\mathbf{1}[z_i(x)>0],$$2

This follows from hyperplane arrangement theory and supports the interpretation that region count grows polynomially with width and exponentially with depth (Dhayalkar, 16 May 2025).

Connectivity is another foundational property. Under mild assumptions—continuity of the network and non-degenerate neuron hyperplanes—the RTG is connected: for any two regions there exists a finite sequence of regions in which each consecutive pair shares a facet (Dhayalkar, 16 May 2025). The proof uses a straight-line path in input space and the fact that only finitely many hyperplanes are crossed (Dhayalkar, 16 May 2025).

Several works then attach graph-theoretic quantities to learning-theoretic notions. One paper defines the diameter

$$a(x)\in\{0,1\}^m,\qquad a_i(x)=\mathbf{1}[z_i(x)>0],$$3

and claims

$$a(x)\in\{0,1\}^m,\qquad a_i(x)=\mathbf{1}[z_i(x)>0],$$4

giving a graph-theoretic interpretation of capacity via transition distance (Dhayalkar, 16 May 2025). Another paper defines region entropy

$$a(x)\in\{0,1\}^m,\qquad a_i(x)=\mathbf{1}[z_i(x)>0],$$5

and states

$$a(x)\in\{0,1\}^m,\qquad a_i(x)=\mathbf{1}[z_i(x)>0],$$6

thereby turning occupancy of activation regions under the data distribution into a data-aware expressivity measure (Dhayalkar, 3 Sep 2025).

Additional structural claims extend RTGs into discrete functional geometry. At random initialization, RTGs are stated to exhibit expander behavior with high probability as width grows, degree distributions $$a(x)\in\{0,1\}^m,\qquad a_i(x)=\mathbf{1}[z_i(x)>0],$$7, and spectral properties relevant to generalization (Dhayalkar, 3 Sep 2025). The same work formulates a spectral-gap bound: $$a(x)\in\{0,1\}^m,\qquad a_i(x)=\mathbf{1}[z_i(x)>0],$$8 where $$a(x)\in\{0,1\}^m,\qquad a_i(x)=\mathbf{1}[z_i(x)>0],$$9 is the second eigenvalue of the normalized Laplacian (Dhayalkar, 3 Sep 2025).

A sparsity statement also appears in the earlier RTG literature: there exists a subset $z_i(x)$0 with

$z_i(x)$1

such that every $z_i(x)$2 satisfies

$z_i(x)$3

where $z_i(x)$4 is the average degree (Dhayalkar, 16 May 2025). This is used as a structural basis for later compression arguments.

5. Generalization, robustness, compression, and topology

RTG-based analyses seek to replace or supplement parameter-count and norm-based complexity measures with structural statistics of the region graph. One such statistic is RTG entropy,

$z_i(x)$5

with $z_i(x)$6 proportional to region volume (Dhayalkar, 16 May 2025). A related lemma states

$z_i(x)$7

linking entropy to average degree and interpreting dense transition structure as more fragmented partitioning (Dhayalkar, 16 May 2025).

Compression is formalized in two complementary ways. One theorem states that there exists a pruned or core network $z_i(x)$8 such that

$z_i(x)$9

for all $i$0 outside a negligible boundary set of measure zero, while

$i$1

for some $i$2 (Dhayalkar, 16 May 2025). A second formulation introduces edge-wise KL divergence across adjacent regions,

$i$3

and claims

$i$4

so local functional smoothness across RTG edges becomes a compression- and generalization-relevant quantity (Dhayalkar, 3 Sep 2025).

Robustness is treated structurally rather than adversarially. If moving between behaviors requires many single-neuron flips, the network has a more intricate transition geometry; if neighboring regions have small output divergence, the network is smoother across boundaries (Dhayalkar, 16 May 2025, Dhayalkar, 3 Sep 2025). This suggests that RTGs offer a graph-local description of stability under perturbation.

RTGs have also been used as a scaffold for topological data analysis. By mapping samples to activation patterns, computing pairwise Hamming distances, and building Vietoris–Rips filtrations, one can recover homological signals of manifolds in the input space from samples (Liu et al., 2023). In that setting, the RTG or dual-graph geometry functions as a coarse quantization of input space that is nevertheless sufficiently robust for persistent homology (Liu et al., 2023).

6. Related graph formalisms, empirical findings, and scope

The term ReLU Transition Graph should be distinguished from other graph-based uses of ReLU structure. In “Training Deep Neural Networks via Optimization Over Graphs,” ReLU and DCutLU activations are rewritten using slack variables $i$5, equality constraints, and indicator penalties, producing a factor graph

$i$6

on which ADMM updates are performed layerwise (Zhang et al., 2017). In that framework, the graph represents optimization variables and constraints rather than adjacency among linear regions. The paper itself does not define a separate RTG with state-transition semantics, although it provides a graph-based constrained reformulation of ReLU behavior (Zhang et al., 2017).

A different use of “transition” appears in the statistical-mechanics analysis of the ReLU soft committee machine. There, the relevant object is not a region-adjacency graph but a phase structure with an unspecialized symmetric phase and a specialized phase, separated by a continuous specialization transition at

$i$7

(Afanah et al., 20 Mar 2026). The paper explicitly frames this as an RTG-like interpretation only if one chooses to view phases as nodes and the critical curve as a transition edge (Afanah et al., 20 Mar 2026). The canonical RTG literature, by contrast, uses nodes for linear regions and edges for single-neuron flips (Dhayalkar, 16 May 2025, Dhayalkar, 3 Sep 2025).

Empirically, RTG studies remain concentrated in controlled low-dimensional settings. One work used fully connected ReLU MLPs in PyTorch on an NVIDIA RTX 4060 GPU, with depth $i$8, width $i$9, synthetic 2D inputs on a uniform grid in $R_a=\{x\in\mathbb{R}^d \mid a(x)=a\},$0, 100,000 input points, and 5 random seeds (Dhayalkar, 16 May 2025). It reported that all tested Hamming-1 pairs were connected, no tested Hamming-$R_a=\{x\in\mathbb{R}^d \mid a(x)=a\},$1 pairs were connected, and nearly all RTGs were connected, with only one extreme case $R_a=\{x\in\mathbb{R}^d \mid a(x)=a\},$2 showing disconnection in finite sampling (Dhayalkar, 16 May 2025). The same study reported modest approximation error after pruning the bottom 50% of nodes by degree, including about $R_a=\{x\in\mathbb{R}^d \mid a(x)=a\},$3 for $R_a=\{x\in\mathbb{R}^d \mid a(x)=a\},$4 and about $R_a=\{x\in\mathbb{R}^d \mid a(x)=a\},$5 for $R_a=\{x\in\mathbb{R}^d \mid a(x)=a\},$6 (Dhayalkar, 16 May 2025).

A second empirical program, also restricted to small 2D-input MLPs, reported mean edge expansion $R_a=\{x\in\mathbb{R}^d \mid a(x)=a\},$7 for random subsets of 10 nodes, spectral gap $R_a=\{x\in\mathbb{R}^d \mid a(x)=a\},$8 together with train accuracy $R_a=\{x\in\mathbb{R}^d \mid a(x)=a\},$9, test accuracy $G=(V,E),$0, and generalization gap $G=(V,E),$1 on a synthetic circular classification task, region entropy rising from $G=(V,E),$2 to $G=(V,E),$3 as width increased from 4 to 1024 while the number of regions rose from 8 to 9998, and empirical degree distributions closely matching fitted binomial laws (Dhayalkar, 3 Sep 2025). These experiments were presented as support for expansion, entropy saturation under overparameterization, and graph-local smoothness claims (Dhayalkar, 3 Sep 2025).

The main limitations are explicit. Current RTG analyses are largely restricted to fully connected ReLU MLPs, synthetic or low-dimensional inputs, and idealized region-volume assumptions for entropy (Dhayalkar, 16 May 2025). Proposed extensions include convolutional networks, residual networks, attention-based architectures, real datasets, and higher-dimensional settings (Dhayalkar, 16 May 2025). A plausible implication is that the principal conceptual contribution of RTGs lies less in a finished universal theory than in establishing a common graph-theoretic language for the polyhedral, combinatorial, and functional structure of ReLU networks.