Gradient Fan-in Asymmetry in Neural Networks

• Gradient Fan-in Asymmetry (GFA) is a phenomenon where uneven gradient aggregation arises from heterogeneous connectivity, creating hubs and specialists in neural networks.
• In sparse networks, GFA is measured by comparing high-fan-in neurons receiving up to 5x larger gradients than low-fan-in ones, highlighting structural influences on learning.
• In residual architectures, GFA manifests as a depth-dependent decline in aggregated gradient channels, guiding design choices for improved gradient utilization.

Gradient Fan-in Asymmetry (GFA) denotes a structural imbalance in backpropagated learning signals induced by heterogeneous aggregation patterns. In sparse feed-forward networks, the term refers to the tendency of high-fan-in neurons to receive systematically larger gradients than low-fan-in neurons, producing a hierarchy between hubs and specialists (Tomczak, 12 Apr 2026). In deep Pre-LayerNorm residual stacks such as Transformers and ResNets, the same term denotes the depth-dependent decline in the number of aggregated gradient channels, so that early layers receive richer gradient information than later ones (Ahmad et al., 25 Jun 2026). Across both settings, GFA is presented as a property of connectivity structure rather than merely parameter count or raw gradient norm, and it is used to explain why networks with similar accuracy can exhibit sharply different internal learning dynamics.

1. Two formalizations of Gradient Fan-in Asymmetry

In sparse neural networks, GFA is defined over neuron-wise fan-in heterogeneity. For a fully-connected layer with $n$ inputs and $m$ outputs, each output neuron $i$ has fan-in

$f_i = |\{j : W_{ij}\neq 0\}|.$

When the set $\{f_i\}$ is heterogeneous, the backward pass concentrates gradient energy at neurons with larger fan-in. High-fan-in neurons function as hubs, while low-fan-in neurons act as specialists. GFA is then the extent to which hubs receive more gradient than specialists (Tomczak, 12 Apr 2026).

In Pre-LayerNorm residual stacks, GFA is defined at the layer level rather than the neuron level. For a model with $N$ residual blocks and hidden states $x_\ell\in\mathbb{R}^d$,

$x_{\ell+1}=x_\ell+F_\ell(x_\ell).$

The backpropagated gradient at layer $\ell$ is decomposed into an identity path and downstream functional paths. The corresponding gradient fan-in is the number of distinct signal channels aggregated at that layer:

$\phi_\ell=(N-\ell)+2.$

This quantity decreases linearly with depth, implying that early layers structurally receive more aggregated gradient channels than late layers (Ahmad et al., 25 Jun 2026).

These two uses of GFA share a common conceptual core: gradient flow is shaped by topology. In one case the asymmetry is induced by heterogeneous per-neuron connectivity inside a sparse layer; in the other it is induced by the residual-path geometry of depth. This suggests that GFA is best understood as a structural account of unequal gradient aggregation rather than as a statement about absolute network sparsity or depth alone.

2. Fan-in profiles and structural parameterization in sparse networks

Profiled Sparse Networks (PSN) replace uniform connectivity with deterministic, heterogeneous fan-in profiles defined by continuous profile functions $m$0 (Tomczak, 12 Apr 2026). A normalized neuron index $m$1 determines relative fan-in, after which the profile is scaled to a target mean fan-in $m$2 and clamped to $m$3. The absolute fan-in is

$m$4

where $m$5 and $m$6.

The paper lists several continuous nonlinear profile families. Examples include linear, quadratic, exponential, and bell-shaped profiles, together with inverse versions obtained by substituting $m$7 (Tomczak, 12 Apr 2026). A complementary family samples fan-in from a lognormal distribution with theoretical parameters

$m$8

then clamps values to $m$9, sorts them descending, and assigns them to neuron indices $i$0. By binary-searching $i$1 to match the desired post-clamp coefficient of variation, this procedure provides precise control over connectivity heterogeneity.

The explicit purpose of these constructions is not merely to vary sparsity, but to separate total parameter budget from the distribution of connectivity across neurons. PSN therefore makes it possible to test whether heterogeneous connectivity alone confers an advantage, and to measure how strongly internal gradient hierarchy scales with fan-in dispersion (Tomczak, 12 Apr 2026).

3. Quantification of gradient hierarchy

In sparse networks, GFA is measured through neuron-wise gradient magnitudes on outgoing weights. Let $i$2 be the weight matrix in layer $i$3, and define

$i$4

Neurons are split into the top half by fan-in, denoted hubs $i$5, and the bottom half, denoted specialists $i$6. The gradient concentration ratio is then

$i$7

Here, $i$8 is a scalar measure of hierarchy strength: values near $i$9 indicate little or no hierarchy, whereas larger values indicate concentration of gradient at hubs (Tomczak, 12 Apr 2026).

The connectivity heterogeneity itself is summarized by the fan-in coefficient of variation,

$f_i = |\{j : W_{ij}\neq 0\}|.$0

Across profiles, the Pearson correlation between $f_i = |\{j : W_{ij}\neq 0\}|.$1 and the hierarchy ratio $f_i = |\{j : W_{ij}\neq 0\}|.$2 is reported as $f_i = |\{j : W_{ij}\neq 0\}|.$3, indicating that stronger fan-in dispersion predicts stronger gradient hierarchy (Tomczak, 12 Apr 2026).

In residual stacks, the corresponding empirical proxy is the accumulated gradient share for layer $f_i = |\{j : W_{ij}\neq 0\}|.$4:

$f_i = |\{j : W_{ij}\neq 0\}|.$5

This quantity is used as an observable proxy for the theoretical fan-in $f_i = |\{j : W_{ij}\neq 0\}|.$6, and post-training functional importance is measured by ablating a block at test time and recording the resulting performance drop (Ahmad et al., 25 Jun 2026).

The metric design in the two papers is notably parallel. Both replace an abstract topological intuition with a measurable hierarchy: sparse-network GFA uses fan-in-conditioned neuron groups and a concentration ratio, while residual-stack GFA uses channel-count fan-in and accumulated gradient share. This suggests a common analytical strategy in which structural asymmetry is first formalized combinatorially and then tested against empirical gradient statistics.

4. Empirical behavior across sparse and residual architectures

For sparse networks, the main empirical finding is that structured fan-in heterogeneity induces pronounced internal gradient hierarchy without conferring a static accuracy advantage when hub placement is arbitrary (Tomczak, 12 Apr 2026). At $f_i = |\{j : W_{ij}\neq 0\}|.$7 sparsity, all static profiles, including the uniform random baseline, achieve accuracy within $f_i = |\{j : W_{ij}\neq 0\}|.$8-$f_i = |\{j : W_{ij}\neq 0\}|.$9 of dense baselines on every dataset. The paper states that this result holds across sparsity levels from $\{f_i\}$0-$\{f_i\}$1, across eight parametric profile families together with lognormal and power-law families, and across fan-in coefficients of variation from $\{f_i\}$2 to $\{f_i\}$3.

The same study reports a detailed monotonic relation between fan-in heterogeneity and gradient concentration. Uniform random sparsity with $\{f_i\}$4 yields $\{f_i\}$5. A quadratic profile with $\{f_i\}$6 gives $\{f_i\}$7; linear with $\{f_i\}$8 gives $\{f_i\}$9; bell with $N$0 gives $N$1; and exponential with $N$2 gives $N$3. Lognormal profiles with $N$4, $N$5, and $N$6 yield $N$7, $N$8, and $N$9, respectively. Summarizing across the profile families at $x_\ell\in\mathbb{R}^d$0 sparsity, gradient concentration at hubs ranges from approximately $x_\ell\in\mathbb{R}^d$1 to approximately $x_\ell\in\mathbb{R}^d$2, versus the approximately $x_\ell\in\mathbb{R}^d$3 baseline of uniform random sparsity (Tomczak, 12 Apr 2026).

For residual architectures, the reported evidence is correlational but spans both Transformers and ResNets. In a vanilla Transformer and ResNet, the per-layer accumulated gradient share follows a near-perfect linear decay, while in LayerSkip, which uses deep supervision, it follows a quadratic decay, matching the theoretical fan-in patterns (Ahmad et al., 25 Jun 2026). Across seeds, the Spearman correlation between accumulated gradient share and post hoc layer importance is $x_\ell\in\mathbb{R}^d$4 with $x_\ell\in\mathbb{R}^d$5 in the vanilla Transformer, $x_\ell\in\mathbb{R}^d$6 in ResNet-50, and $x_\ell\in\mathbb{R}^d$7 in LayerSkip.

Setting	Structural variable	Empirical pattern
Sparse PSN	Neuron fan-in heterogeneity	Hub gradients rise from $x_\ell\in\mathbb{R}^d$8 to $x_\ell\in\mathbb{R}^d$9 as $x_{\ell+1}=x_\ell+F_\ell(x_\ell).$0 increases
Vanilla residual stacks	Layer fan-in $x_{\ell+1}=x_\ell+F_\ell(x_\ell).$1	Accumulated gradient share decays linearly with depth
Deep supervision	Augmented layer fan-in $x_{\ell+1}=x_\ell+F_\ell(x_\ell).$2	Decay becomes quadratic in depth

Taken together, these results indicate that GFA can appear either within layers or across layers. The papers do not present these as identical mechanisms, but both support the more general claim that structural aggregation asymmetry produces measurable learning hierarchies even when headline task performance is only weakly affected.

5. Causal interventions and mechanistic interpretation

The residual-stack formulation of GFA includes two explicit interventions designed to distinguish structure from raw magnitude (Ahmad et al., 25 Jun 2026). The first is gradient-norm equalization, in which a hook rescales each layer’s gradient so that all layers share the same $x_{\ell+1}=x_\ell+F_\ell(x_\ell).$3 norm before the optimizer step. The reported result is that equalization fails to restore late-layer utility and actively degrades it. This is used to argue that amplifying an information-poor gradient does not recreate the missing compositional diversity associated with higher gradient fan-in.

The second intervention is virtual depth via parameter-shared repetition. Starting from an $x_{\ell+1}=x_\ell+F_\ell(x_\ell).$4-layer Transformer, the last four layers are repeated with shared weights in the pattern $x_{\ell+1}=x_\ell+F_\ell(x_\ell).$5, increasing their effective downstream fan-in from $x_{\ell+1}=x_\ell+F_\ell(x_\ell).$6 to $x_{\ell+1}=x_\ell+F_\ell(x_\ell).$7. The paper reports that this structural increase in $x_{\ell+1}=x_\ell+F_\ell(x_\ell).$8 substantially raises deep-layer gradient norms and their functional importance, restoring and even surpassing the relative contribution of shallow layers (Ahmad et al., 25 Jun 2026).

In sparse networks, the mechanistic interpretation is framed less as intervention than as controlled topology variation. Merely reallocating a fixed connectivity budget into hubs and specialists creates strong internal gradient hierarchies, but static hub placement alone does not improve test accuracy (Tomczak, 12 Apr 2026). The paper explicitly concludes that which neurons become hubs matters more than the degree of connectivity variance: random hub placement provides no advantage, whereas optimization-driven placement does.

This suggests a common mechanistic interpretation across the two papers. GFA is not presented as a claim that larger gradients are intrinsically better; rather, it is a claim that richer or more numerous gradient channels can increase functional utility. Where the structure does not align with the task, the hierarchy may be visible internally without producing better accuracy. Where the structure is made adaptive or better matched to downstream information flow, the same asymmetry can become useful.

6. Training implications, architectures, and open questions

In sparse training, GFA is used to motivate topology-aware initialization. Dynamic sparse training methods such as RigL are reported to converge toward a characteristic fan-in distribution regardless of initialization (Tomczak, 12 Apr 2026). When PSN fan-in distributions are used to initialize RigL, lognormal profiles matched to the equilibrium fan-in distribution consistently outperform standard ERK initialization, with advantages that grow on harder tasks: $x_{\ell+1}=x_\ell+F_\ell(x_\ell).$9 on Fashion-MNIST with $\ell$0 and $\ell$1, $\ell$2 on EMNIST, and $\ell$3 on Forest Cover. The interpretation given is that starting at the equilibrium allows the optimizer to refine weights rather than rearrange topology.

In residual architectures, GFA motivates two efficiency methods. CascadeFlow Pruning ranks layers by training-time accumulated gradient share and prunes those with lowest share, requiring no extra calibration set or second pass and yielding more stable rankings with lower perplexity degradation under aggressive pruning than post hoc heuristics (Ahmad et al., 25 Jun 2026). CascadeFormer incorporates GFA directly into architecture design by tapering width with depth. The attention dimension is reduced according to

$\ell$4

and the FFN inner dimension according to

$\ell$5

A high-taper variant, denoted A$\ell$6, matched the perplexity of a vanilla $\ell$7-layer baseline under the same FLOP budget while reducing inference latency by $\ell$8 and increasing throughput by $\ell$9 on an A100 GPU (Ahmad et al., 25 Jun 2026).

Both papers also state important limitations. In the sparse-network setting, heterogeneous connectivity provides no accuracy advantage when hub placement is arbitrary rather than task-aligned (Tomczak, 12 Apr 2026). In the residual-stack setting, it remains open whether gradient magnitude proxies fan-in beyond high-rank regimes, and how the reported dynamics behave at the $\phi_\ell=(N-\ell)+2.$0B$\phi_\ell=(N-\ell)+2.$1 scale; all experiments were conducted on from-scratch models up to $\phi_\ell=(N-\ell)+2.$2B parameters (Ahmad et al., 25 Jun 2026). The residual-stack paper further notes that CascadeFlow Pruning requires access to training gradients and therefore cannot be applied post hoc to closed-source pretrained models.

A common misconception would be to treat GFA as a universal argument for more heterogeneity or for keeping all early layers and removing all late ones. The reported evidence is narrower. In sparse networks, heterogeneity alone changes gradient allocation but not static accuracy when hub placement is arbitrary. In residual stacks, deeper layers often add less value, but their contribution can be restored by increasing downstream path counts. A plausible implication is that GFA is most useful as a diagnostic and design principle for topology-task alignment rather than as a one-directional prescription toward either maximal sparsity heterogeneity or uniformly shallow models.