Heterogeneous Connectivity in Sparse Networks: Fan-in Profiles, Gradient Hierarchy, and Topological Equilibria

Abstract: Profiled Sparse Networks (PSN) replace uniform connectivity with deterministic, heterogeneous fan-in profiles defined by continuous, nonlinear functions, creating neurons with both dense and sparse receptive fields. We benchmark PSN across four classification datasets spanning vision and tabular domains, input dimensions from 54 to 784, and network depths of 2--3 hidden layers. At 90% sparsity, all static profiles, including the uniform random baseline, achieve accuracy within 0.2-0.6% of dense baselines on every dataset, demonstrating that heterogeneous connectivity provides no accuracy advantage when hub placement is arbitrary rather than task-aligned. This result holds across sparsity levels (80-99.9%), profile shapes (eight parametric families, lognormal, and power-law), and fan-in coefficients of variation from 0 to 2.5. Internal gradient analysis reveals that structured profiles create a 2-5x gradient concentration at hub neurons compared to the ~1x uniform distribution in random baselines, with the hierarchy strength predicted by fan-in coefficient of variation ($r = 0.93$). When PSN fan-in distributions are used to initialise RigL dynamic sparse training, lognormal profiles matched to the equilibrium fan-in distribution consistently outperform standard ERK initialisation, with advantages growing on harder tasks, achieving +0.16% on Fashion-MNIST ($p = 0.036$, $d = 1.07$), +0.43% on EMNIST, and +0.49% on Forest Cover. RigL converges to a characteristic fan-in distribution regardless of initialisation. Starting at this equilibrium allows the optimiser to refine weights rather than rearrange topology. Which neurons become hubs matters more than the degree of connectivity variance, i.e., random hub placement provides no advantage, while optimisation-driven placement does.

• The paper introduces Profiled Sparse Networks (PSN) that use deterministic fan-in profiles to explicitly control neuron connectivity in sparse ANNs.
• It rigorously compares static heterogeneous connectivity with uniform random sparsity across tasks, showing negligible accuracy differences under matched capacities.
• It demonstrates that dynamic sparse training benefits from initializing with equilibrium lognormal profiles, yielding measurable accuracy improvements on challenging tasks.

Heterogeneous Connectivity in Sparse Networks: Fan-in Profiles, Gradient Hierarchy, and Topological Equilibria

Introduction

The paper "Heterogeneous Connectivity in Sparse Networks: Fan-in Profiles, Gradient Hierarchy, and Topological Equilibria" (2604.10560) introduces Profiled Sparse Networks (PSN), an architectural framework for sparse artificial neural networks (ANNs) that explicitly parameterizes neuron connectivity with deterministic fan-in profiles. Unlike existing static or dynamic sparse training methods, PSN makes capacity heterogeneity a direct architectural variable by assigning neurons with continuous, nonlinear functions that control their receptive field cardinalities. The core research question is whether the introduction of deterministic, heterogeneous connectivity offers superior inductive bias or improved accuracy compared to uniform random sparsity, especially when parameter count and overall sparsity are constrained to be identical.

Profiled Sparse Networks: Methodology and Theoretical Foundations

PSN defines numerous parametric connectivity profiles for assigning fan-in distributions—linear, quadratic, exponential, bell, and inverse variants—across network layers, as well as power-law and lognormal families, the latter parameterized directly by the coefficient of variation (CV) to control the degree of heterogeneity.

In constructing sparse connectivity, fan-in is determined by profile functions and matches the desired mean according to global sparsity, but the realization may diverge due to clamping on minimum fan-in constraints. Input selection is handled via even or random spreading, the former using deterministic dispersion with a golden-ratio-based offset to ensure equitable input coverage, avoiding the catastrophic layer collapse and input starvation observed with naive sequential strategies.

The study carefully reconciles initialization for neurons with heterogeneous fan-in, evaluating per-neuron versus mean fan-in scaling, and adopts the latter for stability, in conjunction with LayerNorm, for consistent training dynamics. This design enables empirical dissection of the interaction between architectural capacity distribution and performance, with capacity distribution (fan-in) and input coverage (mask construction) decoupled experimentally.

Figure 1: PSN methods overview at 90% sparsity for a 784-to-1024 layer—panel (a) shows eight parametric fan-in profiles, (b) varying lognormal tail behavior with CCV, (c) example binary mask, and (d) multi-peak profile interpolation modulating CCV.

Experimental Evaluation and Results

Static Sparse Networks: Impact of Heterogeneous Fan-in

Across four datasets (MNIST, Fashion-MNIST, EMNIST-Balanced, Forest Cover), PSN is rigorously evaluated at sparsity levels spanning 80–99.9% in standard MLPs with matched hyperparameters. The principal outcome is the absence of statistically significant accuracy differences between any of the parametric or lognormal heterogeneous profiles and the uniform random connectivity baseline at matched parameter counts. This null result holds across all profile shapes, degrees of heterogeneity (fan-in CV from 0 to 2.5), and datasets, with inter-profile accuracy differences almost always within inter-seed stochastic variability.

Figure 2: (a) Test accuracy versus sparsity for multiple PSN profiles and random baseline on MNIST shows all profiles clustering within 0.2% to 98% sparsity; (b) Continuous interpolation of fan-in CV from highly heterogeneous to uniform demonstrates negligible effect on accuracy; (c) Accuracy relative to dense network at 90% sparsity across all datasets, with all profiles residing within ±0.5% of baseline except lognormal CV=2.5 on Forest Cover, which is limited by input dimensionality.

Extreme fan-in heterogeneity (e.g., CV > 3), where almost all connections are funneled into a tiny subset of neurons, does yield a measurable—but small—accuracy deficit, confirming that pathological capacity concentration is harmful when combined with limited input coverage. Moreover, the spatial arrangement of hubs (their position within a layer) or the symmetry of profiles (reversing hub and specialist assignments) is empirically irrelevant in these settings.

Gradient Hierarchy and Topological Equilibria

A central finding is that the gradient magnitude hierarchy is structurally determined by the degree of fan-in heterogeneity. Internal analysis reveals that as the fan-in CV increases, the ratio of gradient magnitudes between high- and low-connectivity (hub vs. specialist) neurons grows proportionally, with a Pearson correlation $r=0.93$. This hierarchy, however, does not translate into accuracy improvement on the tested tasks; it is a mathematical consequence of mask geometry, not a functional benefit.

Dynamic Sparse Training with PSN-Style Initialization

Leveraging the observation that dynamical sparse training methods such as RigL consistently evolve towards characteristic, dataset- and architecture-specific equilibrium fan-in CVs, the authors evaluate whether initializing RigL with lognormal profiles that match these equilibrium CVs confers any advantage versus standard ERK or uniform initializations. Here, matching the initialization to the RigL equilibrium topological statistics yields consistent performance improvements, especially on more challenging tasks (Fashion-MNIST, EMNIST, Forest Cover), with effect sizes up to +0.49% accuracy on Forest Cover. These improvements are monotonic with increased problem difficulty and sparsity; initializing at above or below equilibrium CVs yields systematically lower final accuracy, and starting at equilibrium bypasses early-topology search, enabling the optimiser to focus resources on weight refinement.

Figure 3: RigL test accuracy versus sparsity for six initialization strategies—equilibrium lognormal initialization consistently achieves the highest accuracy on more challenging tasks and high sparsity, with ERK initialization performing poorly when input dimensionality induces bottlenecks.

Implications and Theoretical Reflections

The principal implication is that, at least for overparameterized MLPs and classification tasks with moderate input dimensionality, the detailed structure of static sparse connectivity (uniform vs. heterogeneous distribution) is largely immaterial to test-time accuracy when overall capacity is sufficient and hub assignment is unaligned with the task structure. This is conceptually consistent with random projection theory, where sufficiently high-dimensional random linear subspaces approximately preserve the data geometry, provided capacity exceeds the intrinsic dimensionality of the task. The critical regime where static topological design may matter is when average fan-in approaches the intrinsic dimensionality or in tasks with complex structure or when capacity is genuinely limiting.

In contrast, for dynamic sparse training schemes, exploiting knowledge of the long-term equilibrium structure to design initialization profiles can streamline convergence, reduce the overhead of topological adaptation, and incrementally improve final accuracy, particularly as problem hardness and sparsity increase. This finding bridges the divide between purely emergent (e.g., SET, RigL) and purely designed (e.g., PSN, block sparse, CHT/DNM) sparse architectures by showing that structural endpoints of dynamic methods can inform analytically derived initialization.

Limitations and Future Research Directions

The empirical results are restricted to MLPs on classification datasets where dense baselines are nearly saturated. The study does not address convolutional or transformer-based architectures, large-scale language modeling, or reinforcement learning, all of which may have different sensitivity to capacity allocation and sparse network structure. Moreover, the operationalization of sparsity—in-place masking without computational kernel optimization—means that practical speedups are not realized, only architectural insights are gained.

The small but consistent improvements observed for equilibrium-matched dynamic sparse initialization on harder tasks motivate further investigation at higher scales and in tasks where task-aligned connectivity could be inferred or designed, which may unlock practical gains in real-world settings.

Conclusion

This work provides a comprehensive empirical and theoretical characterization of the role of heterogeneous fan-in distributions in sparse neural networks. It rigorously establishes that, for matched parameter budgets and unaligned profile assignments, static heterogeneous connectivity does not improve test accuracy over random connectivity. The analysis reveals a strong link between structural heterogeneity and gradient concentration but confirms this geometric property does not independently confer functional benefit. In dynamic schemes (e.g., RigL), initializing masks with a lognormal fan-in profile matched to the equilibrium distribution consistently enhances accuracy on difficult tasks and high sparsity, supporting the use of analytical topological priors for efficient sparse neural network training. These findings delineate the boundaries of utility for static vs. dynamic structure and suggest that future work should generalize PSN-style design principles to architectures and tasks where connectivity may play a more critical role in performance.