Cannistraci-Hebb Training in Neural Networks

• Cannistraci-Hebb Training (CHT) is a brain-inspired dynamic sparse training method that adapts network topology via local-community connectivity patterns.
• It achieves ultra-high sparsity—with connectivity as low as 1%—across architectures like deep spiking neural networks, transformers, and large language models while preserving accuracy.
• CHT significantly reduces computation and energy costs by systematically coupling synaptic pruning with regrowth, making it ideal for resource-constrained deployments.

Cannistraci-Hebb Training (CHT) is a brain-inspired, topology-driven dynamic sparse training (DST) method for artificial and spiking neural networks that systematically couples synaptic pruning and regrowth via local-community connectivity patterns. CHT operationalizes the Cannistraci-Hebb (CH) theory, an epitopological model in which network connectivity adapts not only through synaptic weights but also by direct rewiring according to the mesoscopic organization of local communities. This approach achieves ultra-high sparsity (down to 1% connectivity) with minimal or no accuracy loss across a diverse range of machine learning tasks, including deep SNNs, transformers, and LLMs, while offering substantial reductions in computation and energy requirements (Hua et al., 5 Nov 2025, Zhang et al., 31 Jan 2025).

1. Theoretical Foundations and CH Theory

Hebbian learning encompasses both weight plasticity and epitopological (structural) plasticity. While traditional Hebbian updates modify synaptic strengths (“neurons that fire together wire together”), Cannistraci-Hebb theory emphasizes the emergence and reinforcement of local communities: links are preferentially added between nodes that share tightly interlinked neighbors, forming “tunnel-to-ring” or local-ring structures. The CH index formalizes this in link prediction for complex networks:

$$CH(i,j) = \sum_{k \in \Phi(i) \cap \Phi(j)} \frac{|\Phi(k) \cap (\Phi(i) \cap \Phi(j))| } {|\Phi(k)|}$$

where $\Phi(i)$ denotes the 1-hop neighborhood of node $i$. The CH index rewards candidate links whose prospective local-community is highly interconnected and well-isolated from the broader network, a structural motif that often occurs in natural (hyperbolic) graphs such as neural and social networks (Muscoloni et al., 2017).

2. Canonical CHT Algorithm and Network Automaton

The canonical CHT algorithm proceeds through alternating pruning and regrowth phases at each DST update:

• Pruning: Links are ranked for removal using a hybrid link-removal score (LRS) that mixes weight magnitude and relative-importance terms. In SNNs, this takes the form

$$\mathrm{LRS}_{ij} = \frac{|W_{ij}|}{1+\sum_{i'} |W_{i'j}|} + \frac{|W_{ij}|}{1+\sum_{j'}|W_{ij'}|}$$

A fraction $\zeta$ of links are probabilistically chosen for removal, followed by neuron percolation (removal of in-/out-isolated neurons).

• Regrowth: New links are stochastically sampled based on their CH-derived topological score. In path-based CHT [CH3-L3, $CH_3$-$L_3^p$], candidate (non-edge) pairs $u,v$ are scored via:

$$CH3\!-\!L3(u, v) = \sum_{z_1, z_2 \in \ell_3(u, v)} \frac{1}{\sqrt{(1 + de_{z_1})(1 + de_{z_2})}}$$

where $de_{z_k}$ counts external links of intermediate nodes.

• Selection: Top-$K$ links are regrown based on CH scores, or — in later variants — sampled proportionally, ensuring a balance between exploration and exploitation.
• Sparsity Maintenance: The process is iterated with target structural sparsity $S_s$ held constant via alternating pruning and regrowth (Hua et al., 5 Nov 2025, Zhang et al., 31 Jan 2025, Muscoloni et al., 2017).

3. Advances: Soft Sampling, Matrix-Based CH, and Density Decay

Two key limitations of path-based CHT are (i) $O(Nd^3)$ time complexity (for $N$ nodes, degree $d$), and (ii) early-training rigidity due to strict top-$K$ selection. Innovations address these issues as follows (Zhang et al., 31 Jan 2025):

• CHT soft rule (CHTs): Instead of deterministic regrowth, CHTs employs multinomial soft sampling over candidate links, adjusting a “softness” temperature from $\delta=0.5$ to $0.75$ throughout training. This regularizes the regrowth step, preventing topological local minima and encouraging greater structural exploration:
  • Pruning: sample $\zeta |A|$ links for removal with probability proportional to a parametric hybrid score.
  • Regrowth: sample $K$ links for regrowth with probability $$P(u, v) = S_{\text{grow}}(u,v) / \sum S_{\text{grow}}$$ where $S_{\text{grow}}$ is the matrix-based CH2-L3n score.
• Matrix-based approximation (CH2-L3n): By exploiting block matrix multiplications, the regrowth score is computed as a sum over intermediate neighbors in $O(N^3)$ time, enabling efficient GPU acceleration at moderate densities.
• Sigmoid gradual density decay (CHTss): To mitigate instability associated with rapid sparsification, CHTss adopts a smoothed logistic schedule for reducing network density, further improving ultra-sparse regime convergence.

4. Four-Stage Framework in CH-SNN

The CH-SNN instantiation applies CHT in deep SNNs via four modular stages (Hua et al., 5 Nov 2025):

1. Sparse Spike Correlated Topological Initialization (SSCTI): At the input layer, pairwise node correlations (phi coefficient) are computed on binary spike trains, and only the highest-correlation edges are retained. For deeper layers, a uniform random “fan-in” mask achieves the target degree per neuron.
2. Sparse Spike Weight Initialization (SSWI): Nonzero weights are drawn from a Gaussian with variance

$$\sigma^2_l = \begin{cases} \frac{S_t}{n(1-S_s)}, & l=1, \ \frac{\theta^2 \sqrt{\pi}}{\sqrt{2}e^{1/2} n(1-S_s)}, & 1 < l < L, \ \frac{\theta^2 \sqrt{\pi}}{\sqrt{2}e^{1/2} n}, & l=L, \end{cases}$$

ensuring signal variance preservation across layers (here $S_t$ is spike activity, $S_s$ structural sparsity, $n$ fan-in, $\theta$ threshold).

1. Hybrid Link Removal Score (LRS): See above.
2. CH3-L3 Automaton Regrowth: See above.

Dynamic alternation of these steps yields ultra-sparse yet performant SNNs.

5. Empirical Results and Hardware Impact

Extensive benchmarks demonstrate that CHT delivers superior ultra-sparse performance across domains (Hua et al., 5 Nov 2025, Zhang et al., 31 Jan 2025):

Spiking Neural Networks:

• On MNIST, CH-SNN achieves 97.75% sparsity and 98.97% accuracy, exceeding full-connectivity performance. On CIFAR-10/CIFAR-100, CH-SNN maintains 74.5–74.6% link sparsity at accuracy within 0.1–0.3% of FC baselines, significantly outpacing other methods such as SD-SNN and Grad R.

Dataset	Method	Link sparsity	Test accuracy	Δ vs FC
CIFAR-10	CH-SNN	74.6 %	94.60 %	–0.14 %
CIFAR-100	CH-SNN	74.5 %	75.22 %	+3.16 %

Ablation: Removing SSCTI or SSWI modules destabilizes convergence and sharply degrades accuracy, especially below 1% density.

Hardware: On neuromorphic chips (ANP-I), CH-SNN achieves up to 97.5× reduction in synaptic operations and more than 50× drop in energy consumption at 95% sparsity, with per-inference energy falling from ∼948 mJ (FC) to ∼48 mJ (CH-SNN) (Hua et al., 5 Nov 2025).

Other Architectures: CHTs/CHTss extend these gains to MLPs, transformers, and LLMs:

• At 99% sparsity, CHTs attains up to +8.4% accuracy margin over FC on CIFAR-10 (Zhang et al., 31 Jan 2025).
• On transformers (e.g., Multi30k), CHTss at 5% density achieves BLEU scores higher than dense FC and outperforms all other DST baselines.
• In LLaMA-1B, CHTss at 70% density matches or surpasses FC perplexity; at 30% density, it surpasses both FC and DST competitors in both perplexity and zero-shot GLUE metrics.

6. Complexity Analysis, Latent Geometry, and Comparative Perspective

Classical CHT’s $O(Nd^3)$ time limits applicability to ultra-sparse regimes. Matrix-based CHT (CH2-L3n) reduces this to $O(N^3)$, with empirical speed-up on modern hardware once density is moderate (Zhang et al., 31 Jan 2025).

CHT exploits properties associated with hyperbolic latent geometry—scale-free degree distributions and high clustering. Its performance is robust or even superior to global link inference methods (e.g., SBM, SPM) on networks exhibiting pronounced local-community structure, but may degrade on Euclidean or random graphs (Muscoloni et al., 2017).

Comparative studies establish that:

• SPM is the strongest global link predictor on small, non-hyperbolic graphs, but CH surpasses SPM as network scale increases or latent geometry is hyperbolic.
• CHT’s local rule is “parameter-free,” computationally efficient ($O(N^2)$ on sparse graphs) and matches or exceeds the accuracy of conventional global methods in large, realistically structured networks.

7. Practical Integration and Future Directions

The CHT framework is generally applicable to any neural architecture with learnable adjacency. Its non-gradient, topology-based regrowth allows deployment where gradient-flow is noisy or poor (e.g., deep SNNs). For future high-sparsity scenarios, research continues into further algorithmic acceleration, generalization of local-community paradigms, and hybridization with global inference for regimes where underlying geometry departs from hyperbolicity (Hua et al., 5 Nov 2025, Zhang et al., 31 Jan 2025, Muscoloni et al., 2017).

CHT’s modular structure—comprising initialization, pruning, regrowth, and (optionally) soft or density-annealed scheduling—provides a template for structurally efficient, biologically meaningful DST. Its demonstrated benefits include energy and computation savings at scale, robust convergence in ultra-sparse regimes, and compatibility with both spiking and conventional deep architectures.