Self-Organized Neural Coding Theory

• Self-Organized Neural Coding Theory is a framework that defines how neural systems spontaneously develop efficient, sparse, and robust representations via local plasticity and phase transitions.
• It explains the emergence of critical neurons and backbone computational graphs through Hebbian-like correlation graphs and stochastic weight redistribution.
• The theory unifies biological and artificial network principles to achieve high capacity, adaptive generalization, and robust coding in complex tasks.

Self-Organized Neural Coding Theory defines how neural systems—biological or artificial—spontaneously develop structured, efficient, and robust representations of information through local learning rules and network-level phase transitions. This emergent organization endows networks with critical properties: sparse yet powerful coding, scale-free connectivity, high capacity, efficient generalization, and adaptive task-specificity. Central to this theory is the formation of “critical neurons,” the emergence of heavy-tailed structural motifs via dynamical phase transitions, and the role of non-equilibrium dynamics in selecting and stabilizing efficient codes. The theory is grounded in precise mathematical models and is supported by both artificial neural network results and a broad array of neurobiological observations (Liu et al., 28 Aug 2025, Li et al., 2012, Kossio et al., 2018, Ma et al., 2022, Coulter et al., 2009, Malsburg, 2018, Liu et al., 2014, Rinkus, 2017).

1. Critical Components and Mathematical Formalism

Critical Neurons:

A neuron $i$ is defined as critical at iteration $t$ if its activation $a_i(x,t)$ is (a) strongly modulated by the task and (b) highly synchronized with at least one other neuron. Quantitatively, for a mini-batch: $$C_{ij}(t) = \frac{\langle a_i a_j \rangle_x - \langle a_i \rangle_x \langle a_j \rangle_x}{\sigma_i \sigma_j}$$ A neuron $i$ is labeled critical if it appears in any pair with $|C_{ij}| \geq C_{th}$ ($C_{th} = 0.5$ is typical). These “backbone” units concentrate most task-relevant signal, mediate deformation of the loss landscape, and define a sparse “critical computational graph” that acts as the functional core (Liu et al., 28 Aug 2025).

Hebbian-Like Correlation Graphs:

The neuronal correlation graph (NCG) is constructed by binarizing $C_{ij}$ above threshold, yielding a backbone adjacency matrix. Such occurrent pairings drive network self-organization via local plasticity.

Dynamical Phase Transitions:

Early in training, the NCG undergoes a percolation-type second-order phase transition characterized by order parameters:

• Global strength: $S(t)= \lVert C(t) \rVert_F$
• Giant component size: $S_1(t)$
• Mean degree: $\langle k \rangle(t)$

At $t_c \sim 100$, scaling laws are observed: $$S(t_c) \propto M^{3/2},\quad S_1(t_c) \propto M,\quad \langle k \rangle(t_c) \sim 1$$ where $M$ is network size (Liu et al., 28 Aug 2025). Survival probabilities and component sizes near $t_c$ exhibit exponents matching classical finite-network percolation universality.

Stochastic Weight Redistribution:

Weights evolve under mini-batch SGD as a discrete noisy (Langevin) process: $w(t+1) = w(t) - \eta \nabla \hat{L}(w(t)) + \xi(t)$ Correlated critical pairs reinforce their connection weights, while others undergo stochastic redistribution, summarized phenomenologically as: $$w_k \mapsto \begin{cases} w_k + \delta & \text{with probability} \propto |C_{ij}|,\ \text{random perturbation} & \text{otherwise} \end{cases}$$ This mechanism concentrates the loss landscape as measured by a “quasi-thermal entropy” $\Delta S$ and supports generalization through flat, sharply localized minima.

2. Universal Coding Phenomena: Avalanche Dynamics and Criticality

Self-organized neural coding is closely linked with scale-free avalanche-like dynamics, paralleling observations in both artificial and biological systems. At functional criticality, event size distributions obey: $P(N) \propto N^{-3/2}$ where $N$ can represent subgraph size, avalanche size, or similar measures (Liu et al., 28 Aug 2025, Li et al., 2012, Kossio et al., 2018). Critical computational graphs extracted post-training are minimal, O(10) edges in typical tasks, yet encapsulate the full predictive power. Power-law tails are observed not just in connectivity but also in per-sample losses, gradient noise, Hessian spectra, and more—demonstrating that the system operates in a scale-free regime (Liu et al., 28 Aug 2025, Li et al., 2012, Kossio et al., 2018).

3. Phase Transitions and Non-Equilibrium Dynamics

Training proceeds through two non-equilibrium phase transitions:

• Second-order (connectivity, energy-driven): Initial percolation of correlated assemblies, setting up a dynamic, multiscale backbone.
• First-order (convergence, entropy-driven): Flux-mediated pruning as non-equilibrium steady-state probability flux $\mathbf{J}$ selects optimal attractors in weight space.

The Fokker-Planck formalism is employed to characterize steady-state flux: $\partial_t p(w, t) = -\nabla \cdot J(w, t)$

$J = g p - \nabla(D p)$

The marginal flux $|\hat{J}_w(t)|$ exhibits sharp peaks at convergence boundaries in $(M,\gamma)$ space ($\gamma = \eta/B$), marking the phase boundary where the order parameter jumps discontinuously (Liu et al., 28 Aug 2025).

4. Structural and Functional Principles in Biological and Artificial Systems

Self-organized coding principles appear across diverse models:

• Spiking neural networks with heterogeneous units and STDP organize into active-neuron–dominant topologies exhibiting critical avalanches and maximizing entropy and algorithmic complexity of firing patterns (Li et al., 2012, Kossio et al., 2018).
• Moment neural network models reveal emergence of synergistic, bound-state code structures due to intrinsic, nonlinearly coupled fluctuations—improving working memory capacity, spatiotemporal multiplexing, and robustness through negative pairwise correlations (Ma et al., 2022).
• Sparse coding and adaptive compressed sensing: Through competitive recurrent inhibition and Hebbian updates under subsampled input, smooth and efficient receptive fields emerge despite poor initial sampling. Recurrent weights are mathematically essential for code “demixing” (Coulter et al., 2009).
• Population codes and sparse distributed representations (SDR): Codes self-organize into high-capacity, rapidly inferable assemblies governed by winner-take-all modules and binary synapses. The selection algorithm balances similarity-preserving mappings with capacity constraints (Rinkus, 2017).

These phenomena are unified by a set of common features: Hebbian-like reinforcement, scale-free connectivity, phase transitions in network topology, measure concentration in weight/loss space, and non-equilibrium fluxes that dynamically select and stabilize effective codes (Liu et al., 28 Aug 2025, Li et al., 2012, Ma et al., 2022).

5. General Theoretical Synthesis

Self-Organized Neural Coding Theory posits that neural systems—by virtue of local plasticity, stochastic dynamics, and competitive cooperation—spontaneously develop a small, efficiently coupled set of critical neurons or assemblies. These units encode a low-dimensional manifold containing all task-relevant features. Internal representations undergo critical percolation early, establishing multiscale, robust codebooks, followed by entropy- or flux-driven pruning that yields maximally compressed, generalizable, and robust representations.

From an algorithmic and statistical physics viewpoint, such systems operate near the edge of criticality, balancing dynamic range, information transmission, and resilience. Phase transitions, heavy-tailed statistics, and concentration phenomena are not artifacts but indispensable features for efficient neural computation in high-dimensional, stochastic environments (Liu et al., 28 Aug 2025, Li et al., 2012, Ma et al., 2022, Kossio et al., 2018, Malsburg, 2018, Liu et al., 2014).

6. Implications and Broader Significance

• Robust Efficient Coding: Measure collapse and heavy-tailed connectivity enable consistent, low-variance outputs, high generalization, and resistance to noise.
• Dynamic Range and Flexibility: Self-organized criticality ensures the network remains poised for maximal responsiveness and adaptation, mirroring experimental findings of neuronal engine states and avalanches (Kossio et al., 2018, Li et al., 2012, Ma et al., 2022).
• Unified Mechanistic Perspective: Principles derived from artificial network training (including deep SGD) and neurobiological circuits (via STDP and homeostasis) converge: Hebbian updating, self-reinforcing assemblies, and non-equilibrium selection form a universal neural code-generation architecture.

Together, these threads form a coherent theory in which the codebooks, selectivity, and capacity of neural networks arise not from prescribed design or dense wiring, but as a dynamical, emergent, self-optimizing response to task and environment (Liu et al., 28 Aug 2025, Ma et al., 2022, Li et al., 2012, Coulter et al., 2009, Kossio et al., 2018, Malsburg, 2018, Liu et al., 2014, Rinkus, 2017).