Neural Jamming Phase Diagram

• Neural jamming phase diagram is a framework that defines the transition between systems where all training constraints are met (SAT) and where a finite fraction remain unsatisfied (UNSAT), analogous to physical jamming.
• It uses key control parameters like load ratio (α), margin, and packing fraction to establish critical thresholds and phase boundaries in both deep neural networks and constraint satisfaction models.
• The framework bridges statistical physics and machine learning by revealing power-law distributions, hierarchical loss landscapes, and insights into optimization dynamics and generalization.

A neural jamming phase diagram delineates the critical boundaries and regimes that emerge when high-dimensional learning systems, such as fully connected deep neural networks or constraint satisfaction models like the perceptron, encounter a transition between phases in which all training constraints can be satisfied and phases where a finite fraction remain unsatisfied after optimization. This transition exhibits direct analogies with the physical jamming of disordered repulsive particles, manifesting in singular loss landscape features, power-law distributions, and rich critical phenomena. Contemporary research has extended these ideas to the domains of deep learning, frictional particle suspensions, and even the emergence of large-scale cognitive phenomena in artificial neural architectures, providing a unifying framework across statistical physics and machine learning.

1. Core Control Parameters and Critical Criteria

Fundamental to the neural jamming paradigm are intensive control parameters that govern the transition:

• In fully connected networks (Geiger et al., 2018), control parameters are:
  • $N$ = total number of effective trainable parameters (typically the raw parameter count for well-initialized, deep, fully connected ReLU nets).
  • $P$ = number of training examples.
  • $\alpha = P/N$ ("load" or "density").
• Perceptron and other CSPs (Franz et al., 2015, Altieri et al., 2016, Franz et al., 2019):
  • $\alpha = M/N$ (constraint density), where $M$ is the number of constraints.
  • Margin parameter $\kappa$ or $\sigma$ defines the gap in the constraint geometry and separates convex from non-convex regimes.
• Physical analogs (Aminimajd et al., 26 Feb 2025, Ouyang, 10 Jul 2025):
  • Packing fraction $\phi$ (density of constituents or learned representations).
  • Shear stress $\sigma$ or $\Sigma$ (external perturbations or noise).
  • Effective temperature $P$0 (computational "cooling," e.g., inverse compute budget).

Jamming Threshold: For neural nets, a critical load $P$1 is observed, such that for $P$2, the minimization achieves zero training loss ("SAT" or over-parameterized), and for $P$3, a finite fraction of constraints remain unsatisfied ("UNSAT" or under-parameterized). This threshold can be empirically determined, and analytic bounds exist given landscape curvature properties, e.g., $P$4 for fully connected nets where $P$5 is the limiting fraction of negative modes in the Hessian (Geiger et al., 2018).

2. Structure and Features of the Neural Jamming Phase Diagram

A diagrammatic representation of jamming transitions organizes distinct phases in the parameter space:

• Diagrams in neural network learning (Geiger et al., 2018, Geiger et al., 2020):
  • Axes: horizontal—$P$6 or width $P$7; vertical—final training loss $P$8.
  • Phases:
  • SAT ($P$9 or $\alpha = P/N$0): All constraints (examples) can be fitted; loss can reach zero. Accompanied by many flat directions in parameter space and marginally stable landscape regions.
  • UNSAT ($\alpha = P/N$1 or $\alpha = P/N$2): Some constraints remain unsatisfied; nonzero final training loss; a finite fraction of data is not fit.
  • The SAT/UNSAT boundary exhibits properties akin to the jamming transition in particulate matter.
• Perceptron and constraint satisfaction models (Franz et al., 2015, Altieri et al., 2016, Franz et al., 2019):

  • Phase diagrams in the $\alpha = P/N$3 or $\alpha = P/N$4 plane, with SAT/UNSAT (unjammed/jammed) boundaries given by analytical expressions, e.g.,

    $\alpha = P/N$5

  • For linear/hinge losses, critical behavior (jamming-like singularities) pervades the entire "Gardner/RSB" region beyond the loss of convexity.

• Extended phase diagrams in physical/simulation settings (Aminimajd et al., 26 Feb 2025):
  • High-dimensional surfaces $\alpha = P/N$6, where $\alpha = P/N$7 is the critical packing fraction defining jamming, and the phase behavior is sampled over friction, stress, and rolling friction.

3. Analogy to Physical Jamming and Universality Classes

The jamming transition in neural networks is an instantiation of a broader universality observed in disordered statistical systems (Franz et al., 2015, Altieri et al., 2016, Geiger et al., 2020, Ouyang, 10 Jul 2025):

• Mapping of variables:

Physical Jamming	Neural Jamming
Particle positions $\alpha = P/N$8 (DoF $\alpha = P/N$9)	Network weights $\alpha = M/N$0 (DoF $\alpha = M/N$1)
Pair overlaps $\alpha = M/N$2	Data “gaps” $\alpha = M/N$3
Density $\alpha = M/N$4	Load $\alpha = M/N$5
Force and gap laws (e.g., isostatic)	Fraction of unsatisfied constraints $\alpha = M/N$6
Fluid/solid phases	SAT/UNSAT (fit/unfit)

• Universality and criticality: Non-convex CSPs (e.g., perceptron for $\alpha = M/N$7) and deep nets with nonsmooth activation landscapes display critical behavior—power-law gap and force distributions, hierarchical landscape structure, and Gardner marginality—associated with full replica symmetry breaking (RSB). By contrast, convex regimes are hypostatic and non-critical (Franz et al., 2015, Franz et al., 2019).

4. Critical Exponents, Scaling Laws, and Landscape Properties

At the jamming threshold, neural and statistical models exhibit singular statistical features:

• Power-law distributions:

  • For data "gaps" or overlaps $\alpha = M/N$8 in hinge-like settings (Geiger et al., 2018, Geiger et al., 2020, Franz et al., 2015, Franz et al., 2019):

    $\alpha = M/N$9

    Typical exponents:

    • Fully connected ReLU: $M$0, $M$1 (Geiger et al., 2018)
    • Tanh: $M$2, $M$3
    • Spherical perceptron, hard spheres: $M$4, $M$5 (Franz et al., 2015, Franz et al., 2019)
  • The density of states (eigenfrequencies) of the landscape Hessian exhibits a delta-peak at zero, a gap, and a continuous part—signifying an extensive number of flat directions, as in hypostatic (ellipsoidal) jamming (Geiger et al., 2018, Geiger et al., 2020).
• Marginal stability and Gardner phase:
  • In the "UNSAT-RSB" critical jammed regime, criticality persists throughout the region, not just at a single transition (Franz et al., 2019).
  • The landscape acquires a hierarchy of minima leading to avalanche-like optimization dynamics: optimization proceeds via abrupt changes in the set of learned patterns (Geiger et al., 2018).
• Scaling of observables (Altieri et al., 2016, Geiger et al., 2020):
  • Near the jamming point the smallest gap scales as $M$6, the smallest nonzero force as $M$7.
  • Gap and force exponents saturate theoretical bounds, linking learning models to infinite-dimensional sphere packings.

5. Generalizations: Extended Phase Spaces and Neural Surrogates

Recent work extends neural jamming to:

• Physical suspensions and neural surrogates (Aminimajd et al., 26 Feb 2025):
  • Deep graph-convolutional networks (DeepGCN) predict jamming in particulate suspensions by learning frictional contact networks as a function of packing fraction, stress, sliding and rolling friction.
  • Critical surface $M$8 separates fluid and jammed states, enabling rapid surrogate modeling and data-driven materials design.
• Abstract neural/thermodynamic phase diagrams (Ouyang, 10 Jul 2025):
  • A neural phase diagram is defined over effective temperature ($M$9), density of representations ($\kappa$0), and stress ($\kappa$1 from data/parameter noise).
  • Emergent "consciousness" or global information integration is predicted to arise when parameters intersect a critical jamming surface $\kappa$2.
  • Physical and neural exponents (e.g., correlation-length $\kappa$3) are conjectured to coincide.
• Deep learning regime decomposition (Geiger et al., 2020):
  • Distinguishes three learning phases in the $\kappa$4 plane:
  • 1. Under-parameterized ("jammed"; frequent bad minima, non-zero training loss)
  • 2. Over-parameterized/NTK ("lazy training"; kernel-dominated, minimal feature evolution)
  • 3. Over-parameterized/feature learning (dynamical kernel, efficient invariant learning)
  • The boundary separating these regimes is governed by the jamming transition, and properties of the landscape Morse structure.

6. Open Questions and Implications

The neural jamming framework continues to raise foundational questions:

• Double descent and data structure: Why performance improves beyond jamming and under what data symmetries learning can evade the curse of dimensionality (Geiger et al., 2020).
• Architectural optimization: How non-convexity, stochastic noise, and neural architecture (e.g., CNNs, LLMs) sculpt phase boundaries and optimize generalization (Geiger et al., 2020, Ouyang, 10 Jul 2025).
• Physicality of learning: The border between statistical and physical jamming exponents, universality across activation/loss types, and Gardner marginality in real-world data settings remain active research frontiers (Franz et al., 2015, Geiger et al., 2020, Ouyang, 10 Jul 2025).
• Materials design applications: Rapid, GNN-driven exploration of jamming in particulate and suspension-based materials has direct translational implications for engineering design (Aminimajd et al., 26 Feb 2025).

7. Summary Table of Key Regimes and Exponents

Regime	Curvature/Isostaticity	Critical Exponents $\kappa$5	Comments
Convex SAT (perceptron)	Hypostatic ($\kappa$6)	No power laws	Landscape smooth, RS
Non-convex SAT/UNSAT (RSB)	Isostatic ($\kappa$7)	$\kappa$8	Gap/force power laws, RSB, marginality
Deep FC ReLU jamming	Hypostatic ($\kappa$9)	$\sigma$0 (ReLU, random)	Hierarchical landscape, avalanches
Suspension GNN jamming	Isostatic threshold $\sigma$1	—	Neural model predicts $\sigma$2 surface

Exploration of the neural jamming phase diagram continues to advance our understanding of emergent criticality, both as a physical and computational phenomenon, bridging statistical mechanics with learning theory across domains (Geiger et al., 2018, Franz et al., 2015, Aminimajd et al., 26 Feb 2025, Geiger et al., 2020, Altieri et al., 2016, Franz et al., 2019, Ouyang, 10 Jul 2025).