--- title: Feature Learning vs Lazy Regimes url: https://www.emergentmind.com/topics/feature-learning-vs-lazy-regimes type: topic --- # Feature Learning vs Lazy Regimes Feature learning and lazy regimes constitute two sharply distinct, theoretically tractable phases of learning dynamics in neural networks. These regimes emerge from different scalings of network width, initialization, and architecture, and manifest in distinct geometric, statistical, and generalization behaviors. Contemporary research has provided precise mathematical and statistical-mechanical characterizations, revealed collective and microscopic signatures, established transitions between the regimes, and connected these with core phenomena such as generalization, representation geometry, continual learning, and transfer. ## 1. Formal Definitions and Regime Boundaries The “lazy” (or Gaussian-process/NTK) regime is characterized by network parameters that remain infinitesimally close to their initialization throughout training. In this limit (often realized as $N_1\to\infty$ at fixed, small dataset size $P$, so $P/N_1\to0$), the feature activations $\sigma(h^\mu_i)$ experience only $\mathcal{O}(1/N_1)$ fluctuations around their random initialization. The posterior predictive distribution converges to that of a Gaussian process determined by the neural tangent kernel (NTK), $K_{\mu\nu} = \mathbb{E}_w[\sigma(h^\mu)\sigma(h^\nu)]$, and parameter updates produce negligible movement—no nontrivial feature learning occurs [2508.20989, 2409.14623, 2502.03210, 2404.19719]. The “feature-learning” (proportional-width or mean-field) regime requires both $N_1, P\to\infty$ at fixed $\alpha = P/N_1 > 0$. Now, finite, data-dependent weight updates induce $\mathcal{O}(1)$ corrections to both the hidden-layer feature correlations and weight covariances. The posterior predictive remains a GP regression, but with a renormalized kernel $K_{(R)} = \bar Q K/\lambda_1$, where $\bar Q$ solves a nontrivial saddle-point equation enforcing data dependence. The key boundary between regimes is therefore set by the load parameter $\alpha$; increasing $\alpha$ drives the system away from the lazy phase into rich feature adaptation [2508.20989, 2409.14623, 2502.21009, 2502.03210]. ## 2. Mathematical Characterization: Macroscopic and Microscopic Signatures ### Collective (Geometric) Signatures The geometry of internal representations distinguishes the regimes. In a binary classification setup, the squared distance $D^2$ between class centroids in feature space, \[ D^2 = \left\| \frac{2}{P}\sum_{\mu:y^\mu=+1}\sigma(h^\mu) - \frac{2}{P}\sum_{\mu:y^\mu=-1}\sigma(h^\mu) \right\|^2, \] has a posterior average that separates into a “prior” term (present in the lazy regime) and “feature-learning” corrections surviving at $\mathcal{O}(1)$ only in the proportional limit ($\alpha>0$) [2508.20989]. Notably, $\langle D^2\rangle$ in the feature-learning regime displays a nonmonotonic dependence on regularization/temperature $T$, peaking at an inversion temperature $T^*(\alpha)$, whereas in the lazy regime, $D^2$ is monotonic in regularization and independent of $T$. Geometric effects such as the sharpening and untangling of class manifolds thus provide observable macroscopic fingerprints of feature learning [2508.20989, 2503.18114]. ### Microscopic (Weight) Signatures The feature-learning regime induces pattern-dependent two-point correlations among weight parameters that are absent in lazy (NTK) networks. In the lazy limit ($\alpha\to0$), first-layer weights $w_{1h}$ remain independent, with covariances $\langle w_{1h}w_{1k}\rangle \approx \delta_{hk}/\lambda_0$; all cross-neuron correlations vanish. For $\alpha>0$, nontrivial data-dependent corrections $\Lambda_{1,2} = \mathcal{O}(\alpha)$ yield new, genuine weight correlations. This microscopic displacement is a hallmark of feature learning, not present in kernel methods [2508.20989]. ## 3. Regime Transitions, Scaling Laws, and Phase Diagrams Regime transitions are controlled by global and relative scalings of network width, dataset size, and initialization, with critical points and continuous crossovers. - In fully-connected and deep linear nets, a balancing parameter $\lambda$ (difference in squared layer norms) interpolates between strictly lazy (large $|\lambda|$) and rich (balanced, $\lambda\approx0$) regimes; see exact closed-form Riccati solutions and “sigmoidal” versus exponential singular-value dynamics [2409.14623, 2502.21009]. - In practice, the boundary between regimes is sharply controlled by the scaling variable $\alpha = P/N_1$ (or, in other conventions, by a “richness” parameter $r$ that tunes hidden-feature changes as $N^r$ for width $N$). The regime $\alpha\ll1$ (or $r\to0$) is lazy/NTK, while $\alpha=\mathcal{O}(1)$ (or $r\to1/2$) is rich/μP [2404.19719]. - Empirical phase diagrams constructed in the $(h,\alpha)$ plane (with $h$ width, $\alpha$ initialization/output scale) exhibit a “jamming” critical line between under-/over-parameterization, and within the overparameterized phase, a diagonal boundary at $\alpha\sqrt h = \mathcal{O}(1)$ separating lazy from feature-learning behavior [2012.15110, 1906.08034]. ## 4. Connections to Generalization, Capacity, and Representation Geometry The lazy regime strictly limits the generalization power: the induced predictor is equivalent to fixed-kernel (NTK or random feature) regression. Notably, fully trained networks can generalize in underparameterized settings (e.g., $m