---
title: Feature-Learning Regime Overview
url: https://www.emergentmind.com/topics/feature-learning-regime
type: topic
---

# Feature-Learning Regime Overview

A feature-learning regime denotes the operational phase of a neural network in which training induces substantial evolution of hidden representations—contrasting with "lazy" or "kernel" regimes where features remain effectively fixed and the learning dynamics are described by kernel methods. The transition between these regimes is sharply controlled by architectural parameterizations (notably, the output scaling $\gamma$ in maximal-update or $\mu$P scaling) and key optimization hyperparameters such as the learning rate $\eta$. The feature-learning regime is characterized by rich dynamics, distinct loss curves, directional kernel adaptation, and significant improvements in generalization, especially for tasks outside the span of the initial neural tangent kernel (NTK) [2410.04642, 2502.03210, 2409.17858].

## 1. Parameterization and Phase Demarcation: The Role of Output Scaling

The regime in which a neural network operates—lazy/kernel-like versus rich/feature-learning—is principally set by the final-layer scaling hyperparameter $\gamma$. In maximal-update ($\mu$P) scaling, the network output is normalized:

\[
\tilde f(x; \theta) = \frac{1}{\gamma} f(x; \theta)
\]

- **Lazy regime ($\gamma \rightarrow 0$):** The network evolves in a small neighborhood of initialization; the NTK remains essentially constant and feature evolution scales as $O(\gamma)$, resulting in negligible adaptation of hidden representations.
- **Feature-learning ("rich") regime ($\gamma \gg 1$):** Hidden representations undergo $O(1)$ changes; the network must significantly adjust internal weights to fit the data, driving strong feature learning.

Tuning $\gamma$—along with appropriately scaling the learning rate $\eta$—transitions the network between these regimes, with $\gamma$ acting as a feature-learning strength knob [2410.04642].

## 2. Theoretical Scaling Laws and Regime Map

The mathematical structure of the feature-learning regime is concretely specified via scaling laws for the optimal learning rate as a function of $\gamma$ and network depth $L$.

For a feed-forward depth-$L$ network:
- In the underparameterized/lazy regime ($\gamma \ll 1$), stability and optimizing step constraints yield $\eta^* \propto \gamma^2$.
- In the feature-rich regime ($\gamma \gg 1$), stability requires $\eta^* \propto \gamma^{2/L}$. The optimizable window for $\eta$ is $[\gamma/T,\,\gamma^{2/L}]$.

These bounds define phase regions in the $(\gamma,\eta)$ plane:
- **No-training:** $\eta < \eta_{\min}$, loss does not decrease.
- **Lazy-kernel:** $\gamma \ll 1$, $\eta_{\min} \lesssim \eta \lesssim \eta_{\max} \sim \gamma^2$.
- **Catapult (MSE only):** $\eta \gtrsim \gamma^2$, loss briefly diverges before settling.
- **Rich-feature-learning:** $\gamma \gg 1$, $\eta_{\min} \lesssim \eta \lesssim \gamma^{2/L}$ [2410.04642].

## 3. Empirical and Analytical Phenomenology in the Rich Regime

Networks tuned into the ultra-rich ($\gamma \gg 1$) regime exhibit distinctive optimization and loss curves:

- **Long initial loss plateau:** "Silent alignment"—representations reorient with respect to the data manifold, but external metrics (loss) change little.
- **Sudden loss drop-off:** Once alignment is achieved, loss rapidly decreases.
- **Staircase decay:** Multiple abrupt loss drops can occur (notably in deep or certain linear networks).

Loss curves for different $\gamma$ collapse onto a universal trajectory under the rescaled time $\tau = (\eta / \gamma) t$ at early times, revealing an underlying time-reparameterization invariance of the rich regime. The duration of the plateau $\sim \gamma/\eta$ sets the trade-off between feature complexity and practical learning under a step-budget constraint [2410.04642].

## 4. Feature Kernel Adaptation: Scalar Rescaling vs. Directional Deformation

Analytical approaches expose two perspectives on how kernels adapt in the feature-learning regime [2502.03210]:

- **Kernel-rescaling theories:** In linear networks and in the infinite-width mean-field limit, integration of the posterior yields a solution mathematically equivalent to kernel regression with a scalar-rescaled NNGP kernel:
  \[
  K_{\text{eff}} = (Q_{TL}/g_w N^{1-\gamma}) K_{\text{NNGP}}
  \]
  Thus, on the mean output, feature learning appears as simple amplitude rescaling.

- **Adaptive kernel theories:** The more general case, especially for finite width and nonlinear networks, reveals directional deformation of the kernel—selectively amplifying certain modes aligned with the data and residuals (i.e., rank-$P$ corrections). The posterior covariance of the learned function includes rank-one (or higher) corrections that are missed by rescaling-only views, and these corrections directly correlate with the degree of feature learning [2502.03210].

Directional feature learning is identified by projecting the output covariance onto candidate "features" and observing selective enhancement along task-relevant directions.

## 5. Feature-Learning Regime, Scaling Laws, and Task Difficulty

Feature learning transforms the scaling laws of neural network generalization, with the degree of improvement contingent on the task's alignment with the initial NTK RKHS [2409.17858]:

- **Easy/super-easy tasks (within RKHS):** Feature learning does not alter scaling exponents; test loss decays as $t^{-\beta}$ with $\beta > 1$ unchanged across lazy and rich regimes.
- **Hard tasks (outside RKHS):** Feature learning nearly doubles the decay exponent from $t^{-\beta}$ (lazy) to $t^{-2\beta/(1+\beta)}$ in the rich regime.

This acceleration is attributed to the evolving kernel norm, which boosts the learned-mode bandwidth. The improved exponent directly alters the compute-optimal trade-off between model size and training time.

## 6. Empirical Fingerprints, Regime Detection, and Practical Guidelines

Several metrics and structural observables diagnose entry into the feature-learning regime:

- **Feature movement:** $S(t)=1-\langle K(0),K(t)\rangle_F/(\|K(0)\|_F \|K(t)\|_F)\gg0$ signals NTK evolution.
- **Eigenfeature and minimum-projection (CKA) metrics:** Sharp transition in $\kappa_{\text{CKA}}$ and effective-rank identifies the minimal-feature regime, especially in vision models [2410.04264].
- **Soft rank:** Stable or growing soft-rank of hidden feature matrices under SGD signals ongoing feature learning [2502.13110].

**Tuning advice** for practitioners:
- Always sweep both $\gamma$ and appropriately scaled $\eta$ (by $\gamma^2$ or $\gamma^{2/L}$ as regime dictates).
- For maximal feature benefits, choose largest $\gamma$ for which the initial plateau remains within total training-hour constraints.
- Avoid $\eta$ values that induce loss catapults or instability in low-$\gamma$ settings [2410.04642].

## 7. Broader Theoretical Landscape and Transfer Learning Implications

Feature-learning regimes admit rigorous characterization in mean-field/Bayesian frameworks, where phase transitions in order parameters correspond to the onset and qualitative strength of feature learning [2510.15174]. At finite width, feature learning emerges via symmetry breaking, or through mechanisms such as self-reinforcing input feature selection (ARD), which removes ambient dimension dependence and compresses sample complexity thresholds to the intrinsic task dimension.

Transfer learning in the feature-learning regime, as opposed to the lazy regime, is governed by *adapted* feature kernels that depend on both source and target data/labels, with explicit interpolation (via elastic coupling penalties) between feature reuse and full re-learning [2507.04448]. Empirical results confirm that strong feature learning and optimal transfer occur when hidden representations are sufficiently plastic to adapt to the downstream task, governed by $\gamma$ (feature strength) and the coupling parameter.

---

**References:**  
- [2410.04642]  
- [2502.03210]  
- [2409.17858]  
- [2410.04264]  
- [2502.13110]  
- [2510.15174]  
- [2204.10782]  
- [2512.21075]  
- [2507.04448]  
- [2508.20989]

Source: https://www.emergentmind.com/topics/feature-learning-regime