Scaling Laws of Feature Emergence

• Scaling laws of feature emergence are quantitative relationships linking model parameters, training duration, and data volume to the abrupt appearance of interpretable features in deep networks.
• Empirical studies using sparse autoencoders in large language models reveal sharp thresholds—both in training steps and parameter counts—that trigger significant surges in activated concept neurons.
• Theoretical analyses, including sigmoidal emergence and capacity-allocation models, connect task statistics and architectural dynamics to feature learning, informing improved mechanistic interpretability and performance scaling.

Scaling laws of feature emergence describe the quantitative and qualitative relationship between model resources (parameter count, training steps, data size) and the abrupt or progressive appearance of interpretable, high-fidelity feature detectors within deep neural networks. This phenomenon, now extensively documented in LLMs and mechanistically probed using sparse autoencoders, is central to modern mechanistic interpretability, theoretical neuroscience, and the empirical science of neural scaling. The precise transitions where particular semantic or algorithmic features become reliably and explicitly encoded within a model are governed by underlying task statistics—such as Zipfian distributions and compositional hierarchies—as well as the architectural and optimization dynamics at play.

1. Empirical Laws of Feature Emergence in LLMs

A landmark empirical study of the Pythia family of LLMs using sparse autoencoders directly characterized concept detector emergence along three axes: training time, spatial depth (layerwise), and model size (Sawmya et al., 26 May 2025). Key findings:

• Training-Time Thresholds: In a 12 B parameter transformer, only ≲3% of semantic concept neurons are active by 1,000 gradient steps, but this fraction undergoes successive surges—∼19.4% at 1–5k steps, +17.5% at 10k–20k, and a dominant +55.9% at 30k–40k. By 143k steps, >99% of domain-aligned concept neurons are active. The dominant emergence threshold is T_crit ≈ 4 × 10⁴ steps.
• Parameter-Scale Thresholds: Smaller models (N < 2 × 10⁸) activate <5% of concepts. A sharp, domain-consistent transition occurs at N_crit ≈ 4.1 × 10⁸, after which nearly all features are present: activation jumps by +92.9 percentage points to ≈95%, plateauing above ≈98% well before 12 B parameters.
• Domain and Layer Dependence: All nine high-level MMLU domains share two patterns: (i) early-onset (≲1k steps) with a major surge at 30k–40k steps (STEM, law, philosophy), or (ii) late-onset (>10k steps) with a broad surge at 30k–60k (history, business, biology, chemistry). The scale threshold remains 410 M parameters for all except business (which saturates closer to 1 B).
• Spatial Dynamics: In-depth analysis shows semantic concepts appear in the first 1–3 layers, vanish across the processing core, and sharply re-emerge in the final layer. This "appear → disappear → reappear" motif breaks the naïve lexical–syntactic–semantic depth hierarchy.

No explicit power-law fits or exponents are reported; scaling is characterized by critical transition points rather than smooth exponents.

2. Analytic and Theoretical Models: Abrupt and Smooth Skill Emergence

Exactly solvable models for multitask (sparse parity) problems and group-structured tasks have clarified how discrete, stepwise feature emergence interacts with aggregate neural scaling trends (Nam et al., 2024, Tian, 25 Sep 2025).

• Sigmoidal Emergence: In multi-linear or two-layer MLPs, each new skill emerges over a short time window, with a sigmoidal gain in correlation strength R_k(T) (Eq. (7), (Nam et al., 2024)), whose location is set by the feature's frequency and resource bottleneck (i.e., P_s(k) ~ k^{–(α+1)} for Zipfian tasks).
• Scaling Exponents: While individual features appear at sharp thresholds, the cumulative loss curve over many power-law distributed features smooths into a power-law: L(T) ∼ T^{–α/(α+1)}, L(N) ∼ N^{–α}, or L(D) ∼ D^{–α/(α+1)}. The compute-optimal joint scaling is L^*(C) ∼ C^{–α/(α+2)} with resource allocation N^* ∼ C^{1/(α+2)}, T^* ∼ C^{(α+1)/(α+2)}.
• Grokking Dynamics and Sample Complexity: In group arithmetic settings (Tian, 25 Sep 2025), the feature stabilization threshold is n* ≍ d_k² M log M (d_k = irrep dimension, M = group size), and the emergence time scales as t_feature ∝ nK/η (K = hidden width, η = weight decay). The post-grokking regime is marked by exponential focusing of gradients onto missing feature classes.

3. Mechanistic Interpretability: Sparse Autoencoder Scaling Laws

Sparse autoencoders (SAEs), when used to decode LLM or vision model activations, obey precise capacity-allocation scaling laws (Michaud et al., 2 Sep 2025). These emerge from the interplay between the Zipf exponent of feature occurrence (α) and the intrinsic dimension of feature manifolds (via the tiling exponent β):

• Benign Regime (α<β): The expected reconstruction loss scales as ℒ(N) ∝ N^{–α}, and the number of features discovered D(N) ∼ N. This regime is "feature discovery" dominated.
• Pathological Regime (β<α): The loss is governed by the manifold tiling exponent, ℒ(N) ∝ N^{–β}, with D(N) ∼ N^{(1+β)/(1+α)} ≪ N; i.e., the SAE tiles continuous manifolds instead of discovering discrete features. Empirical measurements in real models show proximity to this pathological regime when feature manifolds have high dimension and Zipf α ≈ β.

4. Scaling Law Constraints and the Superposition–Universality Dilemma

A distinct theoretical result demonstrates the incompatibility of a pure superposition hypothesis (linear, sparse feature encoding in each layer) with strict universality of feature sets across fixed-parameter architectures, under empirical scaling laws (Katta, 2024). The compressed-sensing lower bound connects the total number of neuron paths and feature sparsity to available parameters; shape manipulations with fixed N change the per-neuron feature load, which cannot be resolved while preserving universal emergent features and scaling-invariant performance. This imposes a fundamental tension that any complete theory of feature emergence must address, possibly requiring nonlinear or cross-layer representations.

5. Neural Scaling Laws, Nonlinear Regimes, and Spectral Theory

Scaling laws of excess risk—which measure global generalization—are intimately linked to the local, spectral emergence of features in both linear and nonlinear architectures (Defilippis et al., 29 Sep 2025, Bordelon et al., 2024, Ren et al., 28 Apr 2025):

• Spectral Bleed-Out and Power-Law Bulks: Excess risk phase diagrams exhibit crossovers between outlier-dominated (few learned features), bulk-dominated (heavy-tailed inferred spectra), and overfitting plateaus. The “bleed-out” of signal spikes corresponds to feature emergence; as sample-complexity N_eff increases, more true features "escape" the noisy bulk, directly improving generalization performance (Defilippis et al., 29 Sep 2025).
• Feature Learning Impact: For hard tasks (targets outside the NTK RKHS), adaptive feature learning sharply increases the scaling exponent of loss decay with training steps—nearly doubling it compared to the kernel regime. For easy tasks, feature learning leaves exponents unchanged and only improves leading constants (Bordelon et al., 2024). In shallow additive models, the ensemble of stepwise feature jumps produces smooth, global power-law performance scaling (Ren et al., 28 Apr 2025).

6. Compositional Data Structures and Feature Emergence in Hierarchical Systems

In settings where data structure is generated from probabilistic context-free grammars (PCFG) or hierarchies of power-law distributed rules, feature emergence and learning curves are determined by the interplay of rule frequency and structural depth (Cagnetta et al., 11 May 2025):

• Classification: When features (production rules) are power-law distributed (f_k ∝ k^{–α}), the test-error scales as ε(N) ∼ N^{–β} with β=(α–1)/α, tightly matching the count of new features discovered.
• Next-Token Prediction: The exponent in scaling laws is controlled by the hierarchical structure (e.g., γ=–log(m/v^{s–1})/(2 log m)), and not the tail of the feature distribution. Thus, the learning of deep compositional features depends on the architecture’s effective capacity for hierarchical inference rather than rare-feature memorization.

7. Open Directions and Limitations

Across all current studies, certain open challenges remain:

• Absence of Closed-Form Exponents in LLMs: While empirical thresholds are observed, no power-law exponents or offset fits have yet been offered for feature-emergence transitions in large-scale transformers (Sawmya et al., 26 May 2025).
• Shape and Compression Nonlinearities: The superposition–universality dilemma suggests emergent features are likely not universally identical across all width/depth tradeoffs at fixed parameter count (Katta, 2024). Subtle or non-linear mechanisms may govern redistribution or invention of features as scale varies.
• Temporal and Architectural Plateaus: There is evidence for non-monotonic “dips” and plateaus in feature activation fraction over training or layers, reflecting optimization inefficiency or feature reorganization, not captured in simple scaling exponents (Sawmya et al., 26 May 2025).
• Manifold Dimensionality: High-dimensional feature manifolds may pathologically suppress feature discovery in SAEs, an effect whose prevalence is only beginning to be characterized (Michaud et al., 2 Sep 2025).
• Limits of Current Theory for Deep, Stochastic, or Residual Architectures: For ultra-deep nets and residual blocks, new stochastic-dynamical analyses in the infinite-width/depth limit are required to predict the onset and possible collapse of feature learning, as well as diminishing returns under scaling (Yao et al., 24 Dec 2025).

In summary, the scaling laws of feature emergence connect the formation of interpretable, high-fidelity neural features to the statistics and resources of modern deep learning systems, giving a unified but still-evolving picture of the mechanisms underlying abrupt capability transitions and the global smoothness of neural scaling phenomena.