Deriving Neural Scaling Laws from the statistics of natural language

Abstract: Despite the fact that experimental neural scaling laws have substantially guided empirical progress in large-scale machine learning, no existing theory can quantitatively predict the exponents of these important laws for any modern LLM trained on any natural language dataset. We provide the first such theory in the case of data-limited scaling laws. We isolate two key statistical properties of language that alone can predict neural scaling exponents: (i) the decay of pairwise token correlations with time separation between token pairs, and (ii) the decay of the next-token conditional entropy with the length of the conditioning context. We further derive a simple formula in terms of these statistics that predicts data-limited neural scaling exponents from first principles without any free parameters or synthetic data models. Our theory exhibits a remarkable match with experimentally measured neural scaling laws obtained from training GPT-2 and LLaMA style models from scratch on two qualitatively different benchmarks, TinyStories and WikiText.

• The paper introduces a non-parametric framework that quantitatively links natural language statistical properties to neural scaling exponents.
• It validates the theoretical prediction α_D = γ/(2β) using extensive experiments on autoregressive transformers across different corpora.
• The findings offer actionable insights for optimizing pretraining data acquisition and model design within the horizon-limited learning regime.

Deriving Neural Scaling Laws from the Statistics of Natural Language

Introduction

This paper addresses a central open question in the theoretical understanding of LLMs: what determines the exponents governing data-limited neural scaling laws? While empirical observations of neural scaling laws—power-law relationships between generalization performance and dataset/model sizes—are widely leveraged to guide the training of LLMs, their origin has remained opaque. This paper establishes, for the first time, that measurable statistical properties of natural language data alone are sufficient to quantitatively predict the exponents of these scaling laws, specifically in the data-limited regime. The authors introduce a principled, non-parametric framework relating two key dataset-dependent quantities to learning curve exponents and validate the resulting theoretical predictions against extensive experiments with autoregressive transformers across two distinct corpora.

Theoretical Framework

The core theoretical contribution identifies two dataset-level statistics as fundamental controls on the data-limited neural scaling exponent:

1. Exponent $\gamma$: Characterizing the power-law decay of the next-token conditional entropy $H_n$ with context size $n$, i.e., $H_n - H_\infty \sim n^{-\gamma}$, where $H_n$ is the conditional entropy of the next token given the preceding $n$ tokens.
2. Exponent $\beta$: Characterizing the power-law decay of token-token correlations with their temporal lag $n$, specifically through the norm of the covariance matrix $C(n)$ of tokens separated by $n$ positions: $\|C(n)\| \sim n^{-\beta}$.

The theoretical argument proceeds by decomposing test loss into two sources: the predictive horizon determined by data ($n^*(P)$, the maximal context effectively exploited given $P$ training tokens) and the error arising from suboptimal predictions within this accessible horizon. The crux is that $n^*(P)$ itself is determined by the detectability of token correlations, i.e., $n^*(P) \sim P^{1/(2\beta)}$. Combining these scaling ansätze, the theory predicts the data-limited scaling exponent $\alpha_D = \gamma / (2\beta)$, governing how excess loss above entropy rate decays with dataset size $P$:

$\mathcal{L}_{AR}(P) - H_\infty \sim P^{-\alpha_D}$

Figure 1: Measurable language statistics predict the exponents of data-limited neural scaling laws in LLMs, demonstrating scaling collapse and theoretical-experimental alignment.

This framework further predicts a universal collapse of $n$-gram learning curves under appropriate horizontal (by $n^{2\beta}$) and vertical (by $n^{-\gamma}$) rescaling, consolidating diverse curves across horizons onto a single master curve.

Empirical Validation

Estimation of Language Statistics

The exponents $\gamma$ and $\beta$ were measured on two corpora—TinyStories and WikiText-103—using high-capacity autoregressive transformers of varied architectures (GPT-2 with absolute or rotary position encodings, and LLaMA-style models). Conditional entropy decay ($\gamma$) was estimated from converged $n$-gram losses as a function of $n$, while token correlation decay ($\beta$) was extracted from spectral norms of empirical covariance matrices.

Figure 2: Conditional entropy decay with horizon $n$ for TinyStories models, revealing consistent $\gamma$ across architectures.

Figure 3: Decay of two-point token correlations versus lag $n$, defining exponent $\beta$ for TinyStories and WikiText.

The measured values:

• TinyStories: $\gamma=0.34$, $\beta=0.88$
• WikiText: $\gamma=0.27$, $\beta=0.94$ These exponents show architectural invariance and characterize dataset-specific statistical structure.

Empirical Scaling Laws

The theory's central claim—$\alpha_D = \gamma/(2\beta)$—was validated by training transformers of varying context sizes on both benchmarks. The observed power-law exponents in the loss-vs.-data curves closely matched the theoretical predictions, across multiple architectures and context window choices. For example:

• TinyStories: $\alpha_D \approx 0.19$ (theory), empirically verified.
• WikiText: $\alpha_D \approx 0.14$ (theory), empirically verified.

Notably, distinct $n$-gram learning curves collapsed under the prescribed rescalings, confirming that both learning rates and horizon unlocking are dictated by the exponents $\gamma$ and $\beta$.

Figure 4: Scaling collapse and exponent matching for WikiText using GPT-2–style transformers.

Figure 5: Estimation of $\gamma$ for WikiText, demonstrating architecture invariance.

Horizon Limited and Fast Learning Regime

The theory anticipates two learning regimes: (i) horizon-limited, where test loss is dominated by expanding predictive horizon with data, and (ii) within-horizon, where slow convergence within fixed horizons would disrupt the scaling. Empirical evaluation of $n$-gram learning curve asymptotics showed that within-horizon losses decay faster than $P^{-\alpha_D}$, confirming the horizon-limited regime for deep architectures on these datasets.

Figure 6: $n$-gram losses decay to their entropy asymptotes faster than $P^{-\gamma/(2\beta)}$ in TinyStories.

Implications and Outlook

Practical and Theoretical Implications. This work provides a concrete, parameter-free prescription for predicting data-limited scaling exponents from any language dataset, offering actionable guidance for pretraining data acquisition and architecture scaling. The results suggest that, for deep architectures capable of rapidly optimizing within accessible context, data-limited scaling is bounded solely by language statistics rather than model specifics.

Theoretically, the framework situates LLM learning dynamics within a broader universality class marked by "horizon-limited" error—analogous to critical phenomena in statistical physics, where scaling exponents group diverse systems into universality classes. Interestingly, shallow or kernel-based methods, which cannot efficiently exploit hierarchical, long-range structure, lie outside this universality class, incurring much slower learning curves.

A further implication is the existence of a statistical ceiling: for any fixed dataset, the achievable loss at finite data and context is fundamentally constrained by its intrinsic entropy and correlation structure. Nevertheless, the authors speculate on the possible existence of architectures (e.g., convolutional or permutation-equivariant models) that may transcend this regime, e.g., through inductive biases aligning more tightly with compositional or hierarchical language structure—an avenue for future exploration.

Conclusion

This paper establishes a principled, empirical, and theoretical link between neural scaling laws in language modeling and two measurable statistical properties of natural language: the decay rates of conditional entropy and token correlation with context. Through analytical derivation and rigorous experimental support, it is demonstrated that test loss scaling exponents in the data-limited regime follow $\alpha_D = \gamma/(2\beta)$, with no fitted parameters, for a wide class of deep autoregressive models. This result clarifies the origin of scaling behavior in LLMs and provides new conceptual and practical tools for both analysis and model design (2602.07488).