On the origin of neural scaling laws: from random graphs to natural language

Abstract: Scaling laws have played a major role in the modern AI revolution, providing practitioners predictive power over how the model performance will improve with increasing data, compute, and number of model parameters. This has spurred an intense interest in the origin of neural scaling laws, with a common suggestion being that they arise from power law structure already present in the data. In this paper we study scaling laws for transformers trained to predict random walks (bigrams) on graphs with tunable complexity. We demonstrate that this simplified setting already gives rise to neural scaling laws even in the absence of power law structure in the data correlations. We further consider dialing down the complexity of natural language systematically, by training on sequences sampled from increasingly simplified generative LLMs, from 4,2,1-layer transformer LLMs down to language bigrams, revealing a monotonic evolution of the scaling exponents. Our results also include scaling laws obtained from training on random walks on random graphs drawn from Erdös-Renyi and scale-free Barabási-Albert ensembles. Finally, we revisit conventional scaling laws for language modeling, demonstrating that several essential results can be reproduced using 2 layer transformers with context length of 50, provide a critical analysis of various fits used in prior literature, demonstrate an alternative method for obtaining compute optimal curves as compared with current practice in published literature, and provide preliminary evidence that maximal update parameterization may be more parameter efficient than standard parameterization.

• The paper shows that neural scaling laws appear even with data, like random walks on Erdős–Rényi graphs, that lack intrinsic power-law structure.
• It analyzes datasets ranging from simple bigrams to natural language, revealing a monotonic decline in scaling exponents as data complexity increases.
• The study advocates for nonparametric regression methods over traditional Chinchilla 2D fits to enhance predictive validity in scaling law analyses.

On the Origin of Neural Scaling Laws: From Random Graphs to Natural Language — A Technical Analysis

Introduction and Context

The characterization of neural scaling laws—empirical power-law relationships governing test loss as a function of data size ($D$), model parameters ($N$), and compute ($C$)—has underpinned practical advances in deep learning, especially in LLM pretraining. This study probes the origin of these scaling laws by systematically controlling the complexity of the data. In particular, it investigates whether data exhibiting no underlying power-law structure (e.g., random walks on Erdős–Rényi graphs) can lead to power-law-like neural scaling behavior during next-token prediction with transformers. The methodology spans from these simplified settings to natural language, constructing interpolating sequences via datasets sampled from intermediate generative models (bigrams, shallow transformers), and compares the evolution of scaling exponents as data complexity increases.

Scaling Laws in the Absence of Data Power Laws

A central claim is that neural scaling laws emerge robustly even when the data lacks inherent power-law statistics, directly contradicting the notion that scaling exponents are entirely inherited from the compositional structure of the data. The authors empirically validate this by training 2-layer transformers on unbiased random walks over Erdős–Rényi graphs (8k nodes, 50k edges), where auto-correlations and degree distributions exhibit no power-law behavior (confirmed via stationary and transition probability statistics). The test loss as a function of both $N$ and $D$ closely follows a power law:

$$L(N)_D = E_D + A_D N^{-\alpha_D}, \qquad L(D)_N = E_N + B_N D^{-\beta_N}$$

with highly consistent exponents ($\overline{\alpha_D} = 1.028$, $\overline{\beta_N} = 0.749$), minimal deviation, and mean squared error (MSE) 5–100$\times$ lower than the best exponential fit.

Figure 1: Scaling laws for 2-layer transformers trained on unbiased random walks on an Erdős–Rényi graph; neither the walks nor the graph exhibit data power laws.

The implication is that the geometric or algebraic details of the data manifold are not strictly necessary prerequisites for neural scaling to arise; architectural and optimization dynamics in sequence modeling tasks suffice to induce them.

Evolution of Scaling Exponents with Data Complexity

To dissect the dependency of scaling exponents on data complexity, the paper constructs a monotonic pathway from bigram LLMs (random walks on graphs), to synthetic datasets generated by small transformers trained on natural text (T1L, T2L, T4L), up to true natural language. Entropy (measured via fitted irreducible loss $E$) serves as a rough complexity quantifier.

Figure 2: Mean scaling exponents $\overline{\alpha_D}$ and $\overline{\beta_N}$ for random walks, language bigrams, TnL, and natural language, showing monotonic evolution with dataset entropy/complexity.

Key findings:

• Language bigram models: $\overline{\alpha_D} \approx 0.98$, $\overline{\beta_N} \approx 0.55$ (clear power law fit).
• T1L, T2L: Exponents smoothly interpolate between bigram and true natural language ($\overline{\alpha_D}$ decreases with complexity).
• Natural language (FineWeb-edu): For shallow transformers, $\overline{\alpha_D}$ drops to 0.35–0.5 depending on parameterization.

This monotonicity confirms that neural scaling exponents can be modulated by the complexity of the generative process underlying the data, with higher complexity reducing parameter efficiency.

Fitting Methodology and Critique of Prior Practice

The authors emphasize the imperative necessity of including the irreducible loss $E$ when fitting scaling laws, as omitting $E$ leads to significant underestimation of $\alpha_D$ and $\beta_N$. Moreover, the widespread use of two-dimensional Chinchilla-style parametric fits for $L(N,D)$ is shown to yield substantially higher fit error compared to nonparametric alternatives (e.g., neural network or kernel regression fits):

Figure 4: Left: 1D fits to $L(N)_D$ on Pythia models; Right: comparison of parametric (Chinchilla 2D) and nonparametric fitting for Chinchilla data.

Figure 6: Compute-optimal scaling laws extracted using the neural network fit for $L(N,D)$, outperforming 2D Chinchilla in MSE.

The recommendation is clear: practitioners should move away from Chinchilla 2D analytic fits for $L(N,D)$ in favor of ML-based regression techniques, as these provide both improved predictive validity (crucial for extrapolation) and model selection advantage.

Language Modeling: Short Context and Parameterization Effects

The paper demonstrates that the principal discrepancies between the “Kaplan” and “Chinchilla” scaling laws in prior literature (re: compute-optimal balance between data and parameters) are recoverable with shallow transformers (2 layers, context 100). The only substantial variable affecting these results is whether embedding parameters are included in $N$, confirming recent findings that much of the open issue is methodological rather than empirical.

Figure 8: Scaling law fits for 2-layer transformer on natural language with maximal update parameterization ($\mu$P); mean exponents and compute scaling verified.

An additional novel finding is that, at these relatively small scales, maximal update parameterization ($\mu$P) yields improved parameter efficiency (larger $\overline{\alpha_D}$, smaller compute-optimal $a$ exponent), suggesting that compute-optimal scaling is not necessarily achieved at a constant token-per-parameter ratio, contesting established heuristics from literature.

Random Walks on Scale-free and Biased Graphs

The robustness of neural scaling laws is further substantiated through experiments on Barabási–Albert graphs (which do exhibit degree distribution power laws) and on random walks where transition probabilities are power law distributed. In all such cases, scaling persists—with the fit quality sometimes improving when data correlations are heavy-tailed. This supports but does not necessitate the “data power law $\Rightarrow$ neural scaling law” thesis.

Figure 10: Scaling laws for Barabási–Albert graph; exponents and MSE confirm robust scaling regardless of data correlation structure.

Theoretical and Practical Implications

• Data-independence of scaling laws: Neural scaling appears as an emergent property of the optimization/modeling process rather than a direct reflection of explicit power law statistics in the data.
• Parameter efficiency tuning: By adjusting dataset complexity, one can systematically dial the scaling exponents, with potential for designing more compute-efficient training regimens for novel architectures or modalities.
• Best-fitting methodology: Regression (NN or kernel) outperforms analytic formulas for $L(N,D)$, highlighting the need for methodological reevaluation in scaling law research.

Practically, these findings inform how practitioners should interpret scaling exponents and select model/data scaling for a given compute budget. Theoretically, they highlight the necessity for new analytic perspectives on sequence model overparameterization and gradient-based learning dynamics, beyond explanations tying scaling law universality solely to data manifold properties.

Future Directions

The study motivates several lines of investigation:

• Characterization of scaling laws under multi-epoch training, data augmentation, and with more complex graph/hypergraph structures to bridge the gap between artificial and natural complexity.
• Systematic study of the emergence (or lack) of power law statistics within model activations, even when absent from the raw data.
• Sensitivity analyses on hyperparameter optimization and window-of-fit, especially in the context of large-scale, nonstationary datasets.

Conclusion

This paper demonstrates that neural scaling laws in cross-entropy sequence modeling tasks are not contingent on power-law structure in the underlying data and can be observed from random graph walks up to natural language using transformers. The scaling exponents monotonically decline with data complexity, and best practice in empirical fitting calls for robust regression methods over traditional two-dimensional analytic formulas. The results challenge received intuition linking scaling exponents rigidly to data structure, extending the theoretical landscape for understanding the origins and limits of scaling in deep learning systems.