---
title: Scaling Laws in Complex Systems
url: https://www.emergentmind.com/topics/scaling-laws
type: topic
---

# Scaling Laws in Complex Systems

Scaling laws are relations between an observable quantity and a measure of scale—system size, model size, dataset size, compute budget, mass, or characteristic length—most commonly in power-law form, \( f(x) = c x^n \), with homogeneity \( f(\lambda x) = \lambda^n f(x) \) under dilation [1106.1270]. In contemporary usage, the term also covers logarithmic laws, joint multivariate laws with irreducible floors, and exponent identities among critical variables. Across the literature, scaling laws serve two closely related purposes: they expose the organizing constraints of a system, and they enable controlled extrapolation from accessible regimes to otherwise inaccessible ones [2108.07686].

## 1. Mathematical structure and canonical forms

The canonical scaling law is the power law \( f(x) = c x^n \), where the exponent \( n \) encodes how rapidly an observable changes under scale transformation [1106.1270]. In machine learning, the corresponding empirical forms are typically written as a power-law decay of loss or error with respect to a resource variable, such as \( e = k x^{-\alpha} \), or as a joint law such as
\[
\epsilon(m, n) \approx a \cdot n^{-\alpha} + b \cdot m^{-\beta} + c_\infty,
\]
where \( m \) is model size, \( n \) is dataset size, and \( c_\infty \) is an error floor [2402.12813][2108.07686]. In other settings, logarithmic forms are equally central: empirical networks exhibit \( y = a + b \log_{10} n \) for mean geodesic distance and degree assortativity, while power laws remain appropriate for mean degree and clustering [2603.20973].

A recurring structural feature is the presence of additive or asymptotic constants. In acoustic modeling, for example,
\[
L(D) = L_\infty + \left(\frac{D_C}{D}\right)^{\alpha_D}, \qquad
L(N) = L_\infty + \left(\frac{N_C}{N}\right)^{\alpha_N},
\]
and the combined law
\[
L(N, D) =
\left[
\left(L_\infty\right)^{1/\alpha}
+ \left(\frac{N_C}{N}\right)^{\alpha_N/\alpha}
+ \left(\frac{D_C}{D}\right)^{\alpha_D/\alpha}
\right]^\alpha
\]
makes the irreducible loss explicit [2106.09488]. In critical phenomena, scaling laws may instead relate exponents to one another rather than an observable directly to scale, as in gravitational collapse, where exponent identities constrain the admissible critical behavior [1512.07095].

These forms are not interchangeable. A plausible implication is that the functional class—power law, logarithmic law, or exponent identity—tracks the dominant mechanism: self-similarity, small-world shortcutting, finite-resource saturation, or critical-point homogeneity.

## 2. Statistical learning theory and the origin of neural exponents

In deep learning, precise power-law relations between population loss and either dataset size or parameter count motivated a theoretical taxonomy distinguishing four regimes: variance-limited and resolution-limited scaling, each with respect to dataset size and model size [2102.06701]. In the variance-limited regime, the exponents are universal, with
\[
L(D) - L(\infty) = \mathcal{O}(D^{-1}), \qquad
L(w) - L(\infty) = \mathcal{O}(w^{-1}),
\]
whereas in the resolution-limited regime the exponents depend on intrinsic manifold dimension and smoothness, yielding laws such as \( L(D) = \mathcal{O}(D^{-n/d}) \) and \( L(P) = \mathcal{O}(P^{-n/d}) \) [2102.06701]. In large-width or random-feature settings, kernel spectra furnish a dual description: if \( \lambda_i \sim i^{-(1+\alpha_K)} \), then both \( L(D) \) and \( L(P) \) scale with exponent \( \alpha_K \) [2102.06701].

A more recent formalization identifies scaling laws as redundancy laws. Under kernel regression with covariance eigenvalues \( \lambda_i \asymp i^{-1/\beta} \), the expected excess risk obeys
\[
\mathbb{E}\mathcal{E}(f_{\lambda^*,n}) \asymp n^{-\alpha},
\qquad
\alpha = \frac{2s}{2s + 1/\beta},
\]
where \( s \) is a source-condition smoothness parameter and \( 1/\beta \) is the redundancy index [2509.20721]. The exponent is therefore non-universal: flatter spectra imply higher redundancy and slower returns to scale, while steeper spectra increase \( \alpha \) [2509.20721].

Linear regression theory provides an explicit reconciliation between empirical neural scaling laws and the classical bias–variance decomposition. In an infinite-dimensional linear model trained by one-pass SGD, the reducible test error is
\[
\Theta\!\left(M^{-(a-1)} + N^{-(a-1)/a}\right),
\]
under a power-law covariance spectrum \( \lambda_j \propto j^{-a} \) with \( a > 1 \) [2406.08466]. The variance term, although it increases with model size in conventional analyses, is dominated by the other errors because of the implicit regularization of SGD, and therefore disappears from the dominant bound [2406.08466]. This suggests that monotone neural improvement with model size is not a contradiction of classical theory so much as a consequence of a particular optimization regime.

The same literature extends beyond dense training. Iterative magnitude pruning admits a joint law in width, depth, density, and data, together with an error-preserving invariant that permits width, depth, and pruning density to be interchanged at fixed error [2108.07686]. That result locates pruning within the same scaling-law framework as dense training rather than outside it.

## 3. Large-scale AI systems and compute-optimal scaling

The most direct large-model comparison in the supplied literature concerns linear-complexity language models. Three efficient architectures—TNL, HGRN2, and cosFormer2—were compared with LLaMA after training six model sizes from \(70\)M to \(7\)B parameters on \(300\)B tokens, with \(1{,}376\) intermediate checkpoints across validation loss, commonsense reasoning, long-context retrieval, and generation [2406.16690]. All families obeyed compute-based power laws of the form
\[
L(C)=\beta_C C^{\alpha_C},
\]
with closely matched loss exponents, approximately \(-0.075\) to \(-0.080\), and linear models supported slightly larger optimal model and data sizes at fixed compute [2406.16690]. The same study found that existing linear-complexity language models exhibit similar scaling capabilities as conventional transformer-based models while also demonstrating superior linguistic proficiency and knowledge retention, yet transformers remain significantly better on Needle In A Haystack retrieval, especially in very long contexts [2406.16690].

Code understanding exhibits the same phenomenology. For masked-language-model pretraining on code, the test error follows
\[
e = k x^{-\alpha},
\]
with \( x \) taken as model size, training data, or compute [2402.12813]. The study scaled transformer encoders from \(124\)M to \(1.5\)B parameters and trained CoLSBERT with \(1.5\)B parameters on approximately \(304\)GB and \(351\)B training tokens, using \(64 \times\) A100-40G GPUs, and reported higher Code Search MRR and Clone Detection MAP than prior \(\sim125\)M-parameter baselines [2402.12813]. Acoustic models trained with auto-predictive coding show analogous laws, and on \(11\)-layer transformers the fitted constants were \(L_\infty = 0.316\), \(D_C = 7.35 \times 10^{-23}\), \(\alpha_D = 0.01946\), \(N_C = 9.41 \times 10^{-25}\), and \(\alpha_N = 0.01601\) [2106.09488].

Scaling laws also organize reinforcement learning. In AlphaZero, playing strength scales as
\[
\gamma \propto N^{\alpha_N}
\quad\text{and}\quad
\gamma \propto C^{\alpha_C},
\]
with \(\alpha_N = 0.88\) for Connect Four and \(0.87\) for Pentago, and \(\alpha_C = 0.55\) for both games [2210.00849]. Combining these yields an optimal-size law
\[
N_{\mathrm{opt}}(C) = \left(\frac{C}{C_0}\right)^{\alpha_C^{\mathrm{opt}}},
\qquad
\alpha_C^{\mathrm{opt}} = \frac{\alpha_C}{\alpha_N},
\]
with empirical exponents \(0.62\) and \(0.63\) [2210.00849]. The same work reports that larger AlphaZero models are more sample efficient [2210.00849]. For transformer development, small-scale experiments can be predictive, but only conditionally: some fine-tuning tasks exhibit clear power laws, and extrapolation was found useful when \(R^2 \ge 0.95\), yet careful hyperparameter tuning and multiple runs for uncertainty estimation were necessary and partially offset the computational savings [2202.06387].

The idea generalizes to problem size. In Hex, AlphaZero frontiers across board sizes showed that performance achievable with a fixed amount of compute degrades predictably as the game gets larger and harder, and train-time and test-time compute can be traded off almost linearly in log-space while maintaining performance [2104.03113]. A plausible implication is that compute-optimal scaling is not restricted to fixed-task asymptotics.

## 4. Networks, geometry, and criticality

In spatial network formation, geometric constraints change the scaling class itself. When contacts are added under excluded-volume and planarity constraints, graph diameter \( D \) and radius of gyration \( R_{\mathrm g} \) scale algebraically,
\[
D \sim N^\nu, \qquad R_{\mathrm g} \sim N^\nu,
\]
rather than logarithmically as in unconstrained random networks [1607.01435]. The exponent lies between the space-filling and self-avoiding-random-walk limits; experimentally measured protein tertiary structures in \(3\)D give \(\nu \approx 0.42 \pm 0.04\), while the \(2\)D model gives \(\nu \approx 0.62 \pm 0.04\) [1607.01435]. The change from \( \log N \) to \( N^\nu \) is not a minor quantitative correction but a change in universality class induced by geometry [1607.01435].

Empirical network science displays a mixed picture of universal functional forms and domain-specific rates. Across \(254\) real-world networks from social, biological, informational, and technological domains, mean degree scales as \( \langle k \rangle = a n^b \), mean geodesic distance as \( \langle \ell \rangle = a + b \log_{10} n \), clustering as \( C = a n^b \), and degree assortativity as \( r = a + b \log_{10} n \) [2603.20973]. The domains share these functional forms, but the fitted exponents differ: social networks densify fastest, biological networks show the steepest logarithmic growth in path length, and technological networks show the fastest decay in clustering [2603.20973]. Configuration-model random graphs reproduce the scaling behavior remarkably well, while degree-corrected stochastic block models are slightly better [2603.20973].

Critical gravitational collapse supplies yet another use of scaling laws: relations among critical exponents. For spherically symmetric collapse, the exponents satisfy
\[
\alpha_n^{\pm} + \beta_n^{\pm}(\delta_n + 1) = 2,
\qquad
\beta_n^{\pm} = \nu_n^{\pm}(D - 2 + z_n^{\pm}),
\]
independently of the details of the gravity theory under consideration [1512.07095]. These identities are explicitly compared to scaling relations in thermodynamic and geometrical phase transitions [1512.07095]. The analogy does not imply identical microscopic dynamics; it does imply a shared homogeneity structure near criticality.

## 5. Biological and biomedical allometry

A recent biological program derives scaling laws from invariance of a macroscale system of coupled partial differential equations for fluid dynamics, nonlinear elasticity, and rigid body mechanics [2502.11398]. Under the transformation from one organism to another, length scales as \( \lambda L \), time as \( \alpha T \), density as \( \eta \rho \), elastic modulus as \( \frac{\eta \lambda^2}{\alpha^2}E \), viscosity as \( \frac{\lambda^2}{\alpha}\nu \), and gravity as \( \frac{\lambda}{\alpha^2}g \), while the Reynolds, Froude, Strouhal, and Elastic Mach numbers are preserved [2502.11398]. For large, geometrically similar animals in the inertial regime,
\[
\frac{f_2}{f_1} = \left(\frac{L_2}{L_1}\right)^{-1/2}
= \left(\frac{M_2}{M_1}\right)^{-1/6},
\qquad
\frac{v_2}{v_1} = \left(\frac{L_2}{L_1}\right)^{1/2}
= \left(\frac{M_2}{M_1}\right)^{1/6},
\]
and
\[
\frac{E_2}{E_1} = \frac{L_2}{L_1}
= \left(\frac{M_2}{M_1}\right)^{1/3}
\]
[2502.11398]. In microorganisms at low Reynolds number, the relevant laws reverse direction, with \( v_2/v_1 = (L_2/L_1)^{-1} \) and \( f_2/f_1 = E_2/E_1 = (L_2/L_1)^{-2} \) [2502.11398].

Cancer scaling departs sharply from classical sublinear metabolic laws. Across several malignant histologies, total lesion activity and metabolic tumor volume obey
\[
\mathrm{TLA} = \alpha\, \mathrm{MTV}^{\beta},
\]
with superlinear exponents in the approximate range \(1.2\)–\(1.4\), typically clustering around \(5/4\) [2104.12708]. The corresponding phenomenological growth model,
\[
\frac{dV}{dt} = \alpha V^\beta,
\]
implies explosive, super-exponential volumetric growth for \(\beta > 1\) [2104.12708]. The mechanistic account invokes evolutionary dynamics among subclones, and the associated nonlocal Fisher–Kolmogorov analysis yields \( \beta = 2 - 2/d \), so only in three dimensions does the theory produce the superlinear value \( \beta = 4/3 \) [2104.12708]. The paper further defines a Distance to Scaling Law metric,
\[
\mathrm{DSL}_j = \mathrm{TLA}_j - \alpha\,\mathrm{MTV}_j^{\beta},
\]
and reports prognostic value for deviations above or below the fitted allometric curve [2104.12708].

These biological examples underscore that a scaling exponent may encode dominant transport physics, elasticity, geometry, or evolutionary competition rather than a generic notion of “size effect.”

## 6. Physical sciences: from hadrons to coronal loops

In particle physics, scaling laws often express constituent structure. The dimensional quark counting rules give
\[
F_a(t) \sim \frac{1}{t^{n_a-1}}
\]
for electromagnetic form factors of a composite object with \( n_a \) constituents, and for exclusive two-body reactions \( a+b\to c+d \),
\[
\frac{d\sigma}{dt} \sim \frac{1}{s^{n-2}},
\]
where \( n = n_a + n_b + n_c + n_d \) [1106.1270]. The same paper also discusses generalized spin–mass relations, including
\[
J = \hbar \left(\frac{m}{m_p}\right)^{1 + 1/n}
\]
for string-, disk-, and ball-like morphologies, and the Kerr bound
\[
J_{\mathrm{Kerr}} = \frac{Gm^2}{c}
= \hbar \left(\frac{m}{m_{\mathrm{Planck}}}\right)^2
\]
for rotating black holes [1106.1270].

Planetary-dynamo theory uses power laws to connect hidden core dynamics to observable dipole moments and rotation rates. Under the assumption that the nonlinear inertial term is negligible because of the small planetary Rossby number, the proposed laws include
\[
Ro \sim \left(\frac{P}{\Omega^3 l^2}\right)^{4/9},
\qquad
\frac{(B^2/\rho\mu)}{u^2} \sim Ro^{-2/3},
\qquad
\tau_\Omega \sim Ro^{-3/4},
\]
together with \( Rm \sim A/Ro^{1/2} \) and a convection-column aspect ratio scaling as \( Ro^{1/2} \) [1302.7140]. The novelty relative to earlier dynamo scalings is the explicit exclusion of inertial balance from the derivation [1302.7140].

Solar-loop thermodynamics furnish a further example of a scaling law surviving a change in physical regime. The classic static coronal-loop laws are
\[
T_M \simeq 1.4 \times 10^3 (PL)^{1/3},
\qquad
E_H \simeq 9.8 \times 10^4 P^{7/6} L^{-5/6},
\]
and for collision-dominated thermal conduction the dynamic extension preserves the same functional dependencies on pressure and length, while replacing the constants of proportionality by Mach-number-dependent factors [2010.02837]. In turbulence-limited conduction, by contrast, the temperature dependence of the conductivity changes enough that the scaling laws depart from their static forms and the return enthalpy flux acquires a limiting Mach number set by the turbulent mean free path [2010.02837].

## 7. Predictive use, universality, and caveats

Scaling laws are attractive because they convert expensive search into measured extrapolation. In particle-physics amplitude surrogates, the test loss is modeled as
\[
L(X,Y,Z) = (X_c/X)^{\alpha_X} + K_X(Y,Z),
\]
with \( X \) taken as network size, training data, or compute [2601.13308]. The exponents are linked to the intrinsic dimension \( d = 3n_f - 4 \) of the phase-space manifold, and the paper reports the estimate \( \alpha_X \approx 4/d \) as a practically useful rule for reaching predefined precision targets [2601.13308]. In jet classification, six physically motivated classifiers obey
\[
L(T) = A T^{-\alpha_T} + C,
\]
but with distinct exponents—\(0.105 \pm 0.003\) for kNN EMD, \(0.101 \pm 0.004\) for DNN on LOT, down to \(0.037 \pm 0.001\) for kNN EFP in the large-\(T\) regime—so the best classifier can change as the dataset is scaled [2312.02264]. That finding makes fixed-dataset comparisons potentially misleading [2312.02264].

The same predictive logic appears in small-scale transformer experiments and board games. For language understanding tasks that exhibit clear scaling laws, larger-model performance can be predicted from much smaller models, but only after scale-specific hyperparameter tuning and hierarchical-bootstrap uncertainty estimation [2202.06387]. In Hex, laws fitted on small boards predict larger and harder boards, and the residual prediction error drops exponentially as more small board sizes are included in the fit [2104.03113]. These are operational successes, not merely descriptive ones.

At the same time, the literature repeatedly rejects a single universal exponent. Neural exponents vary across variance-limited, resolution-limited, and redundancy-controlled regimes [2102.06701][2509.20721]. Network statistics share functional forms across domains while differing in rate parameters [2603.20973]. Linear-complexity language models match transformers on many linguistic measures yet remain weaker at sparse long-context retrieval [2406.16690]. A careful summary is therefore that scaling laws are widespread, but the exponent is usually a compressed statement of domain-specific structure—data redundancy, manifold dimension, transport regime, geometric constraint, modularity, or evolutionary dynamics—rather than a field-independent constant.

Source: https://www.emergentmind.com/topics/scaling-laws