---
title: Scaling Laws Re-evaluated
url: https://www.emergentmind.com/topics/scaling-laws-re-evaluated
type: topic
---

# Scaling Laws Re-evaluated

Scaling laws in machine learning and the physical sciences establish quantitative relationships between measures of scale (such as model size, data size, or available compute) and empirical or theoretical performance. Over the past several years, empirical scaling laws—most commonly power-law relationships—have been extensively documented in modern deep learning, statistical physics, and complex systems. This encyclopedia entry provides a re-evaluation of scaling laws, integrating recent developments across application domains, highlighting both universal findings and emerging limits or caveats.

## 1. Mathematical Foundations of Scaling Laws

Scaling laws traditionally describe how a performance metric $L$ (error, loss, or success probability) varies as a function of a resource or design parameter $x$ (model size $N$, dataset size $D$, compute $C$, etc.). The canonical form is a power law:
\[
L(x) = A x^{-\alpha} + B
\]
where $A, B > 0$, and $\alpha > 0$ is the scaling exponent. This model predicts that increasing $x$ yields diminishing improvements. In practice, dependencies on multiple factors are modeled additively or multiplicatively, such as:
\[
L(N, D, C) = A_N N^{-\alpha_N} + A_D D^{-\alpha_D} + A_C C^{-\alpha_C} + L_\infty
\]
for language and vision models [2108.07686, 2506.04598].

However, empirical studies reveal that this simple form is insufficient in several settings: phase transitions, inflection points, or nonmonotonic “double descent” phenomena necessitate more general functional forms. The Broken Neural Scaling Law (BNSL) is designed to address these cases:
\[
L(x) = C x^{-\alpha_1} [1 + (x / x_b)^s ]^{(\alpha_1-\alpha_2)/s} + D
\]
allowing for smoothly broken power-laws and “regime” changes [2210.14891].

## 2. Universal and Domain-Specific Findings

### Empirical Universality

Scaling laws with power-law form have been empirically observed in a wide range of domains:

- **Language Modeling**: Cross-entropy loss decreases as a power-law in compute and model/data size, with fitted exponents in the range $0.04$–$0.1$ for models like GPT-3, LLaMA, and their derivatives [2108.07686, 2406.16690].
- **Vision**: Image classification and zero-shot/few-shot adaptation exhibit power-law scaling in data and model size, with exponents typically between $0.3$ and $1.1$ (steeper for few-shot, out-of-distribution tasks) [2110.06990].
- **Dense Retrieval and Reranking**: Retrieval loss and reranker metrics (NDCG, MAP) obey saturating power laws in model and data scale, even for discontinuous downstream metrics [2403.18684, 2603.04816].
- **Symbolic Regression**: Deep transformer models for symbolic regression display steep scaling laws, with loss $L(C) \approx 1.3\times 10^{3} C^{-0.21}$, outperforming analogous language model exponents per FLOP [2510.26064].

### Theory and Mechanistic Insights

Mathematical results establish that power-law scaling exponents are determined by the spectral (eigenvalue) decay of data or feature covariances:

- In kernel and linear regression:
  \[
  \mathbb{E}[L] = \sigma^2 + \Theta(m^{1-a}) + \Theta((N_{\rm eff}\gamma)^{-(a-1)/a})
  \]
  where $a$ is the exponent in the covariance tail $\lambda_i \sim i^{-a}$ [2503.01314].
- In kernel ridge regression, the learning curve exponent is
  \[
  \alpha = \frac{2s}{2s + 1/\beta}
  \]
  where $\beta$ controls the tail of the covariance spectrum, and $s$ is a source smoothness parameter; this explicit dependence formalizes the "redundancy law" for learning efficiency [2509.20721].
- The "redundancy law" further predicts that systems with flatter (slower-decaying) spectra see slower scaling improvements; highly compressible data enables faster scaling.

### Robustness and Practical Use

Scaling laws robustly extrapolate future gains from smaller experiments in many regimes [2202.06387, 2104.03113, 2603.04816]. For instance, NDCG and MAP for billion-parameter rerankers can be predicted accurately from $<$400M-parameter models [2603.04816]. In pre-training/fine-tuning, performance extrapolation with high $R^2$ is possible provided careful hyperparameter tuning and sufficient scale diversity [2202.06387].

## 3. Limits, Deviations, and Interpretability

### Regularity Violations and Non-Power-Law Behavior

Not all domains or metrics exhibit smooth scaling laws:

- Certain tasks or evaluation metrics (e.g., MRR, contrastive entropy) can show non-monotonic or erratic scaling behavior, breaking power-law predictability even as other metrics scale reliably in the same system [2603.04816].
- "Double descent" and regime-shift phenomena (common in overparameterized models) appear as sharp inflection points or nonmonotonicity in the scaling law. These cannot be captured by single-slope power laws and necessitate a smoothly broken form [2210.14891].

### Influence of Data, Architecture, and Optimization

- Redundancy bottlenecks: Mixed-domain pretraining, dataset overlap, and multi-modal training can sharply reduce scaling exponents if the heaviest-tailed (most redundant) component dominates [2509.20721, 2507.09404].
- Architectural changes: Mixture-of-Experts, attention mechanisms, and feature learning can quantitatively shift scaling behavior; there remains ongoing work to link such changes to precise exponent shifts [2510.26064].
- Hyperparameters, especially batch size and learning rate, may scale nontrivially with model or compute size. In symbolic regression, optimal learning rate increases with scale, in contrast to the decreasing trend in standard LMs [2510.26064].

### Theoretical Challenges in Explanation

Multiple works have shown that traditional VC-based theory is largely vacuous in the scaling regime; instead, random matrix theory, spectral analysis, and deterministic equivalence are key technical tools to explain observed learning curves and error plateaus [2503.01314, 2502.01585]. The norm-based capacity theory predicts that when model norm is the relevant complexity measure, classical U-shaped learning curves reappear, and double descent disappears [2502.01585].

## 4. Comparison, Optimization, and Automated Discovery

### Model and Dataset Comparison

Cross-scale scaling law derivation enables principled comparison between models and pre-training procedures:

- In vision-language pre-training, scaling-law fits reveal systematically better scaling exponents and lower irreducible errors for generative+contrastive training (MaMMUT) versus contrastive-only (CLIP), consistently across classification, retrieval, and segmentation tasks [2506.04598].
- Data mixture optimization: Closed-form (or numerically tractable) scaling law formulas allow direct calculation of the performance-maximizing domain mixture for any compute budget, outperforming costly trial-and-error approaches [2507.09404].

### Automated Law Discovery

Evolutionary algorithms guided by LLMs can now autonomously rediscover and in some cases surpass human-derived scaling laws, co-optimizing symbolic expressions and fitting routines for best cross-group fit. The EvoSLD system yields parsimonious, interpretable forms and outperforms traditional symbolic regression or naive approaches by orders of magnitude on held-out NMSE in multiple real-world settings [2507.21184].

## 5. Caveats, Critiques, and Social-Scientific Reappraisal

### Sociotechnical Limitations of "Universal" Scaling

Recent research raises critical concerns regarding the universal applicability of scaling laws:

- Metrics as proxies: The metrics used in most scaling analyses may not adequately capture the plural notions of "quality" relevant to diverse user communities; large models can degrade on minoritized subgroups or alternate constructs as size increases [2307.03201].
- Inverse scaling and subgroup degradation: Empirical work demonstrates that beyond a critical point, models may plateau or even worsen on specific constructs (truthfulness, fairness, low-resource language coverage), contradicting the simplicity of universal scaling curves [2307.03201].
- Need for localized scaling laws: Disaggregated scaling curves per subgroup, participatory metric design, and non-scalable, community-specific architectures are recommended to avoid misleading averages masking harm [2307.03201].

### Predictability and Fundamental Uncertainty

A limit to predictability arises when sharp phase transitions or inflection points fall outside the training regime; BNSL fitting can only extrapolate such features if candidate breaks are sampled by experimental designs [2210.14891].

## 6. Practical Methodologies and Implementation Guidance

Scaling-law guided experimentation benefits from precise protocol design:

- Fit in (log,log) space after extracting the compute-optimal Pareto frontier.
- Use rigorous cross-validation and, for automated discovery, co-evolve both algebraic form and optimization subroutine [2507.21184].
- Optimize data/model allocation for fixed budget by closed-form constraint minimization, incorporating costs for annotation, training, and even inference [2403.18684].
- For new tasks or domains, use multiple small-scale runs to probe for regime breaks or anomalous scaling, revising law forms as needed [2202.06387, 2210.14891].
- Quantify and report uncertainty by bootstrapping and out-of-sample extrapolation errors; R$^2$ values $>0.95$ indicate reliable forecasting but special attention is needed near regime shifts or when observed performance departs systematically from the predicted law [2202.06387].

## 7. Future Directions and Open Questions

- Extension of scaling-law analysis to new domains including symbolic regression with more variables, denser/correlated regimes in kernel methods, DNA denaturation transitions with environmental/sequence heterogeneity, or complex multi-stage retrieval pipelines [2510.26064, 2503.01314, 2103.08725, 2603.04816].
- Active design of experiments for optimal law estimation and detection of regime changes.
- Direct measurement and control of feature covariance spectra in large language models as a predictor of scaling exponents [2503.01314, 2509.20721].
- Determination of scaling behavior in models beyond current compute regimes, including multi-billion/multi-trillion parameter models, and test of law extrapolation beyond currently observed domains.
- Sociotechnical integration: development of plural, community-specific scaling laws and metrics, acknowledging irreducible value tensions in large-scale AI deployment [2307.03201].

---

**References**  
- [2510.26064] Towards Scaling Laws for Symbolic Regression  
- [2503.01314] Scaling Law Phenomena Across Regression Paradigms: Multiple and Kernel Approaches  
- [2509.20721] Scaling Laws are Redundancy Laws  
- [2108.07686] Scaling Laws for Deep Learning  
- [2210.14891] Broken Neural Scaling Laws  
- [2507.09404] Scaling Laws for Optimal Data Mixtures  
- [2403.18684] Scaling Laws For Dense Retrieval  
- [2603.04816] Scaling Laws for Reranking in Information Retrieval  
- [2506.04598] Scaling Laws for Robust Comparison of Open Foundation Language-Vision Models and Datasets  
- [2104.03113] Scaling Scaling Laws with Board Games  
- [2110.06990] Scaling Laws for the Few-Shot Adaptation of Pre-trained Image Classifiers  
- [2202.06387] Scaling Laws Under the Microscope: Predicting Transformer Performance from Small Scale Experiments  
- [2502.01585] The Shape of Generalization through the Lens of Norm-based Capacity Control  
- [2507.21184] EvoSLD: Automated Neural Scaling Law Discovery With Large Language Models  
- [2307.03201] Scaling Laws Do Not Scale

Source: https://www.emergentmind.com/topics/scaling-laws-re-evaluated