Scaling Laws in Architectural Discovery

• Scaling Law for Architectural Discovery is the formulation of empirical guidelines that relate model size, dataset size, and compute to performance metrics in deep learning.
• It derives mathematically precise prescriptions—such as the double-asymptote law and overfitting penalty—enabling systematic model scaling and resource optimization.
• The analysis shows that macro-scale adjustments in model capacity drive performance improvements, reducing the need for fine-tuning detailed architectural features.

A scaling law for architectural discovery in deep learning and neural computation characterizes how the performance metrics of learning systems evolve as model size, dataset size, and computational resources scale, with particular attention to the impact—or lack thereof—of detailed architectural modifications. Scaling law analysis provides mathematically precise prescriptions for optimal resource allocation, model scaling strategies, and principled model selection, transforming architectural discovery from ad hoc tuning into a theoretically informed procedure. The following sections articulate the formal foundations, quantitative relationships, and practical implications of scaling laws for model architecture discovery, drawing primarily on large-scale empirical studies, statistical and random-matrix theory models, and systematic ablation experiments.

1. Formal Scaling Law Relationships

At the core of scaling law analysis is the empirical observation that loss (typically test cross-entropy or generalization error) obeys power-law behavior with respect to key scaling variables—model non-embedding parameter count ($N$), total number of training tokens ($D$), and training compute ($C$):

$$\begin{align*} L(N) &\approx (N_c / N)^{\alpha_N} \qquad \text{with } \alpha_N \approx 0.076 \ L(D) &\approx (D_c / D)^{\alpha_D} \qquad \text{with } \alpha_D \approx 0.095 \ L(C_{\min}) &\approx (C_c^{(\min)}/C_{\min})^{\alpha_C^{(\min)}} \qquad \text{with } \alpha_C^{(\min)} \approx 0.05 \end{align*}$$

The overfitting penalty and the joint influence of $N$ and $D$ are unified by the "double-asymptote" law:

$$L(N, D) = \left[ (N_c / N)^{\alpha_N/\alpha_D} + (D_c / D) \right]^{\alpha_D}$$

When training under a fixed compute budget ($C = 6 N B S$, with $B$ the batch size and $S$ the number of steps), the compute-optimal scaling strategies can be derived analytically. The optimal parameter count and training steps scale as:

$$\begin{align*} N &\propto C_{\min}^{\alpha_C^{(\min)}/\alpha_N} \approx C_{\min}^{0.73} \ S &\propto C_{\min}^{\alpha_C^{(\min)}/\alpha_S} \approx C_{\min}^{0.03} \end{align*}$$

The scaling law further prescribes that, in order to avoid overfitting, dataset size should scale sublinearly with model size, specifically:

$D \propto N^{\alpha_N / \alpha_D} \approx N^{0.74}$

2. Architectural Factors: Depth, Width, and Hyperparameters

Empirical investigation reveals that, within broad operating ranges, the detailed "shape" of the architecture—depth, width, number of attention heads—has only a weak effect on scaling performance once $N$ is held constant. Redistribution of parameters between depth and width alters test loss by only a few percentage points. Thus, it is the effective model scale (here, the non-embedding parameter count) that overwhelmingly determines predictive performance, rather than the micro-allocation of those parameters among layers or features.

This observation supports the prescription that architectural discovery, for large-scale models, should first concentrate on overall scaling and resource balance before pursuing fine architectural refinements. The focus is on "macro-architecture" rather than meticulous "shape" optimization.

3. Overfitting, Early Stopping, and Learning Curves

The interaction between model and data size brings about a predictable overfitting regime, quantified by:

$\delta L \propto N^{0.74}/D$

Performance degradation owing to overfitting is thus predictable and can be managed by sublinear dataset scaling or by appropriately tuning early stopping criteria. The dynamics of training are well approximated by a two-term power law:

$$L(N, S) = (N_c / N)^{\alpha_N} + (S_c / S_{\min})^{\alpha_S} \quad \text{with } \alpha_S \approx 0.76$$

This mathematical form allows projection and extrapolation of final performance from early training data, establishing when it is computationally optimal to stop training and reset resources for a new scaling regime.

4. Implications for Compute Allocation and Efficiency

The scaling law analysis demonstrates that, under a fixed compute budget, the most efficient path is to preferentially allocate resources towards scaling up model size, rather than increasing the length of training. Optimal models should be trained for fewer steps, on a relatively small but sufficient dataset, and stopped well before convergence to minimize unnecessary compute expenditures.

Practical recommendations for compute allocation deduced from the scaling exponents are:

• Prioritize increasing $N$ over $S$ for performance improvements under compute constraints.
• Increase $D$ only sublinearly with $N$.
• Utilize early stopping when further training yields diminishing or negative returns.

The strategic gain is that large models trained with carefully apportioned data and judicious early stopping achieve far better sample and compute efficiency than smaller models trained to convergence.

5. Quantitative Blueprint for Architectural Discovery

Taken together, these laws provide a quantitative and predictive framework for architectural discovery:

• New model proposals should be scaled along the $N, D, C$ axes according to the empirical exponents.
• Exploration of novel architectures should prioritize increasing total model capacity, with secondary emphasis on architectural "shape" only where macro-scaling is already optimized.
• Efficient architecture search is best performed in regimes where the dominant determinant of loss is not architecture shape, but overall scale—applying scaling law projections rather than full convergence for each candidate.

The underlying implication is that scaling phenomena in LLMs, by analogy with thermodynamic laws, provide a kind of "universal equation of state" for model optimization—offering actionable prescriptions for model sizing, data budget, and compute allocation.

6. Broader Impact and Extensions

These empirical and theoretical findings have substantially influenced the design principles for state-of-the-art LLMs and have provided the theoretical justification for prioritizing large-scale scaling—leading to the development of billion-parameter models trained on massive textual corpora. Subsequent investigations (e.g. in image recognition, time series forecasting, multimodal learning) have frequently extended or refined these scaling laws, while noting that specific exponents and forms must be calibrated to each learning context or domain.

The research has unified previously heuristic architectural discovery workflows under a predictive, resource-aware model. Its adoption has led to the acceleration of architectural innovation, shifting the field from bespoke engineering toward systematic scaling guided by empirically determined laws.

7. Summary Table of Key Scaling Relationships

Factor(s)	Scaling Law	Typical Exponent
Model size ($N$)	$L(N) = (N_c/N)^{\alpha_N}$	$\alpha_N \approx 0.076$
Data size ($D$)	$L(D) = (D_c/D)^{\alpha_D}$	$\alpha_D \approx 0.095$
Compute ($C$)	$$L(C_{\min}) = (C_c^{(\min)}/C_{\min})^{\alpha_C^{(\min)}}$$	$\alpha_C^{(\min)} \approx 0.05$
Data-model scaling	$D \propto N^{\alpha_N / \alpha_D}$	$0.74$
Overfitting penalty	$\delta L \propto N^{0.74}/D$	$0.74$
Training speed	$S \propto C^{0.03}$ (minimal dependence)	$0.03$

These relationships codify a paradigm in which architectural discovery is reframed: detailed design is subordinate to global scaling, with the latter governed by robust, empirically determined power-law laws.