Iso-Depth Scaling Laws

• Iso-depth scaling laws are mathematical regularities defining how performance scales with width at fixed model depth in various architectures.
• They decouple the effects of width from depth, allowing precise measurement of improvements under specific resource and latency constraints.
• Empirical studies across transformers, looped models, and quantum networks validate power-law fits that guide efficient resource allocation.

Iso-depth scaling laws are mathematical regularities that describe how the performance of neural networks or related models improves as a function of architectural width or analogous resources, with model depth (or its quantum, looped, or context analogues) held strictly constant. These laws allow the isolation and quantification of width, data, or other scaling axes—independently of depth—in regimes where the interplay between architectural shape and scale is nontrivial, such as large transformer LLMs, looped networks, hybrid quantum-classical classifiers, and deep linear analogues. Iso-depth analysis has become central in the empirical and theoretical literature on model scaling, providing actionable prescriptions for resource allocation under fixed-latency, memory, or stability constraints.

1. Formal Definition and Theoretical Motivation

Iso-depth scaling laws focus on the regime where model depth or analogous "virtual depth" is held fixed. The number of layers, recurrent applications, or circuit layers, denoted $d$, $r$, or $L$, is set to a constant, and performance is studied as a function of width ($w$), number of qubits ($Q$), or overall parameter count ($p$), as well as data scale ($T$, $D$), while all other hyperparameters are controlled. In canonical transformer LLMs, the iso-depth law quantifies how performance (e.g., loss, perplexity) scales with width at a specified depth, decoupled from the width–depth entanglement that underpins standard parameter scaling laws. This procedure clarifies which axes of model scaling—width, depth, data—yield the steepest marginal benefits in loss reduction or task performance (McLeish et al., 7 Feb 2025).

The primary motivation is that single-parameter scaling laws (e.g., as in Kaplan et al. 2020 or Hoffmann et al. 2022) collapse architecture choices onto a single size axis and implicitly assume a canonical width–depth or parameterization ratio. Iso-depth studies expose the distinct role of width (or width analogues), supporting optimized model design for fixed-depth, fixed-latency, or memory-constrained deployments. They also inform practical questions such as: given a latency limit (depth fixed), how much does increasing width reduce perplexity, and with what exponent? (McLeish et al., 7 Feb 2025)

2. Mathematical Formulation and Model Classes

Joint and Iso-Axis Power Laws

In standard empirical scaling, validation loss $L$ (in nats) is assumed to decompose as a sum of asymptotic power laws over effective parameters and data:

$$L(p, T) = \frac{A}{p^\alpha} + \frac{B}{T^\beta} + \varepsilon$$

with log-log linearization: $r$0

The Gemstones model suite extends this to a joint width–depth–parameters–tokens law: $r$1 An iso-depth law is obtained by fixing $r$2 and profiling $r$3 as a function of $r$4, $r$5, and $r$6 at that value:

$r$7

In looped and quantum models, the axes are adapted:

• In looped models, effective depth is $r$8 (Parcae), with $r$9 fixed (iso-depth), and scaling in loop count $L$0 (Prairie et al., 14 Apr 2026).
• In quantum neural networks, circuit depth $L$1 is fixed, and scaling is primarily in number of qubits $L$2 (Vyskubov et al., 7 Apr 2026).

Example: Looped Models

A representative iso-depth power law for looped LLMs is

$L$3

where $L$4 is the recurrence count, $L$5 and $L$6 the unique and recurrent parameter budgets, and $L$7 the recurrence-equivalence exponent quantifying how much capacity each recurrence adds at fixed depth (Schwethelm et al., 22 Apr 2026).

Example: Quantum-Classical Hybrid Classifiers

At fixed circuit depth $L$8, model performance $L$9 with respect to qubit count obeys a saturating exponential:

$w$0

where $w$1 is any target metric (Accuracy, QCE, PR-AUC) (Vyskubov et al., 7 Apr 2026).

3. Empirical Characterization Across Domains

Transformers and Gemstones

The Gemstones suite spans widths $w$2, depths $w$3, and up to 2 billion parameters, with iso-depth fits constructed by selecting checkpoints at fixed $w$4. Empirically, the width exponent in the iso-depth regime is $w$5 (near invariance for $w$6–$w$7), with $w$8 for log–log fits. In contrast, iso-width studies (fixed $w$9) yield a depth exponent $Q$0. Thus, width carries steeper marginal returns than depth in this regime (McLeish et al., 7 Feb 2025).

Looped LLMs

In looped architectures with recurrence count $Q$1, iso-depth scaling isolates the effect of repeated application at fixed unique depth. The recurrence-equivalence exponent $Q$2 (95% CI $Q$3) is interpreted as the fraction of unique parameter capacity gained per recurrence. For instance, $Q$4 recurrences yield only $Q$5 unique-block equivalents, i.e., less than half the returns of full unrolling. The scaling law fits with $Q$6 over a 50-fold compute budget span (Schwethelm et al., 22 Apr 2026).

Stable Looped Architectures (Parcae)

For Parcae architectures, the iso-depth law at fixed unique-parameter count $Q$7 and unique depth $Q$8 reflects validation loss decreasing as $Q$9 ($p$0), with optimal resource allocation along the frontier $p$1, $p$2 for training FLOP budget $p$3. At test time, further increases in loop count (compute) yield exponential decay toward a loss floor, i.e., rapidly diminishing returns beyond $p$4 ($p$5 in recurrences for 140M-parameter models) (Prairie et al., 14 Apr 2026).

Hybrid Quantum Neural Networks

For hybrid QNNs with fixed depth $p$6, accuracy and quantum expressibility increase with qubit count according to saturating exponentials that plateau above $p$7–$p$8 for all tested benchmarks. The corresponding iso-depth law provides precise guidance for optimal $p$9 at target performance, avoiding superfluous resource expenditure (Vyskubov et al., 7 Apr 2026).

Linear In-Context Learning Models

In the "ISO" regime for linear self-attention, width never bottlenecks, and iso-depth scaling shows that depth only improves performance when context length $T$0 is small. For $T$1, even shallow models (depth $T$2) saturate risk, and there is no substantive width–depth trade-off to exploit in the iso-depth regime (Bordelon et al., 1 Oct 2025).

4. Methodological Considerations and Fitting Practices

Iso-depth analyses require careful experimental design:

• Sweep width (or analogous axis) over a dense and well-spaced grid at each fixed depth.
• Hold data (tokens, examples) and optimization hyperparameters (learning rate, schedule) as constant as feasible.
• Avoid early-phase artifacts by restricting to checkpoints well past the early data regime (e.g., $T$3B tokens for LLMs).
• For architectures reliant on discrete parameters (e.g., number of attention heads), handle integer-induced "jagginess" by interpolation or temporary relaxation where necessary.
• Use robust loss functions (typically Huber loss) and repeated random-restart optimization for parameter fitting in joint models (McLeish et al., 7 Feb 2025, Schwethelm et al., 22 Apr 2026).

In looped models, special care must be taken to separate capacity scaling (additional loop recurrences) from compute scaling (increased FLOPs) and to use stability-enforcing parameterizations to avoid training divergence at high recurrence counts (Prairie et al., 14 Apr 2026).

5. Practical Implications and Prescriptive Guidance

Iso-depth scaling laws yield concrete prescriptions for model scaling:

• At fixed depth, increasing width (or equivalent resources) delivers power-law improvements in loss, typically with exponents $T$4 (transformers, Parcae).
• For looped models, the recurrence-equivalence exponent $T$5 quantifies the effective capacity gained per recurrence, enabling budget-constrained design calculations for desired validation loss.
• When maximizing performance under compute constraints, both loop count (depth proxy) and data should be increased according to power-law allocations along the efficient frontier (Prairie et al., 14 Apr 2026).
• In hybrid quantum models, early qubit increases yield substantial benefit, after which returns saturate; iso-depth fits guide practitioners to the minimal $T$6 needed for target error (Vyskubov et al., 7 Apr 2026).

A plausible implication is that for deployments constrained by memory, latency, or custom hardware limits (fixed depth), width and data scaling is generally more effective for quality improvement—subject to the exponents obtained from empirical iso-depth fits.

6. Limitations, Sensitivities, and Future Directions

Several caveats and open problems are noted in the literature:

• Iso-depth exponents can shift ($T$7) under alternative learning-rate schedules, early-phase data exclusion, or embedding parameter handling.
• Integer constraints (attention head count, discrete loops) can introduce noise into scaling fits.
• Empirical power-law regimes may not extrapolate to extreme model sizes ($T$8 FLOPs) or very small models (Prairie et al., 14 Apr 2026).
• In linear or isotropic in-context learning, iso-depth scaling is largely trivialized as width ceases to be a meaningful bottleneck and context length dominates (Bordelon et al., 1 Oct 2025).

Future research aims to systematically tabulate per-depth iso-width and per-width iso-depth exponents, extend studies to deeper and wider regimes, and generalize findings to more structured architectural modifications (e.g., MLP expansion, attention head grouping) or new modalities. In looped and nonstandard architectures, raising the recurrence-equivalence exponent $T$9 above baselines (e.g., via efficiency optimizations) is identified as a priority goal for recovering more unique-capacity equivalence (Schwethelm et al., 22 Apr 2026, Prairie et al., 14 Apr 2026).