How Model Growth, Recursion, and Boundary Operators Influence Scaling Exponents

Abstract: Scaling laws predict how loss decreases with increases in computation. We show, contrary to conventional wisdom, that architectural interventions can modify scaling exponents in pre-training, leading to exponential improvements in performance with increases in computation. As an anchoring point, we consider the architectural formulation of looped transformers. Although not typically used in this way, looping, also known as recursive depth, provides a mechanism for model growth, by increasing the number of loops during training. Model growth, with and without shared weights, provides the biggest changes to the scaling exponents. In particular, a 7.4B model growth architecture matches GPT-3 13B on CORE with roughly $20\times$ less compute, and has compute efficiency gains that increase with scale. Moreover, simply using a boundary operator in a vanilla transformer, which normalizes and injects an earlier block, also provides increasing compute-efficiency gains, although to a lesser extent. In the data-constrained, multi-epoch setting, standard looping has a useful regularizing effect, where we find it is compute-optimal to increase the number of loops with scale. These results can be understood through the lens of computational depth: for a given computational budget, we wish to increase the usable depth of the transformer, which can lead to efficiency gains that increase with scale.

• The paper reveals how specific architectural changes, particularly model growth and boundary operators, influence the compute-optimal scaling exponents of pretraining models rather than just shifting the constants.
• Architectural interventions like boundary normalization with input re-injection and depth increase the scaling exponents of vanlla models.
• Different strategies are favoured depending on the specific computational regime, the paper highlights the optimal ways like model growth, boundary normalization, especially tied recurrence is preferable in some situations.

The paper studies whether architectural changes can alter the exponent of compute-optimal pretraining scaling laws rather than merely shifting their constant. Its central claim is that model growth and a suitable boundary operator produce scale-dependent compute-efficiency gains, while weight sharing primarily provides parameter efficiency or regularization. The empirical analysis is conducted within a controlled prelude–core–coda family of decoder-only transformers, enabling separate comparisons of boundary conditioning, weight sharing, and depth growth (2609.19107).

Experimental framework and architectural factorization

The authors formulate each model as a prelude $\mathcal{P}$, a recurrently applied core $\mathcal{R}$, and a coda $\mathcal{C}$. The prelude maps the input to a representation $e$; the core transforms a state $h$ for $K$ passes; and the coda produces the output. Executed depth is therefore the sum of the prelude and coda depths plus $K$ times the core depth, whereas stored depth counts distinct parameterized blocks.

The main architectural axes are isolated as follows:

• Boundary operator: between core passes and before the coda, the model applies $\mathrm{Norm}(h)+\alpha e$. This both controls residual-stream growth and re-injects an early input representation.
• Weight sharing: Loop-$K$ reuses one core across all passes, whereas Untied-$K$ assigns independent weights to each pass.
• Model growth: training begins with two active core passes and later activates four. In the tied case, the existing core is reused; in the untied case, the trained core is copied and subsequently optimized independently.

The matched comparisons are designed to hold FLOPs, executed depth, or stored parameter count fixed as appropriate. Vanilla versus Operator-1 isolates the boundary operator; Loop-2 versus Untied-2 isolates weight sharing; and fixed-depth versus grown variants isolate activation of additional computation during training.

Figure 1: The prelude–core–coda family unifies standard transformers, boundary-conditioned looping, tied recurrence, untied depth, and model growth.

The training protocol uses FineWeb with a GPT-2 tokenizer, context length 2048, a fixed width-to-depth ratio of 128, Muon for matrix parameters, and AdamW for embeddings and the language-model head. Scaling ladders span approximately $\mathcal{R}$0 to $\mathcal{R}$1 FLOPs. Crucially, every architecture receives its own tuned recipe: base hyperparameters, tokens per stored parameter, growth timing, and learning-rate scaling are separately optimized. This is methodologically important because transferring the Vanilla recipe to Operator-1 incurs a loss penalty of approximately $\mathcal{R}$2 at $\mathcal{R}$3 and eliminates the apparent exponent improvement.

The compute-optimal loss is modeled as a power law above a fitted irreducible floor. The paper compares architectures through a compute multiplier: the amount of Vanilla compute required to reach the loss attained by a competing architecture at a specified budget. A constant multiplier indicates a vertical shift in the scaling curve; an increasing multiplier indicates an exponent improvement.

Architectural interventions change the scaling exponent

The main single-epoch result contradicts the conventional expectation that architectural changes affect only scaling constants. Operator-1, despite matching Vanilla in parameters and FLOPs, has a compute multiplier that increases from $\mathcal{R}$4 at $\mathcal{R}$5 FLOPs to $\mathcal{R}$6 at $\mathcal{R}$7 FLOPs. The boundary operator therefore changes the slope of the compute–loss relationship rather than merely improving the loss at every scale by a fixed amount.

Model growth produces a larger effect. Untied-Grow improves over Vanilla from $\mathcal{R}$8 at $\mathcal{R}$9 FLOPs to $\mathcal{C}$0 at $\mathcal{C}$1 FLOPs. Relative to Untied-2, whose core depth remains fixed during training, growth increases the multiplier from $\mathcal{C}$2 to $\mathcal{C}$3. The corresponding tied Loop-Grow model retains the exponent improvement with the same stored parameter count as Vanilla, although it trails Untied-Grow by a constant factor.

Figure 2: Boundary conditioning and model growth increase the scaling exponent, whereas untying primarily shifts the scaling curve by a constant.

The decomposition is informative. Untied-2 is consistently better than Loop-2, but its advantage is approximately constant: the multiplier is $\mathcal{C}$4 at $\mathcal{C}$5 FLOPs and $\mathcal{C}$6 at $\mathcal{C}$7 FLOPs. Similarly, Untied-Grow exceeds Loop-Grow by approximately $\mathcal{C}$8 at the lower budget and $\mathcal{C}$9 at the higher budget. Thus, weight sharing costs a nearly scale-independent compute factor, while growth determines the scale-dependent improvement.

Additional controls show that executed depth alone is insufficient. Deep Vanilla, which has a more favorable width-to-depth ratio but no boundary operator, yields an approximately flat $e$0 advantage. Operator ablations indicate that normalization and input re-injection both matter, including re-injection before the coda. Fixing the prelude and coda while scaling only the core also degrades the asymptotic advantage, supporting proportional scaling of the three architectural regions.

The fitted FineWeb exponents quantify these differences. Untied-Grow has an exponent of approximately $e$1, compared with $e$2 for Vanilla. On FineWeb-Edu, the corresponding values are approximately $e$3 and $e$4. The absolute differences are small, but because the compute multiplier depends exponentially on the exponent difference in log–log scaling, they produce increasingly large advantages at high compute.

Extrapolation and downstream evaluation

The authors test extrapolation by training an Untied-Grow model at approximately eight times the largest compute used to fit the scaling law. A $e$5B-parameter model, which begins training with approximately $e$6B parameters before growth, lands close to the predicted FineWeb-Edu loss curve. Its measured CORE accuracy is $e$7, compared with an estimated GPT-3 13B reference of $e$8. The model uses $e$9 FLOPs, compared with an estimated $h$0 FLOPs for GPT-3 13B.

Figure 3: An Untied-Grow run outside the fitting range follows the extrapolated loss and downstream scaling curves.

The resulting roughly $h$1 comparison is explicitly not a controlled head-to-head experiment: the systems use different training data, architectures, and evaluation pipelines. It should therefore be interpreted as an indicative comparison rather than a causal estimate of architectural efficiency.

Within the paper’s controlled FineWeb-Edu comparison, Untied-Grow requires approximately $h$2 less compute than Vanilla at the $h$3B model’s budget. The fitted curves project a $h$4 multiplier at $h$5 FLOPs. The downstream forecast uses task-level answer NLL scaling, shared NLL-to-accuracy sigmoids, and a calibration from a filtered 17-task score to full CORE. The held-out model’s predicted CORE score is $h$6, close to its observed $h$7.

These downstream projections remain conditional on extrapolating small-scale taskwise laws and calibration curves. Nonetheless, the agreement of the held-out model with both loss and downstream predictions supports the claim that the architectural exponent difference is not confined to FineWeb validation loss.

Why growth and boundary conditioning help

The paper interprets the results through computational depth rather than executed depth. In a pre-norm transformer, residual-stream magnitude can grow with depth, causing later updates to become relatively small. A network may therefore execute many blocks without all of them materially changing the predictive distribution. The boundary operator addresses this failure mode by normalizing the state before each pass and re-injecting the prelude representation.

The authors measure a proxy for usable depth using KL effective depth. They decode intermediate residual states with the logit lens and identify the point after the KL peak at which the decoded distribution comes within two nats of the final output. At $h$8 FLOPs, Untied-2 reaches KL effective depth 24, compared with 20 for Deep Vanilla at comparable executed depth. Untied-Grow reaches approximately 36, compared with 24 for Untied-2.

Figure 4: Boundary conditioning and growth increase KL effective depth; tied recurrence is particularly effective when parameters, rather than data, are the limiting resource.

The interpretation is that the boundary operator increases the fraction of executed blocks that remain computationally active, while growth avoids paying for maximum depth before the optimization trajectory requires it. The growth sweeps support this account: transitions that occur too early or too late are inferior, and grown models prefer higher token allocations than fixed-depth models. The latter result is consistent with allocating more training to a smaller initial network before activating additional depth.

The KL measure is only a proxy. It can miss inactive blocks in the middle of a network, and proximity to the final distribution does not imply that later blocks make no meaningful contribution. Consequently, the effective-depth analysis supports the proposed mechanism but does not establish it as the unique explanation for the scaling exponent changes.

Recurrence in the data-constrained regime

The paper distinguishes fresh-data scaling from repeated-data scaling. On fresh data, the compute-optimal fixed recurrence is between one and two passes, and additional recurrence is not preferred when model size and data are jointly optimized. Under a fixed pool of 100M unique FineWeb tokens repeated for ten epochs, the result changes substantially.

With increasing compute, the optimal loop count rises from approximately 1.4 to 6.7. Every fixed-loop curve eventually overfits, but higher recurrence delays the increase in validation loss and reaches a lower minimum. At the largest reported comparison, scaling the loop count matches the best model-size-scaled Operator-1 model with tuned weight decay using $h$9 less compute.

Figure 5: Under ten-epoch training on 100M unique tokens, optimal recurrence increases with compute and outperforms parameter scaling as a strategy for controlling overfitting.

The mechanism differs from the single-epoch result. Tied looping increases executed depth without increasing stored parameter count, whereas increasing model size introduces additional parameters that can overfit the repeated corpus. The optimal weight decay rises substantially with stored depth but is nearly invariant to loop count. Weight-decay tuning on top of loop scaling improves loss by at most approximately 0.02, indicating that recurrence provides a comparatively robust regularization mechanism.

The controls are consistent with this interpretation. Untied recurrence, which adds the same executed depth while increasing stored parameters, does not outperform tied Operator-1 under repetition. The preference for recurrence is strongest at low weight decay, where model-size overfitting is most severe, but it persists after weight decay is tuned. Thus, the data-constrained result is not simply an artifact of an untuned regularization coefficient.

This distinction yields a clear regime dependence:

Training regime	Preferred scaling intervention	Main benefit
Fresh data	Model growth, especially untied growth	Improved scaling exponent
Fresh data with parameter constraints	Tied model growth	Exponent improvement at lower storage
Repeated data	Increase tied recurrence	Regularization and greater usable depth
Repeated data with untied recurrence	Less effective	Additional stored parameters exacerbate overfitting

Training prescriptions and methodological significance

The experimental recipe is more demanding than the architectural description alone suggests. The authors find that compute-optimal tokens per stored parameter are approximately 5 for Vanilla, 6 for Operator-1 and fixed recurrence, and 7–8 for growth variants. Growth typically begins with two passes and transitions to four, with the transition occurring during the latter part of training. The optimal growth fraction is broad enough that a fixed schedule is generally adequate; replacing fitted growth timing with a constant fraction changes loss by only approximately $K$0 to $K$1 in the reported ablation.

The global learning rate must decrease with stored model size. Under a constant recipe, Vanilla exhibits a particularly strong learning-rate drift, approximately proportional to $K$2. Applying architecture-specific learning-rate scaling reduces the need for further hyperparameter scaling. Adding output-multiplier or weight-decay scaling changes the Vanilla matched-loss compute multiplier by no more than approximately 3% relative to learning-rate scaling alone.

These findings qualify the exponent claims in two ways. First, the measured exponent is a property of an architecture–optimization–allocation system, not of the computation graph in isolation. Second, insufficient tuning can make a genuine exponent improvement appear to be only a constant improvement. This is especially relevant because the paper’s main conclusion depends on comparing slopes over a relatively limited range of approximately two orders of magnitude in compute.

Limitations and open questions

The main empirical ladders are small by contemporary pretraining standards, use a fixed context length and batch size, and rely primarily on FineWeb and FineWeb-Edu. The scaling exponents are fitted over approximately $K$3–$K$4 FLOPs, with larger-scale conclusions depending on extrapolation. Although the held-out $K$5B run supports the extrapolation, a single out-of-range point cannot establish the asymptotic validity of the fitted laws.

The downstream comparison to GPT-3 is explicitly confounded by different data, architectures, and evaluation pipelines. The CORE forecasts additionally depend on taskwise scaling fits, sigmoid calibration, task filtering, and extrapolation beyond the measured domain. The paper reports these dependencies, but the downstream compute multipliers should not be interpreted as a controlled estimate of GPT-3 replacement cost.

The computational-depth explanation is also not fully identified. KL effective depth correlates with the architectural interventions, but it is a diagnostic proxy rather than a causal variable. The experiments do not determine whether the observed exponent changes arise primarily from residual normalization, input re-injection, optimization conditioning, staged parameter activation, implicit regularization, or interactions among these factors. The role of Muon is similarly unresolved: the paper finds that its advantage over Adam can shrink with coupled width–depth scaling, indicating an interaction between optimizer and architecture that is not characterized systematically.

Finally, the study concentrates on a two-to-four-pass growth schedule and a fixed prelude–core–coda allocation. Whether multi-stage growth, adaptive growth triggers, growth of width or context length, or growth in sparse expert count yields additional exponent improvements remains an open empirical question specific to this framework.

Conclusion

“How Model Growth, Recursion, and Boundary Operators Influence Scaling Exponents” argues that pretraining scaling exponents are not invariant to architectural design. In controlled compute-optimal ladders, boundary normalization with input re-injection and, more strongly, model growth produce compute multipliers that increase with scale. Untying improves the constant, while tied growth preserves the exponent gain with lower parameter storage. Under repeated data, the preferred strategy reverses: increasing tied recurrence becomes more efficient than scaling stored parameters because it adds usable depth without proportionally increasing overfitting capacity (2609.19107).

The paper’s strongest contribution is methodological as much as empirical: architectural claims about scaling must be evaluated with architecture-specific hyperparameter transfer, compute-optimal token allocation, explicit exponent estimates, and extrapolation tests. The remaining question is whether the observed computational-depth mechanism generalizes beyond this prelude–core–coda family, optimizer, data distribution, and measured compute range.