Parcae: Scaling Laws For Stable Looped Language Models

Abstract: Traditional fixed-depth architectures scale quality by increasing training FLOPs, typically through increased parameterization, at the expense of a higher memory footprint, or data. A potential alternative is looped architectures, which instead increase FLOPs by sending activations through a block of layers in a loop. While promising, existing recipes for training looped architectures can be unstable, suffering from residual explosion and loss spikes. We address these challenges by recasting looping as a nonlinear time-variant dynamical system over the residual stream. Via a linear approximation to this system, we find that instability occurs in existing looped architectures as a result of large spectral norms in their injection parameters. To address these instability issues, we propose Parcae, a novel stable, looped architecture that constrains the spectral norm of the injection parameters via discretization of a negative diagonal parameterization. As a result, Parcae achieves up to 6.3% lower validation perplexity over prior large-scale looped models. Using our stable looped architecture, we investigate the scaling properties of looping as a medium to improve quality by increasing FLOPs in training and test-time. For training, we derive predictable power laws to scale FLOPs while keeping parameter count fixed. Our initial scaling laws suggest that looping and data should be increased in tandem, given a fixed FLOP budget. At test-time, we find that Parcae can use looping to scale compute, following a predictable, saturating exponential decay. When scaled up to 1.3B parameters, we find that Parcae improves CORE and Core-Extended quality by 2.99 and 1.18 points when compared to strong Transformer baselines under a fixed parameter and data budget, achieving a relative quality of up to 87.5% a Transformer twice the size.

• The paper proposes spectral norm constraints on the injection matrix to stabilize looped architectures and prevent recurrent state explosions.
• The study demonstrates that per-sequence depth sampling and prelude normalization significantly improve training stability and lower validation perplexity.
• The research establishes unified scaling laws, linking loop-depth and compute budgets to enable efficient, parameter-scaled language model performance.

Parcae: Scaling Laws for Stable Looped LLMs

Motivation and Background

The study introduces Parcae, a looped architecture achieving stable and scalable performance in transformer-based LLMs while controlling the memory footprint. Traditional transformers scale quality primarily through parameter growth and increased training data, resulting in higher memory and deployment costs. Looped architectures, such as recurrent depth models (RDMs), reuse blocks via recursion, scaling compute independently from parameter count. Historically, looped models suffered from training instability due to residual state explosion and loss spikes. These instabilities have impeded both practical deployments and the formalization of predictable scaling laws for looped models. Leveraging a discrete-time dynamical systems view, the paper rigorously analyzes causes of instability and proposes architectural and algorithmic remedies.

Dynamical Systems Framework for Looping

Using a linear time-invariant (LTI) framework, Parcae recasts the recurrent update of looped architectures as a dynamical system over the residual stream. Classic control theory demonstrates that unconstrained spectral norms in the injection matrix $A$ result in unstable models via exploding hidden state trajectories during recurrent computation. Existing looped models typically fail to rigorously enforce the necessary spectral constraints, leading to marginal stability or divergence. Empirical studies show that models with spectral radius $\rho(A)\geq 1$ exhibit recurrent state explosion and loss spikes.

Figure 1: Constrained residual normalization enables model convergence, while unconstrained Pre-Norm looped models diverge, evidenced by exploding hidden embedding norms.

Parcae Architecture and Training Stabilization

Parcae enforces model stability by constraining the injection matrix $A$ to a negative diagonal parameterization—guaranteeing all eigenvalues are negative and, thus, a spectral radius strictly less than 1. Discretization via Zero Order Hold and explicit normalization of the prelude output minimize both forward and backward computational instabilities. Additionally, Parcae modifies the stochastic depth sampling paradigm by introducing per-sequence depth sampling within micro-batches. This procedural change ensures a faithful expectation over the recurrence distribution, mitigating loss spikes and sharpening fixed-point learning dynamics.

Figure 2: Per-sequence depth sampling eliminates loss spikes observed with per-microbatch sampling, confirming improved training stability.

Empirical Performance and Scaling Laws

Quality Evaluation

Parcae consistently outperforms parameter- and data-matched RDMs and fixed-depth Transformers. When scaled to 1.3B parameters and trained on 100B tokens, Parcae achieves up to 6.3% lower validation perplexity compared to large-scale looped baselines, and surpasses Transformers of equivalent parameter count by margins of 2.99 and 1.18 points on CORE and Core-Extended benchmarks, respectively.

Training Compute Scaling

Parcae enables compute-efficient scaling via increased looping at fixed parameter budgets, establishing loop-depth as an orthogonal axis to model size and data. IsoFLOP curves and optimal fit analysis reveal that optimal training requires simultaneous scaling of loop depth and data, following robust power law relationships.

Figure 3: Training compute efficiency emerges as looping and data increase together; parametric isoLoss contours illustrate optimal loop scaling at isoFLOP budgets.

Figure 4: Empirical fits show optimal recurrence depth and token budget follow predictable power laws across scales.

Looped architectures trained at optimal loop depth systematically outperform fixed-depth counterparts at equivalent FLOP budgets, exhibiting strict Pareto optimality in downstream quality.

Figure 5: Spectral norm constraint and input normalization stabilize the residual stream; looping augments computational scaling following a power law.

Test-Time Scaling and Unified Law

Test-time looping yields predictable performance improvements, governed by a saturating exponential decay law. Validation loss rapidly declines as recurrence increases, plateauing near the training depth—the irreducible loss floor imposed by the training law.

Figure 6: Validation loss versus test-time recurrence fits an exponential decay model, confirming predictable scaling behavior.

The unified scaling law merges FLOP-optimal training constraints and test-time exponential decay into a single parametric model, accurately predicting performance at unseen training budgets and test-time depths with less than 1.31% mean relative error.

Figure 7: Unified parametric fit predicts out-of-distribution validation loss as function of loop-depth and test-time recurrence.

Implementation Details and Ablations

Extensive ablations confirm that constraining $A$ is necessary for convergence at large loop depths, per-sequence sampling improves fixed-point behavior and downstream metrics, and prelude normalization is critical for preventing state explosion in large models. Selection of mean recurrence and mean backward steps affects capacity and generalization, with $\mu_{\text{rec}}$ and $\mu_{\text{bwd}}$ set at proportional values for stability and compute efficiency.

Figure 8: Loss spikes and state explosions are resolved via prelude normalization in large-scale Parcae runs.

Implications and Future Directions

The theoretical and empirical results establish looped recurrence as an independent dimension for scaling compute in LLMs. Parcae bridges the gap between parameter efficiency and model quality, enabling competitive downstream performance with reduced memory footprint. The scaling laws formalized allow for principled planning of training and inference budgets in deployment scenarios, notably for edge and resource-constrained environments.

Future work should explore looped unit placement, hierarchical/parallel architectures, full-rank parameterizations, and the optimal interplay between loop-depth, parameter, and data scaling as models reach higher compute regimes. Further research is warranted into techniques that maintain quality with reduced inference steps, as increasing loop-depth increases test-time latency.

Conclusion

Parcae leverages dynamical systems analysis to stabilize looped architectures and develop robust scaling laws governing both training and inference. With explicit spectral norm constraints and algorithmic refinements, Parcae delivers superior parameter-efficient quality and predictable scaling behavior. The architecture advances practical and theoretical understanding of looped transformer models, providing a foundation for future exploration in efficient compute scaling (2604.12946).