Can Scale Save Us From Plasticity Loss in Large Language Models?

Abstract: The loss of plasticity - the ability of a network to learn new information after having already learned older information - is a fundamental challenge in creating artificial neural networks capable of continual learning. Although this phenomenon has been known for decades, it has mostly been studied in older, relatively small architectures and rarely in natural-language domains. To determine whether loss of plasticity remains a problem in the modern transformer-based LLM paradigm, we study plasticity loss in GPT-style Transformer models trained on a multilingual continual learning problem. Consistent with prior work, we find evidence of plasticity loss across models ranging from 5M to 314M non-embedding parameters, as measured by deterioration on a held-out Vietnamese probing task. We further find that the onset of plasticity loss follows a predictable scaling law, growing sublinearly with model size. These results suggest that larger models may delay the measurable effects of plasticity loss, but that increasing parameter count alone is likely to be insufficient to completely prevent it. We also find evidence of plasticity loss under stationary multilingual training, challenging the view that the phenomenon is exclusive to continual learning with abrupt task changes. Overall, our results suggest that even large Transformer LLMs trained on natural-language will eventually lose the ability to efficiently adapt to new data after sufficiently long training, in both continual and stationary settings.

• The paper demonstrates that plasticity loss—a decline in adaptive learning—is inevitable in Transformer models, with a power-law delay as model size increases.
• Empirical measurements using a Vietnamese probing task reveal that even large models (e.g., 314M parameters) eventually experience adaptation degradation after prolonged training cycles.
• Structural correlates such as increased dormant units and pathological attention head behavior suggest intrinsic shifts in representation dynamics impacting sustained model performance.

Plasticity Loss at Scale in Transformer LLMs

Overview of Objectives and Contributions

The paper "Can Scale Save Us From Plasticity Loss in LLMs?" (2606.24752) addresses a fundamental limitation in deep neural network optimization: plasticity loss, defined as a progressive decline in a network's ability to efficiently learn new information after extended training. Distinct from catastrophic forgetting—which involves the loss of previously acquired knowledge—plasticity loss is characterized by diminishing capacity to adapt, regardless of memory retention. The primary research question is whether increasing parameter count in modern Transformer-based LLMs attenuates or eliminates this effect, particularly in realistic multilingual continual and stationary learning regimes. The study spans models from 5M to 314M non-embedding parameters, employing a novel evaluation protocol based on Vietnamese probing after repeated cycles of nonstationary language tasks, and provides strong empirical evidence for persistent, scale-dependent plasticity loss.

Experimental Setup and Continual Learning Protocol

The authors operationalize plasticity loss via degradation in adaptation performance to a held-out Vietnamese task, periodically probed after each cycle of multilingual language modeling. Eight languages (English, written Chinese, French, Japanese, Spanish, German, Portuguese, Russian) are presented sequentially, where each language-modeling task consumes 5B tokens; after each cycle, a 5B-token Vietnamese probe assesses plasticity. This design ensures that transfer effects are minimal and the loss curve reflects intrinsic plasticity. Models use pre-layer-normalized causal Transformer architectures with fixed aspect ratios and attention head dimensions across scales, allowing attribution of effects to parameter count rather than architectural confounds.

Figure 1: Multilingual continual learning problem. At the end of each cycle, a Vietnamese probe quantifies plasticity.

Empirical Evidence of Plasticity Loss and Scaling Law

Analysis of adaptation (validation-loss AUC) as a function of cycle count reveals that all models eventually exhibit increasing AUC, signifying reduced plasticity. Crucially, larger models experience delayed onset of plasticity loss, but not exemption from it. For example, the 5M model deteriorates after 1–2 cycles, while the 314M model withstands 30–40 cycles before showing measurable loss.

Figure 2: Percentage change in AUC on the probing task relative to the first cycle demonstrates plasticity loss across scales, with larger models maintaining plasticity longer.

To further quantify this dependence, inter-cycle probing (extended at various cycle points) enables identification of the minimum in adaptation performance. Fitting the onset as a function of model size yields a power-law relation: $T = 1.3 \times 10^{-5} P^{0.8269}$, indicating that the threshold for plasticity loss (in task instances) grows sublinearly with parameter count. This directly contradicts any claim that increasing model size linearly extends adaptive lifetime.

Figure 3: Log-log scaling law for the onset of plasticity loss; sublinear exponent demonstrates diminishing returns from mere scaling.

Stationary Training and Universality of Plasticity Loss

Plasticity loss is not restricted to continual learning with abrupt task changes. Training under stationary, mixed-language data consistently results in an eventual increase in adaptation AUC (deterioration), confirming that plasticity loss is endemic to extended gradient descent, even without distribution shifts.

Figure 4: Plasticity loss emerges even under stationary multilingual training, substantiating its generality.

Structural Correlates of Plasticity Loss

The study interrogates intra-network correlates including parameter magnitude, MLP unit dormancy, and pathological attention head statistics. While average parameter magnitude and dormant units tend to accumulate with extended training, these metrics do not reliably predict the precise onset or severity of plasticity loss. Nevertheless, they reveal fundamental shifts in representation dynamics: in both 53M and 106M models, single layers accrue near-total dormancy, implicating loss of effective dimensionality.

Figure 5: At plasticity-loss onset, dormant unit count and parameter magnitude increase, but do not perfectly track adaptation collapse.

Figure 6: Dormant unit pathology is concentrated in specific Transformer layers, rather than distributed.

Attention heads also exhibit collapse (low entropy) or laziness (high entropy/uniformity), with layer-wide pathologies emerging independently of plasticity loss metrics. These observations suggest systemic loss of functional capacity in both MLP and attention modules.

Figure 7: Pathological attention head behaviors (collapse/laziness) arise over cycles, sometimes involving entire layers.

Architectural Choices and Evaluation Protocol

All models implement absolute positional encodings, tied embeddings, pre-norm stabilization, and GeLU nonlinearities. Learning rates are tuned and interpolated to assure comparable optimization regimes across sizes. The Vietnamese probing task, decoupled from the training stream, provides a robust measure of adaptive potential, neutralizing confounds from transfer and memorization.

Figure 8: The specific pre-norm causal Transformer architecture employed for all models.

Data-Driven Fitting and Predictive Models

Extended probing and moving average smoothing enable precise determination of the plasticity-loss onset. Multiple predictive functional forms are tested; the log-log (power law) achieves highest empirical fit and generalization, underscoring multiplicative—rather than additive or exponential—growth in plasticity-loss tolerance with scaling.

Figure 9: Extended probing reveals that plasticity loss onset is delayed but not eliminated as model size increases.

Figure 10: Comparative fitting of predictive models for plasticity loss onset.

Implications, Practical Consequences, and Future Directions

The demonstration that scale merely delays—not prevents—plasticity loss has profound implications for LLM deployment, continual learning benchmarks, and model re-training. Current practices, including weight decay and activation management, address symptoms but do not mechanistically resolve plasticity limitations. Future research directions include targeted regularization, activity-based reinitialization (e.g., Continual Backprop, ReDo), architectural innovations that resist dormancy and attention pathologies, and the pursuit of mechanisms that sustain effective dimensionality post-pretraining.

Figure 11: Structural metrics (weight magnitude, dormancy, attention pathologies) for additional models exhibit similar non-monotonicity.

Figure 12: Correlates under stationary training mirror those observed in continual regimes, highlighting structural features across scenarios.

Conclusion

Empirical results decisively demonstrate that even large-scale, modern Transformer LLMs suffer from sublinearly mitigated but inevitable plasticity loss, irrespective of continued pretraining or stationary data streams. This paper's findings invalidate the hypothesis that mere parameter count ensures perpetual adaptive capacity, and instead suggest plasticity loss is a universal property intrinsic to deep neural network optimization. The study also establishes robust, scalable methodologies for quantifying plasticity, and motivates the development of principled interventions beyond naïve scaling. These results inform both theoretical modeling of LLM learning dynamics and practical strategies for sustained model adaptation in deployment environments.