---
title: Phase Transitions in Large Language Models
url: https://www.emergentmind.com/topics/phase-transitions-in-large-language-models
type: topic
---

# Phase Transitions in Large Language Models

A phase transition in large language models (LLMs) is an abrupt, collective shift in their internal representations or external behaviors as a function of a control parameter—such as model size, data scale, temperature, or training progression. These transitions, analogous to critical phenomena in statistical mechanics, are central to understanding “emergent abilities” and structural reorganizations underlying the performance and alignment characteristics of contemporary LLMs. They may manifest as non-analytic changes in order parameters, singularities in macroscopic observables, or sharp shifts in performance and statistical properties, often invisible to conventional loss curves. Phase transitions have now been rigorously characterized and quantitatively diagnosed across a range of axes and model regimes.

## 1. Formal Definitions and Theoretical Foundations

Phase transitions in LLMs are mathematically characterized by abrupt, non-analytic changes in one or more order parameters as a control variable is tuned. In analogy to physical systems, there exist two principal classes:

- **Second-order (continuous) transitions:** These are marked by the divergence of a susceptibility (e.g., variance of an internal statistic), without discontinuity in the order parameter itself, and by power-law scaling near a critical point (e.g., temperature or model depth). Diverging susceptibility is exemplified by the static susceptibility $\chi(T)$ in [2406.05335], where the integrated correlation of part-of-speech (POS) sequences diverges at a critical temperature $T_c$.
- **First-order (discontinuous or higher-depth) transitions:** Here, an order parameter exhibits a finite jump at a critical value of the control variable, as in the sharp increase of logical reasoning error at a threshold complexity metric in [2601.02902].

Mathematical tools are adapted from classical and quantum statistical mechanics (O($N$) models, spin-glass theory, renormalization group flows), probability theory (Poisson and sub-Poisson statistics), and information theory (f-divergences, Fano factors). Across these frameworks, an “order parameter” is any model-internal or output-level statistic that changes non-trivially at the critical point, such as cross-channel coherence, overlap in spin-glass models, or task accuracy as a function of data complexity.

## 2. Critical Phenomena in Model Scaling, Training, and Data Regimes

### Model Size and Capacity Transitions

- **Sharp onsets of capability:** Experiments reveal that as model size crosses a critical threshold—with all other factors held fixed—discrete emergent skills (such as knowledge memorization, arithmetic, logical reasoning) appear suddenly, not smoothly [2505.18091, 2412.07386, 2303.13112].
- **Information-theoretic explanation:** Capacity allocation in finite models can be rigorously described by a knapsack framework, with phase transitions occurring as optimal allocation shifts discontinuously across datasets [2505.18091]. The memorized fraction $f(r,M)$ transitions sharply from zero to $\mathcal{O}(1)$ as either mixing ratio $r$ or model size $M$ surpasses a critical value, governed by scaling relations such as $r_{\text{thres}}\propto M^{-\beta}$.

### Training Progression and Data Size

- **Delayed generalization as phase transition:** Under grokking and classic data scaling, there is a critical data size $N_c$ for each model size such that the test accuracy remains at chance until $N_c$ is crossed, after which rapid generalization occurs [2401.10463]. For fixed architecture, as $N \uparrow N_c$, one observes an abrupt transition in test behavior, often with a characteristic delay (grokking time $T_{\text{grok}}$), especially for modular or synthetic tasks.
- **Three-regime dynamics:** The triple phase transition framework links brain-model alignment, probing accuracy, and downstream performance: (1) rapid alignment with instruction following, (2) detachment and associated task stagnation, (3) realignment with emergent downstream mastery [2502.20779].

## 3. Statistical and Structural Transition Metrics

### Output Distribution and Statistical Distances

- **System-agnostic diagnostics:** f-divergences (KL, JS, TV) and classifier-based “g-dissimilarities” robustly detect macroscopic reorganizations of the output distribution under parametric sweeps (temperature, prompt type, or epoch) [2405.17088, 2406.05335, 2508.20015]. 
- **Joint multi-metric analysis:** Poisson and sub-Poisson statistics on word/event counts, dispersion indices, and KL divergence from Poisson capture phase transitions in the formation of lexical coherence and error suppression [2305.05061, 2511.12768]. These transitions are evidenced by narrow temporal windows where Fano factors for correct and incorrect words cross unity, aligning with coherence emergence in hidden-state dynamics.

### Internal Representation and Hidden-State Structure

- **Coherence as order parameter:** Cross-channel correlation matrices ($C_{jk}$), auto-correlation manifolds, and the off-diagonal coherence serve as robust witnesses of phase transitions in internal wave dynamics [2305.05061].
- **Topological and spectral signatures:** The covariance spectrum of residual stream activations transitions from a random-matrix (Marchenko–Pastur) bulk (“liquid” phase) to a low-rank, spiked spectrum (“solid” phase), marked by a discontinuity in an analytically defined localization order parameter $\Omega(h)$ at critical layer depth $\gamma_c\approx 0.42$ [2601.19942]. This regime change corresponds to the appearance of Transient Class Objects (TCOs), stable deep-layer basins representing discrete semantic classes.

### Logical and Algorithmic Phase Transitions

- **Logical complexity collapse:** In symbolic reasoning tasks, LLMs maintain stable accuracy across increasing logical depth or complexity until a sharply-defined “critical” LoCM is reached, at which point performance collapses to random chance [2601.02902]. These transitions, termed “Logical Phase Transitions,” are not smooth, paralleling capacity thresholds in physical phase transitions.
- **Algorithmic instability:** Mechanistic studies reveal abrupt flips between distinct problem-solving circuits as task specification (e.g., number of digits in arithmetic) is varied, indicating algorithmic phase transitions. Fine-grained analysis with activation patching and subcircuit clustering confirms that models do not smoothly interpolate but instead select from a few discrete algorithmic regimes [2412.07386].

## 4. Physical Mappings, Universality, and Statistical Models

- **Spin-glass and O($N$) models:** Transformers can be rigorously mapped onto O($N$) statistical field theories or spin-glass Hamiltonians, with temperature, model size, or depth serving as control variables [2501.16241, 2505.02879]. Observable phase transitions include:
  - *Temperature-driven transitions*: At $T_c \approx 1.2$, the energy per token and specific heat undergo singularities, marking a transition from ordered (coherent) to disordered (noisy or creative) output.
  - *Parameter-size transitions*: At $P_c \approx 7 \times 10^9$, models acquire meta-awareness of incoherence, manifesting as a sign change in $\Delta E=E(T_c)-E(T\to\infty)$—a practical capability criterion.
- **Self-organized criticality of language:** Statistical analyses of LLM-generated outputs and natural text show that natural language operates at or near the critical point, characterized by power-law correlations, divergent susceptibility, and critical slowing down [2406.05335].

## 5. Behavioral, Alignment, and Safety Transitions

- **Fine-tuning phase transitions:** Targeted or broad distributional changes may be detected during alignment or misalignment fine-tuning using rigorous statistical distances over the output distributions and a suite of order parameters (alignment, style, confidence, completeness, etc.) [2508.20015]. Notably, the main behavioral transition can lag behind classical warnings such as gradient-norm peaks.
- **Emergent misalignment and order parameter decomposition:** Only a fraction of the total behavioral shift during a phase transition is attributable to any one property (e.g., alignment), with style and confidence often explaining greater fractions of the transition. The residual variation points to latent or as-yet-unmonitored axes of behavioral change.

## 6. Broader Implications and Practical Takeaways

- **Predictability and universality:** Many phase transitions in LLMs can be quantitatively predicted by power-law scaling relations (e.g., critical mixing ratio vs. model size, critical temperature), enabling targeted resource allocation and mixture design [2505.18091, 2501.16241].
- **Diagnostic and engineering tools:** Lightweight statistical probes—Poisson windowing, f-divergences, activation-patching, and order parameter tracking—afford early warnings for phase transitions, even where external benchmarks are insensitive [2511.12768, 2405.17088].
- **Control and intervention:** Regularization or curriculum schemes can be designed to delay, advance, or smooth phase transitions, potentially preserving desired “human-like” characteristics or extending the regime of stable logical/algorithmic generalization [2502.18802, 2601.02902].
- **Emergent ability engineering:** The non-ergodic, resource-constrained TAP framework formalizes the interplay of model, data, and context constraints in shaping the landscape of accessible capabilities; new emergent abilities arise precisely when combined constraints cross a critical threshold [2501.01638]. Finite-size scaling and RG-style analyses provide guidance for anticipating and controlling stepwise emergent behaviors [2601.19942].

| Transition Axis            | Order Parameter(s)                   | Key Observable(s)                                | arXiv IDs                 |
|----------------------------|--------------------------------------|--------------------------------------------------|---------------------------|
| Model size/capacity        | Memorized fraction, $\Delta E$, $m(\gamma)$ | Sharp accuracy jump, energy, spectral collapse    | 2505.18091, 2501.16241, 2601.19942 |
| Training/data scale        | Generalization accuracy, $T_{\text{grok}}$      | Onset of generalization (grokking), stagnation    | 2401.10463, 2502.20779    |
| Decoding temperature       | Energy, overlap, susceptibility      | Order/disorder, creativity, singularity in output | 2501.16241, 2505.02879, 2406.05335 |
| Output distribution        | f-divergence, dispersion indices     | Peaks in $D_g$, Poisson→sub-Poisson transitions   | 2405.17088, 2511.12768, 2305.05061 |
| Internal representation    | Coherence, localization $\Omega(h)$  | Cross-channel correlation, covariance spectrum    | 2305.05061, 2601.19942    |
| Logical/algorithmic depth  | LoCM, circuit stability metrics      | Performance collapse, circuit switching           | 2601.02902, 2412.07386    |
| Behavioral alignment       | Alignment, style, confidence OPs     | Distributional break, OP decomposition            | 2508.20015, 2502.18802    |

## 7. Limitations and Directions for Future Study

- **Finite-size effects:** Many phase transitions broaden or shift at larger model or data scales, necessitating careful finite-size scaling analyses to identify universal exponents and critical thresholds [2406.05335, 2601.19942].
- **Hidden transitions:** Standard loss and validation curves may conceal phase transition points; tailored diagnostics are necessary for reliable detection [2511.12768, 2305.05061].
- **Black-box and scaling challenges:** Output-only or sampled-model access limits the power of f-divergence-based diagnostics, especially in very large LLMs, and may necessitate advanced classifier-based estimators [2405.17088].
- **Generalization to other domains:** While criticality and phase-transition frameworks have been validated in text-based LLMs, extension to vision, multi-modal, and RL models is an open area of research [2502.18802].

Phase transitions in large language models provide a quantitative, predictive, and physically principled lens through which to study emergent abilities, internal reorganizations, and failure modes. The formalization of these phenomena brings interpretability, methodology, and new training and safety pathways to the frontier of language modeling.

Source: https://www.emergentmind.com/topics/phase-transitions-in-large-language-models