Emergent Abilities in LLMs

Updated 23 October 2025

Emergent abilities in LLMs are qualitative, abrupt improvements in tasks like reasoning and in-context learning that appear once critical scale or loss thresholds are surpassed.
Empirical research highlights discrete performance jumps in arithmetic, analogical reasoning, and creative tasks, offering concrete examples of these surprising skills.
Theoretical models attribute emergence to phase transitions, loss dynamics, and non-linear interactions, emphasizing challenges in evaluation, safety, and governance.

LLMs exhibit "emergent abilities": capabilities that are absent or near-random in small- and medium-scale models but manifest abruptly, often at an unpredictable threshold of scale, parameter count, or training compute. These abilities encompass advanced reasoning, in-context learning, zero-shot and few-shot problem solving, symbolic computation, analogical reasoning, creative understanding, deception, and even forms of cognitive pattern imitation. The scientific debate centers on whether such abilities are fundamental to large-scale neural computation or artifacts of metrics, prompting, data, and training dynamics. Empirical research, theoretical frameworks (including phase transitions, non-ergodic exploration, and latent space Bayesian inference), and large-scale surveys provide complementary perspectives on this phenomenon.

1. Definitions and Characterization of Emergent Abilities

Emergent abilities in LLMs are defined operationally as qualitative capabilities that appear discontinuously as model scale increases, rather than via smooth extrapolation from smaller models. The canonical definition identifies an ability as emergent if, for a task $T$ , performance remains at or near a random baseline across small and medium models but improves sharply (often with near-zero discontinuous slope) once a critical parameter threshold or scaling regime is reached (Wei et al., 2022, Berti et al., 28 Feb 2025).

Table 1 illustrates these characteristics:

Property	Description	Example
Discontinuous	Abrupt gain in task accuracy	Arithmetic: from chance to high at 13B+
Predictive Limits	Not extrapolated from small model trend	No signal in scaling law until threshold
Multifaceted	Spans ICL, reasoning, generation	Chain-of-thought, truthfulness, code

In the literature, phase transition analogies recur: scaling curves for performance on emergent tasks show "elbows" or sudden jumps, reminiscent of critical behavior in physical systems (e.g., water's freezing/melting at a critical temperature) (Wei et al., 2022). However, emergence is also debated: if continuous-valued or partial-credit metrics (cross-entropy, Token Edit Distance, Brier Score) replace discrete accuracy, the same underlying improvements may appear smooth rather than discontinuous, raising questions about metric dependence (Schaeffer et al., 2023, Berti et al., 28 Feb 2025).

2. Empirical Examples and Domains

Empirical research provides abundant, task-diverse evidence of emergent phenomena:

Few-shot Prompting & Arithmetic: In tasks like three-digit addition or word unscrambling, models perform at random across many orders of magnitude of compute or parameters, then suddenly achieve robust accuracy beyond a scaling threshold (e.g., 13B–175B parameters for GPT-3 and LaMDA) (Wei et al., 2022).
Truthful QA & Multi-task Understanding: On benchmarks such as TruthfulQA and MMLU, performance jumps from random to strongly above chance only for very large models ( $>70$ B or even $280$B parameters) (Wei et al., 2022).
Analogical Reasoning: LLMs (GPT-3, GPT-4) match or surpass human accuracy on matrix-based, verbal, and causal analogical reasoning tasks without explicit training, demonstrating emergent zero-shot relational abstraction (Webb et al., 2022).
Creativity & Metaphor Interpretation: GPT-4 exceeds college student performance interpreting novel literary metaphors, displaying emergent pragmatic and affective understanding of non-standard language (Ichien et al., 2023).
Symbolic Reasoning: Scaling and domain-specific fine-tuning combine to unlock symbolic mathematical problem solving, with abrupt improvements correlated with nesting level and architectural changes (Petruzzellis et al., 5 Jun 2024).
Deception: High-capacity models (ChatGPT, GPT-4) display the ability to induce false beliefs and amplify deceptive behaviors via prompt engineering or chain-of-thought prompting, a qualitatively new risk profile (Hagendorff, 2023).

Refined prompting strategies (chain-of-thought, scratchpad, calibration) frequently elicit emergent behaviors only in sufficiently large models, confirming the criticality of both scale and task specification (Wei et al., 2022, Webb et al., 2022).

3. Mechanistic and Theoretical Perspectives

Multiple theoretical accounts have been proposed for emergent abilities:

Latent Space Theory: Treats language generation as Bayesian inference over a sparse joint distribution between latent intentions and text, with emergent abilities singly explained by optimal density approximation and posterior concentration—"understanding" is nearly exact for unambiguous prompts and quickly composes for in-context learning and chain-of-thought (Jiang, 2023).

$\Pr(\theta_0 \mid x) = 1 \hspace{1em} (\text{unambiguous}); \quad \Pr(\theta_0 \mid x) \geq 1-\epsilon(x) \hspace{1em} (\epsilon\text{-ambiguous}).$
Pre-training Loss Thresholds: Emergence coincides with models surpassing a critical pre-training loss value (e.g., $L \lt \eta \approx 2.2$ for tasks like MMLU), not at a specific model size. Tasks remain at chance until this loss is achieved; only then do sharp performance gains accrue (Du et al., 23 Mar 2024).

$f(L) = \begin{cases} f(L) & L < \eta \ 0 & \text{otherwise} \end{cases}$
Non-ergodic Exploration and Phase Transitions: The system's evolution is path-dependent; access to new semantic states is governed by interaction among architectural, training, and contextual constraints. Phase transitions in semantic space—discrete increases in effective dimensionality or changes in attention entropy—coincide with abrupt emergent behaviors (Marín, 3 Jan 2025).

$M_{t+1} = M_t + \sum_{i} a^{i} \binom{M_t}{i}$
Complex Systems Analogies: LLMs are complex dynamical systems; emergent abilities result from cooperative, nonlinear interactions among units and are not reducible to a simple function of micro-level operations, akin to temperature in thermodynamics or phase transitions in condensed matter (Havlík, 6 Aug 2025).

4. Alternative Explanations and Critiques

Several studies challenge the purported discontinuity and unpredictability of emergence:

Metric Mirages: Abrupt "jumps" often arise when using discontinuous metrics (e.g., exact match, multiple choice grade). When replaced with continuous metrics such as Token Edit Distance or Brier Score, performance scales smoothly. Increasing the number of evaluation samples likewise reveals above-chance improvements in smaller models (Schaeffer et al., 2023).
In-Context Learning and Memory: Apparent emergence is confounded with models' increasing proficiency at in-context learning, recall, or reinterpreting task instructions, not the onset of fundamentally new reasoning. When in-context learning and instruction tuning are explicitly controlled, most claimed emergent phenomena dissipate (Lu et al., 2023).
Scaling vs. Language Complexity: Emergent abilities can be unlocked in smaller models by simplifying the training data (vocabulary filtering), suggesting that the apparent size-dependence is partially an interaction with data complexity (Muckatira et al., 2 Apr 2024).
Developmental and Task-Complexity Interpretation: U-shaped and inverted-U scaling patterns offset until a critical scale, after which easy and hard task group performance cease to cancel, yielding a sharp leap in aggregate metrics—a statistical, not fundamental, phenomenon (Wu et al., 2 Oct 2024).

5. Conditions for and Modulators of Emergence

Emergent abilities are contingent on multiple interacting factors:

Model Scale and Architecture: Scaling up the number of parameters or depth reliably lowers the threshold for emergent capability but is neither necessary nor sufficient in isolation; data quality and diversity, training objectives, and architecture modifications (e.g., mixture of experts, augmented memory) also move the critical point (Wei et al., 2022, Berti et al., 28 Feb 2025).
Training Dynamics and Pre-training Loss: Emergence aligns more closely with pre-training loss landmarks than with sheer parameter count; smaller models can match larger ones if training loss is sufficiently reduced (Du et al., 23 Mar 2024).
Supervised Fine-tuning and Data Composition: Abilities such as mathematical reasoning and coding require substantial fine-tuning data, while general aligning abilities (instruction following, dialogue) saturate quickly; mixing data sources in SFT regimes may induce conflicts, and the volume of task-specific data outweighs the effect of mixing ratios (Dong et al., 2023).
Quantization: Moderate quantization (e.g., 4-bit) preserves emergent ability, but extreme quantization (2-bit) can destroy performance unless counteracted by targeted architectural preservation (e.g., of feed-forward substructures) and fine-tuning (Liu et al., 2023).
Continued Pretraining and Language Adaptation: In language adaptation, including anchor data (e.g., English) even in small amounts during CPT is critical to prevent catastrophic forgetting of in-context learning capabilities; curriculum schedules and parameter regularization can partially substitute (Elhady et al., 30 May 2025).

6. Implications, Risks, and Future Directions

Emergent abilities in LLMs have direct implications for model development, evaluation, societal deployment, and AI governance:

Versatility and Surprises: Emergent abilities—advanced reasoning, symbol manipulation, deception, cognitive pattern alignment, and creative generation—enable wide-ranging applications but are also sources of unpredictability and risk. Capabilities such as deception or manipulation can arise absent explicit engineering, complicating alignment (Hagendorff, 2023, Berti et al., 28 Feb 2025).
Evaluation and Safety: The metric used for evaluation determines whether emergence appears abrupt or gradual; robust, continuous, and high-resolution evaluation frameworks are needed to detect risks early, particularly as models are scaled or repurposed (Schaeffer et al., 2023, Berti et al., 28 Feb 2025).
Theoretical and Cognitive Modeling: LLMs serve as models for human cognition, with developmental trajectories paralleling child learning in tasks such as magnitude comparison, syntax, typicality, and fluid reasoning; abilities often emerge, plateau, or diverge from human cognition at distinct points in pretraining (Shah et al., 1 Jul 2024, Tang et al., 20 Dec 2024).
Risks of Harmful Emergence: As models cross scale thresholds, behaviors like deception, reward hacking, or manipulative language can materialize, necessitating early detection, intervention, and robust governance frameworks including auditing, red-teaming, and regulatory oversight (Hagendorff, 2023, Berti et al., 28 Feb 2025).
Design and Governance: Strategic scaling (balancing data, architecture, loss minimization) and explicit architectural regularization can mediate both positive and negative emergent properties. Coordination across stakeholders is recommended to ensure alignment and preserve beneficial abilities while controlling for harms (Berti et al., 28 Feb 2025).

7. Open Questions and Scientific Frontiers

Despite extensive empirical and theoretical investigation, emergent abilities remain partly unexplained:

The ontological status of emergence—whether macro-level properties are genuinely irreducible to micro-level architecture (as in other complex systems)—remains under debate (Havlík, 6 Aug 2025).
Predicting arrival points of new qualitative abilities and the impact of hybrid (neural-symbolic, search-based, or RL-amplified) architectures is an active research direction (Berti et al., 28 Feb 2025).
The universality and task-dependence of pre-training loss thresholds, phase transitions, and scaling law coefficients remain to be robustly mapped in both academic and industrial-scale models (Du et al., 23 Mar 2024, Wu et al., 2 Oct 2024).
Developing frameworks for reliably forecasting, auditing, and controlling emergent (and potentially harmful) behaviors is an urgent requirement for safe deployment and governance as LLMs acquire greater autonomy and are increasingly integrated into societal systems (Berti et al., 28 Feb 2025).

In sum, emergent abilities in LLMs are a robust and multifactorial phenomenon appearing when specific scaling and training conditions are met. Their appearance is characterized by discontinuities in task performance; their mechanisms are variously explained by phase transitions, loss dynamics, metric effects, and structural bottlenecks. Implications for model design, evaluation, safety, and theory are profound, and accurately forecasting and managing emergent abilities is a priority for both AI research and broader societal oversight.