Emergent Capabilities in AI

• Emergent capabilities are unexpected, qualitative behaviors that manifest when AI models cross specific scale or complexity thresholds.
• Investigations using theoretical, mechanistic, and empirical methods reveal sudden improvements in tasks like multi-step reasoning and tool use.
• These phenomena have significant implications for model design and practical applications, influencing strategies in large language models and robotics.

Emergent capabilities in artificial intelligence refer to qualitatively novel behaviors or skills that arise unexpectedly at a particular threshold of model scale, loss, or context complexity, and cannot be reliably extrapolated from the performance of smaller or less capable models. These phenomena are especially prominent in LLMs, robotic foundation models, and energy-based policies, where scaling up parameters, optimizing pre-training loss, or introducing new architectural regimes has led to abrupt transitions in task competence, reasoning, and generalization. Theoretical, mechanistic, and empirical investigations have revealed a variety of frameworks for describing, predicting, and leveraging emergent behaviors across modalities.

1. Definition and Formal Characterization

An emergent capability is defined as a downstream behavior—such as accurate multi-step reasoning, advanced tool use, or robust cross-task generalization—that switches abruptly from random or near-baseline performance to well-above-chance once a model crosses a critical threshold in parameters, training data, compute, or pre-training loss. This discontinuity is formally expressed as:

$$P_{A}(s) \approx P_{\text{rand}} \quad \text{for } s < s_{0}$$

$$P_{A}(s) \gg P_{\text{rand}} \quad \text{for } s \geq s_{0}$$

where ( P_{