A Limit Theory of Foundation Models: A Mathematical Approach to Understanding Emergent Intelligence and Scaling Laws

Abstract: Emergent intelligence have played a major role in the modern AI development. While existing studies primarily rely on empirical observations to characterize this phenomenon, a rigorous theoretical framework remains underexplored. This study attempts to develop a mathematical approach to formalize emergent intelligence from the perspective of limit theory. Specifically, we introduce a performance function E(N, P, K), dependent on data size N, model size P and training steps K, to quantify intelligence behavior. We posit that intelligence emerges as a transition from finite to effectively infinite knowledge, and thus recast emergent intelligence as existence of the limit $\lim_{N,P,K \to \infty} \mathcal{E}(N,P,K)$, with emergent abilities corresponding to the limiting behavior. This limit theory helps reveal that emergent intelligence originates from the existence of a parameter-limit architecture (referred to as the limit architecture), and that emergent intelligence rationally corresponds to the learning behavior of this limit system. By introducing tools from nonlinear Lipschitz operator theory, we prove that the necessary and sufficient conditions for existence of the limit architecture. Furthermore, we derive the scaling law of foundation models by leveraging tools of Lipschitz operator and covering number. Theoretical results show that: 1) emergent intelligence is governed by three key factors-training steps, data size and the model architecture, where the properties of basic blocks play a crucial role in constructing foundation models; 2) the critical condition Lip(T)=1 for emergent intelligence provides theoretical support for existing findings. 3) emergent intelligence is determined by an infinite-dimensional system, yet can be effectively realized in practice through a finite-dimensional architecture. Our empirical results corroborate these theoretical findings.

• The paper introduces a limit-theoretic formalism that rigorously defines emergent intelligence via infinite-limit behavior of performance functions.
• It establishes necessary and sufficient conditions for stable model scaling by leveraging Lipschitz constants and operator contraction properties.
• The framework decomposes generalization error into weight, architecture, and sample errors, aligning theory with empirical model behavior.

Mathematical Limit Theory of Foundation Models: Emergent Intelligence and Scaling Laws

Introduction

The paper "A Limit Theory of Foundation Models: A Mathematical Approach to Understanding Emergent Intelligence and Scaling Laws" (2604.24037) proposes a rigorous mathematical framework for analyzing emergent intelligence and scaling laws in foundation models. The authors address the theoretical deficit in the current literature, where almost all knowledge about emergence and scaling has been empirically driven, by developing a limit-theoretic formalism utilizing nonlinear operator theory. Central elements of the framework include the notion of "limit architecture", the introduction of the Lip (Lipschitz) constant as a critical analytical tool, and a formal decomposition of generalization error into discrete, quantifiable contributions. The results provide necessary and sufficient conditions for both the existence of emergent intelligence and the emergence of predictable scaling behavior, and directly connect these phenomena to the properties of the model’s constitutive architectural blocks.

Formalization of Emergent Intelligence as a Limit

The paper introduces a three-dimensional performance function $\mathcal{E}(N, P, K)$, parameterized by training data size $N$, model size $P$, and number of training iterations $K$. Intelligence emergence is rigorously defined as the existence of the limit

$\lim_{N, P, K \to \infty} \mathcal{E}(N,P,K)$

where emergent abilities correspond to non-trivial limiting behaviors not observed in finite systems. This abstraction directly maps to knowledge acquisition and model/compute scaling in practical deep learning.

Performance is tightly linked to excess risk, and importantly, the authors highlight that for discontinuous or discrete performance metrics (e.g., accuracy), abrupt transitions and unpredictable behaviors—reminiscent of empirical emergence—naturally arise in this limit-theoretic framework. This gives mathematical substance to previously empirical observations of phase transitions and abrupt thresholds in model capabilities.

Limit Architecture and Operator-Theoretic Foundations

To analyze the infinite model size regime, the foundation model is recast as the composition of an effectively infinite sequence of architectural building blocks, or "basic blocks" $T_i$. The limit architecture, denoted as $f^* = \left(\prod_{i=1}^\infty T_i\right) f_0$, encodes the infinite-depth, potentially infinite-width model as a nonlinear operator product.

The existence of a stable limit architecture is shown to depend crucially on the sequence $\{T_i\}$ possessing particular operator-theoretic properties. Specifically, the key analytical device is the Lip constant, $\mathrm{Lip}(T)$, of a nonlinear operator $T$. This constant generalizes the spectral radius to nonlinear operators and admits several equivalent forms, including

$N$0

where $N$1 is the usual Lipschitz constant.

The main theoretical result is that necessary and sufficient conditions for the existence of a limit architecture are:

• For some $N$2, all $N$3: $N$4
• There exists a projection operator $N$5 such that $N$6 with $N$7

This corresponds, in practical model design, to having eventual contractivity (or non-expansiveness) and a "condensing property" wherein the sequence of blocks approximates a common projection. Notably, the critical value $N$8 aligns with empirical findings on criticality in deep learning [vock2025critical; cai2025learning].

Error Decomposition and Scaling Law Formalism

The authors decompose the total generalization error (performance deficit from the limit) into weight error, architectural (model) error, and sample error. Each admits a precise mathematical upper bound:

• Weight error: controlled by optimization and learning dynamics, converging exponentially fast in the strongly convex regime.
• Architecture error: controlled by how closely a finite model approximates the limit architecture; shown to decay exponentially in depth/size when $N$9.
• Sample error: determined by the usual statistical complexity of the hypothesis class, decaying as a power law in $P$0 under standard covering-number arguments.

The overall scaling law is then characterized as

$P$1

where $P$2 depends on optimization particulars (e.g., step size, smoothness).

It is particularly emphasized that for Transformer-like and other deep architectures, the analytical determination of $P$3 for key blocks (e.g., self-attention, LayerNorm, residuals, MLPs) can be explicitly performed, connecting theoretical predictions to concrete architecture design choices.

Empirical Assessment: Model Instability and Lip Constant

The framework is empirically verified by comparing GPT-1 (post-LayerNorm) and GPT-2 (pre-LayerNorm) models on the OpenWebText dataset. GPT-1, whose architecture does not respect the $P$4 condition, exhibits instability at increased depths, with divergence and NaN/Inf training behavior as model size increases, while GPT-2 remains stable across all tested depths.

Figure 2: Comparison of the training loss evolution of GPT-1 and GPT-2 models on OpenWebText under nanoGPT, showing GPT-1 divergence with deeper models.

Figure 1: Comparison of the Lip constant evolution of GPT-1 and GPT-2 under the same benchmark, highlighting stable Lip growth in GPT-2 and instability in GPT-1.

This illustrates the predictive power of the Lip constant as a diagnostic and design tool: only architectures with blocks satisfying $P$5 can be reliably scaled to depth and display convergence to stable—hence potentially emergent—behavior.

Condensing Property of Modern Foundation Models

The authors further validate the necessary "condensing" property for emergence by examining a range of state-of-the-art open-source models, including dense (Llama 3.1 8B/70B, Qwen-2-7B/72B) and sparse (Deepseek-MoE-16B) architectures. Layerwise measurements reveal that, beyond a certain depth, the action of additional layers becomes increasingly similar to the identity (projection operator), supporting the theoretical condition that the blocks condense to a common operator in the deep limit. This evidence aligns with empirical findings regarding shared subspaces in weight space across diverse initializations and domains [kaushik2025universal].

Practical and Theoretical Implications

The results provide verifiable criteria for safe, effective scaling of architectures. The limit-theoretic approach:

• Explains why only certain architectural motifs (e.g., LayerNorm placement, block design) allow scaling without instability or degeneracy.
• Clarifies the architectural basis for phase transitions and sudden jumps in model capabilities, encoding them as mathematical phase changes in the limit $P$6.
• Predicts when empirical scaling laws (power law for data, exponential for size/steps) hold, and pinpoints failures to respect these laws as failures to meet the critical operator bounds.
• Suggests new avenues for architecture search, emphasizing block design with provable non-expansiveness or contractivity, and invites work on more complex composition mechanisms (residual, MoE, mixture-based architectures) within this analytical framework.

Conclusion

This paper constructs a comprehensive mathematical foundation for analyzing emergence and scaling in deep architectures by leveraging nonlinear operator limit theory. Emergent intelligence is recast as the existence and properties of an infinite composition limit, with scaling laws emerging naturally from precise error decompositions. The pivotal role of the Lip constant as both a theoretical and practical diagnostic is established, and extensive empirical evidence is marshaled to confirm theoretical predictions for both architectural stability and the universality of the condensing property. The limit theory developed here provides both a blueprint for future foundation model design and a bridge between empirical discoveries and first-principles mathematical understanding.