Limit Architecture in Foundation Models

• Limit architecture is a framework defining an infinite-depth model as the pointwise limit of repeated basic blocks, capturing emergent intelligence.
• It decomposes performance into optimization, model-size, and statistical errors using precise scaling laws based on Lipschitz constants.
• The theory guides finite network design by approximating the ideal limit architecture, ensuring stability with critical conditions like Lip(T)=1.

Limit architecture denotes the parameter–limit architecture that arises when a foundation model is analyzed in the joint limit of training sample size, model size, and training steps. In the formulation introduced in "A Limit Theory of Foundation Models: A Mathematical Approach to Understanding Emergent Intelligence and Scaling Laws" (Shu et al., 27 Apr 2026), the central object is an infinite-depth map

$f^*(x)=\Bigl(\prod_{i=1}^{\infty}T_i\Bigr)f_0(x),$

obtained as the pointwise limit of compositions of basic blocks. Within this framework, emergent intelligence is defined by the existence of the three-parameter limit

$$\mathcal{E}(\infty,\infty,\infty)\equiv \lim_{N,P,K\to\infty}\mathcal{E}(N,P,K)<\infty,$$

where performance depends jointly on data size $N$, parameter count $P$, and training steps $K$. The concept is therefore both architectural and asymptotic: it seeks to identify what an infinite-depth, effectively infinite-parameter system computes, and to relate that object to emergent abilities, scaling laws, and practical finite-network design (Shu et al., 27 Apr 2026).

1. Formal definition in the limit-theory framework

The paper measures performance through a three-parameter function

$\mathcal{E}(N,P,K)=g\bigl(E(N,P,K)\bigr),$

where $N$ is the training sample size, $P$ is the total number of parameters, $K$ is the number of training steps, and

$E(N,P,K)=R(f_{W_K})-\inf_W R(f_W)$

is the excess risk. The map $$\mathcal{E}(\infty,\infty,\infty)\equiv \lim_{N,P,K\to\infty}\mathcal{E}(N,P,K)<\infty,$$0 is any right-continuous, nonincreasing transformation; the paper gives $$\mathcal{E}(\infty,\infty,\infty)\equiv \lim_{N,P,K\to\infty}\mathcal{E}(N,P,K)<\infty,$$1 as an example for turning loss into accuracy (Shu et al., 27 Apr 2026).

Under this definition, a model exhibits emergent intelligence precisely when the full limit $$\mathcal{E}(\infty,\infty,\infty)\equiv \lim_{N,P,K\to\infty}\mathcal{E}(N,P,K)<\infty,$$2 exists. The paper further states that the emergence of new abilities corresponds to qualitative changes in this limit that cannot be foreseen by extrapolating any two-dimensional slice of $$\mathcal{E}(\infty,\infty,\infty)\equiv \lim_{N,P,K\to\infty}\mathcal{E}(N,P,K)<\infty,$$3. This places the notion of emergence in a genuinely three-axis setting rather than in a one-factor or two-factor scaling curve.

The significance of the definition is methodological. Instead of treating emergence as an empirical irregularity in benchmark curves, the framework defines it through the existence and structure of a limiting object. This suggests that the correct explanatory target is not merely a large but finite network, but the architecture that finite systems approximate along the axes of data, parameters, and optimization.

2. Construction from basic blocks

The limit architecture is constructed from a finite architecture written as a composition of $$\mathcal{E}(\infty,\infty,\infty)\equiv \lim_{N,P,K\to\infty}\mathcal{E}(N,P,K)<\infty,$$4 basic blocks:

$$\mathcal{E}(\infty,\infty,\infty)\equiv \lim_{N,P,K\to\infty}\mathcal{E}(N,P,K)<\infty,$$5

where each $$\mathcal{E}(\infty,\infty,\infty)\equiv \lim_{N,P,K\to\infty}\mathcal{E}(N,P,K)<\infty,$$6 is a nonlinear Lipschitz operator

$$\mathcal{E}(\infty,\infty,\infty)\equiv \lim_{N,P,K\to\infty}\mathcal{E}(N,P,K)<\infty,$$7

As $$\mathcal{E}(\infty,\infty,\infty)\equiv \lim_{N,P,K\to\infty}\mathcal{E}(N,P,K)<\infty,$$8—and therefore $$\mathcal{E}(\infty,\infty,\infty)\equiv \lim_{N,P,K\to\infty}\mathcal{E}(N,P,K)<\infty,$$9 if each block has fixed finite width—the architecture induces an infinite-depth or parameter-limit map

$N$0

with

$N$1

The paper calls $N$2 the limit architecture (Shu et al., 27 Apr 2026).

Its existence means that the infinite-depth map is well-defined in $N$3 or another suitable Banach space, so that one can write

$N$4

This point is essential: the theory is not merely about sending depth to infinity formally, but about determining when the infinite composition converges to a mathematically meaningful operator.

The construction places strong emphasis on the role of basic blocks. The paper explicitly states that emergent intelligence is governed by three key factors—training steps, data size, and the model architecture—and that the properties of basic blocks play a crucial role in constructing foundation models. In this sense, the limit architecture is not an additional module layered on top of an existing network; it is the asymptotic object generated by repeated block composition.

3. Lipschitz criteria and existence theorems

The decisive quantity is the Lip constant of a nonlinear operator $N$5. The paper defines

$N$6

and then

$N$7

When $N$8 is $N$9, Proposition 3.1 gives the equivalent expression

$P$0

with $P$1 the spectral radius of the Jacobian. Proposition 3.3 further states that for the Lipschitz dual $P$2,

$P$3

A spectral decomposition of $P$4 then yields the condensing condition and isolates the critical case $P$5 (Shu et al., 27 Apr 2026).

The one-block existence theorem is stated as Theorem 4.4. Let $P$6 be a single Lipschitz self-map on a closed convex domain $P$7, with $P$8 a uniformly convex Banach space. Then

$P$9

exists, and equals a fixed point, if and only if

$K$0

For varying blocks, Theorem 4.6 gives a more general condition: if eventually all blocks satisfy $K$1 and condense to a common projection $K$2 at summable speed,

$K$3

then the infinite product $K$4 converges. Conversely, if the product converges, one must have $K$5 for large $K$6 and the blocks condense to a single projection (Shu et al., 27 Apr 2026).

The theory distinguishes three regimes:

• $K$7: $K$8 is strictly contractive and has a unique fixed point.
• $K$9: the iterates diverge.
• $\mathcal{E}(N,P,K)=g\bigl(E(N,P,K)\bigr),$0: there is a nontrivial family of fixed points.

Accordingly, the paper identifies the condition $\mathcal{E}(N,P,K)=g\bigl(E(N,P,K)\bigr),$1 as both necessary and sufficient for the existence of a nontrivial limit architecture, and therefore for emergent intelligence. This is a more precise statement than the weaker condition $\mathcal{E}(N,P,K)=g\bigl(E(N,P,K)\bigr),$2, which guarantees existence but not nontriviality.

4. Scaling laws from the limit architecture

Once $\mathcal{E}(N,P,K)=g\bigl(E(N,P,K)\bigr),$3 exists, the paper decomposes the deviation from the limiting performance into optimization, model-size, and statistical terms:

$\mathcal{E}(N,P,K)=g\bigl(E(N,P,K)\bigr),$4

The three terms are interpreted respectively as weight or optimization error, model or architecture error, and statistical or sample error (Shu et al., 27 Apr 2026).

Under $\mathcal{E}(N,P,K)=g\bigl(E(N,P,K)\bigr),$5-smoothness and $\mathcal{E}(N,P,K)=g\bigl(E(N,P,K)\bigr),$6-strong convexity of the loss, Proposition 5.1 gives

$\mathcal{E}(N,P,K)=g\bigl(E(N,P,K)\bigr),$7

which is an exponential law in $\mathcal{E}(N,P,K)=g\bigl(E(N,P,K)\bigr),$8.

When $\mathcal{E}(N,P,K)=g\bigl(E(N,P,K)\bigr),$9, Theorem 5.2 gives

$N$0

which is exponential in $N$1. The paper states that if one cycles through a finite palette of operators each with $N$2, the same bound holds with $N$3.

By standard covering-number arguments, Theorem 5.3 gives

$N$4

which is a power law in $N$5.

A concise summary is:

Error term	Interpretation	Rate
$N$6	Optimization / weight error	$N$7
$N$8	Architecture / model-size error	$N$9
$P$0	Statistical / sample error	$P$1

Combining the three terms yields the rough scaling law

$P$2

This decomposition is one of the framework’s main contributions because it makes the architectural term explicit rather than absorbing it into a generic parameter-count heuristic.

5. Finite approximation and design implications

Although $P$3 is infinite-depth and infinite-parameter, the theory is intended for finite networks. The paper states that whenever $P$4—or more generally $P$5 and the blocks condense—the tail

$P$6

decays exponentially fast in $P$7. It follows that a truncated architecture of depth $P$8 approximates $P$9 within $K$0 (Shu et al., 27 Apr 2026).

This directly supports the paper’s claim that emergent intelligence is determined by an infinite-dimensional system, yet can be effectively realized in practice through a finite-dimensional architecture. A plausible implication is that the infinite object serves as an analytic reference model, while the deployed network is a controlled truncation along the model-size axis.

The same section of the paper extracts concrete design guidance. The necessity of $K$1, and in the critical case exactly $K$2, is presented as a quantitative design rule for each building block—basic MLP, attention, normalization, and related components—so that the infinite limit converges. The paper specifically states that this explains why pre-LayerNorm Transformers, which can be shown to have bounded Lip, are stable in deep regimes, whereas post-LayerNorm variants can blow up (Shu et al., 27 Apr 2026).

Another implication concerns architecture search across scales. The paper states that having an analytic description of $K$3—for instance in kernel or mean-field limits—opens the door to principled architecture search and hyperparameter tuning that are consistent across scales. In this reading, limit architecture is not only a theoretical endpoint but also a design prior.

6. Emergent intelligence, criticality, and related architecture-limit theories

Within the limit-theory paper, emergent intelligence is not treated as a vague increase in benchmark score. It is tied to the existence and behavior of the limit system, with the critical condition $K$4 identified as the mathematical heart of emergence. The paper also reports empirical results corroborating these theoretical findings (Shu et al., 27 Apr 2026).

A nearby but distinct line of work is "The Deterministic Horizon: Impossibility Results as Design Specifications for Trustworthy AI Systems" (Guo, 21 May 2026). That thesis proposes an architecture-only accuracy ceiling for decoder-only transformers, defined by an effective reasoning depth $K$5 and a Deterministic Horizon $K$6, the largest depth at which per-step error remains below $K$7. Its architectural scaling law is

$K$8

with empirical fit

$K$9

Across twelve architectures, the reported measured values lie between nineteen and thirty-one, and the probability of solving all steps correctly is bounded by a super-exponential decay beyond the horizon (Guo, 21 May 2026).

The two frameworks are not identical. The limit architecture theory studies the existence of an infinite-depth limiting system and the scaling of deviations from it, whereas the Deterministic Horizon studies an architecture-dependent ceiling on sequential reasoning depth. Nevertheless, a plausible connection is that both assign a primary role to architectural structure rather than to training alone. In both cases, architecture is treated as a source of mathematically characterizable limits: convergence and emergence in one framework, and reasoning-depth ceilings in the other.

For this reason, limit architecture occupies a distinctive place in current theory. It recasts emergent intelligence as the behavior of a limit system, ties that system to nonlinear Lipschitz operator theory, and yields explicit exponents for optimization, model-size, and sample scaling. Its central claim is not that infinite models are practically deployable, but that finite models become intelligible when viewed as approximations to a well-defined infinite-depth architecture.