---
title: Limit Architecture in Foundation Models
url: https://www.emergentmind.com/topics/limit-architecture
type: topic
---

# Limit Architecture in Foundation Models

Limit architecture denotes the **parameter–limit architecture** that arises when a foundation model is analyzed in the joint limit of training sample size, model size, and training steps. In the formulation introduced in "A Limit Theory of Foundation Models: A Mathematical Approach to Understanding Emergent Intelligence and Scaling Laws" [2604.24037], the central object is an infinite-depth map
$$
f^*(x)=\Bigl(\prod_{i=1}^{\infty}T_i\Bigr)f_0(x),
$$
obtained as the pointwise limit of compositions of basic blocks. Within this framework, emergent intelligence is defined by the existence of the three-parameter limit
$$
\mathcal{E}(\infty,\infty,\infty)\equiv \lim_{N,P,K\to\infty}\mathcal{E}(N,P,K)<\infty,
$$
where performance depends jointly on data size $N$, parameter count $P$, and training steps $K$. The concept is therefore both architectural and asymptotic: it seeks to identify what an infinite-depth, effectively infinite-parameter system computes, and to relate that object to emergent abilities, scaling laws, and practical finite-network design [2604.24037].

## 1. Formal definition in the limit-theory framework

The paper measures performance through a three-parameter function
$$
\mathcal{E}(N,P,K)=g\bigl(E(N,P,K)\bigr),
$$
where $N$ is the training sample size, $P$ is the total number of parameters, $K$ is the number of training steps, and
$$
E(N,P,K)=R(f_{W_K})-\inf_W R(f_W)
$$
is the excess risk. The map $g:\mathbb{R}_{\ge 0}\to\mathbb{R}_{\ge 0}$ is any right-continuous, nonincreasing transformation; the paper gives $g(x)=e^{-x}$ as an example for turning loss into accuracy [2604.24037].

Under this definition, a model exhibits **emergent intelligence** precisely when the full limit $\mathcal{E}(\infty,\infty,\infty)$ exists. The paper further states that the emergence of new abilities corresponds to qualitative changes in this limit that cannot be foreseen by extrapolating any two-dimensional slice of $\mathcal{E}$. This places the notion of emergence in a genuinely three-axis setting rather than in a one-factor or two-factor scaling curve.

The significance of the definition is methodological. Instead of treating emergence as an empirical irregularity in benchmark curves, the framework defines it through the existence and structure of a limiting object. This suggests that the correct explanatory target is not merely a large but finite network, but the architecture that finite systems approximate along the axes of data, parameters, and optimization.

## 2. Construction from basic blocks

The limit architecture is constructed from a finite architecture written as a composition of $L$ basic blocks:
$$
f_W(x)=T_L\circ T_{L-1}\circ\cdots\circ T_1\circ f_0(x),
$$
where each $T_i$ is a nonlinear Lipschitz operator
$$
T_i:\mathbb{R}^{n_i}\to\mathbb{R}^{n_{i+1}},
\qquad
W=\{w_1,\dots,w_L\}.
$$
As $L\to\infty$—and therefore $P\to\infty$ if each block has fixed finite width—the architecture induces an infinite-depth or parameter-limit map
$$
f_L(x)=T_L\bigl(f_{L-1}(x)\bigr)\xrightarrow[L\to\infty]{}f^*(x),
$$
with
$$
f^*(x)=\Bigl(\prod_{i=1}^{\infty}T_i\Bigr)f_0(x).
$$
The paper calls $f^*$ the **limit architecture** [2604.24037].

Its existence means that the infinite-depth map is well-defined in $L^2$ or another suitable Banach space, so that one can write
$$
\mathcal{E}(\infty,\infty,\infty)=g\bigl(R(f^*)-\inf R(f)\bigr).
$$
This point is essential: the theory is not merely about sending depth to infinity formally, but about determining when the infinite composition converges to a mathematically meaningful operator.

The construction places strong emphasis on the role of **basic blocks**. The paper explicitly states that emergent intelligence is governed by three key factors—training steps, data size, and the model architecture—and that the properties of basic blocks play a crucial role in constructing foundation models. In this sense, the limit architecture is not an additional module layered on top of an existing network; it is the asymptotic object generated by repeated block composition.

## 3. Lipschitz criteria and existence theorems

The decisive quantity is the **Lip constant** of a nonlinear operator $T$. The paper defines
$$
L(T)=\sup_{x\neq y}\frac{\|T(x)-T(y)\|}{\|x-y\|},
$$
and then
$$
\mathrm{Lip}(T)=\lim_{n\to\infty}\bigl[L(T^n)\bigr]^{1/n}.
$$
When $T$ is $C^1$, Proposition 3.1 gives the equivalent expression
$$
\mathrm{Lip}(T)=\sup_{x\in D}\rho\bigl(T'(x)\bigr),
$$
with $\rho(\cdot)$ the spectral radius of the Jacobian. Proposition 3.3 further states that for the Lipschitz dual $T_L^*$,
$$
\rho(T_L^*)=\mathrm{Lip}(T).
$$
A spectral decomposition of $T_L^*$ then yields the condensing condition and isolates the critical case $\mathrm{Lip}(T)=1$ [2604.24037].

The one-block existence theorem is stated as Theorem 4.4. Let $T$ be a single Lipschitz self-map on a closed convex domain $D\subset X$, with $X$ a uniformly convex Banach space. Then
$$
\lim_{L\to\infty}T^Lf_0
$$
exists, and equals a fixed point, **if and only if**
$$
\mathrm{Lip}(T)\le 1.
$$

For varying blocks, Theorem 4.6 gives a more general condition: if eventually all blocks satisfy $\mathrm{Lip}(T_i)\le 1$ and condense to a common projection $T$ at summable speed,
$$
\|T_i-T\|\le \epsilon_i,
\qquad
\sum_i \epsilon_i<\infty,
$$
then the infinite product $\prod_{i=1}^\infty T_i$ converges. Conversely, if the product converges, one must have $\mathrm{Lip}(T_i)\le 1$ for large $i$ and the blocks condense to a single projection [2604.24037].

The theory distinguishes three regimes:

- $\mathrm{Lip}(T)<1$: $T$ is strictly contractive and has a unique fixed point.
- $\mathrm{Lip}(T)>1$: the iterates diverge.
- $\mathrm{Lip}(T)=1$: there is a nontrivial family of fixed points.

Accordingly, the paper identifies the condition $\mathrm{Lip}(T)=1$ as both necessary and sufficient for the **existence of a nontrivial limit architecture**, and therefore for emergent intelligence. This is a more precise statement than the weaker condition $\mathrm{Lip}(T)\le 1$, which guarantees existence but not nontriviality.

## 4. Scaling laws from the limit architecture

Once $f^*$ exists, the paper decomposes the deviation from the limiting performance into optimization, model-size, and statistical terms:
$$
\mathcal{E}(N,P,K)-\mathcal{E}(\infty,\infty,\infty)
=
\bigl[\mathcal{E}(N,P,K)-\mathcal{E}(N,P,\infty)\bigr]
+
\bigl[\mathcal{E}(N,P,\infty)-\mathcal{E}(N,\infty,\infty)\bigr]
+
\bigl[\mathcal{E}(N,\infty,\infty)-\mathcal{E}(\infty,\infty,\infty)\bigr].
$$
The three terms are interpreted respectively as weight or optimization error, model or architecture error, and statistical or sample error [2604.24037].

Under $G$-smoothness and $\mu$-strong convexity of the loss, Proposition 5.1 gives
$$
\bigl|\mathcal{E}(N,P,K)-\mathcal{E}(N,P,\infty)\bigr|
\lesssim
\beta^K,
\qquad
\beta=1-\frac{\mu}{G},
$$
which is an exponential law in $K$.

When $\mathrm{Lip}(T)<1$, Theorem 5.2 gives
$$
\bigl|\mathcal{E}(N,P,\infty)-\mathcal{E}(N,\infty,\infty)\bigr|
\lesssim
(\mathrm{Lip}(T))^P,
$$
which is exponential in $P$. The paper states that if one cycles through a finite palette of operators each with $\mathrm{Lip}<1$, the same bound holds with $\max_i \mathrm{Lip}(T_i)$.

By standard covering-number arguments, Theorem 5.3 gives
$$
\bigl|\mathcal{E}(N,\infty,\infty)-\mathcal{E}(\infty,\infty,\infty)\bigr|
=
\bigl|R(f^*)-\hat R(f^*)\bigr|
\lesssim
N^{-1/2},
$$
which is a power law in $N$.

A concise summary is:

| Error term | Interpretation | Rate |
|---|---|---|
| $\mathcal{E}(N,P,K)-\mathcal{E}(N,P,\infty)$ | Optimization / weight error | $\beta^K$ |
| $\mathcal{E}(N,P,\infty)-\mathcal{E}(N,\infty,\infty)$ | Architecture / model-size error | $(\mathrm{Lip}(T))^P$ |
| $\mathcal{E}(N,\infty,\infty)-\mathcal{E}(\infty,\infty,\infty)$ | Statistical / sample error | $N^{-1/2}$ |

Combining the three terms yields the rough scaling law
$$
\bigl|\mathcal{E}(N,P,K)-\mathcal{E}(\infty,\infty,\infty)\bigr|
\lesssim
\beta^K+(\mathrm{Lip}(T))^P+N^{-1/2}.
$$
This decomposition is one of the framework’s main contributions because it makes the architectural term explicit rather than absorbing it into a generic parameter-count heuristic.

## 5. Finite approximation and design implications

Although $f^*$ is infinite-depth and infinite-parameter, the theory is intended for finite networks. The paper states that whenever $\mathrm{Lip}(T)<1$—or more generally $\mathrm{Lip}(T_i)\le 1$ and the blocks condense—the tail
$$
\prod_{i=1}^{\infty}T_i-\prod_{i=1}^{P}T_i
$$
decays exponentially fast in $P$. It follows that a truncated architecture of depth $P$ approximates $f^*$ within $O(e^{-cP})$ [2604.24037].

This directly supports the paper’s claim that emergent intelligence is determined by an infinite-dimensional system, yet can be effectively realized in practice through a finite-dimensional architecture. A plausible implication is that the infinite object serves as an analytic reference model, while the deployed network is a controlled truncation along the model-size axis.

The same section of the paper extracts concrete design guidance. The necessity of $\mathrm{Lip}(T)\le 1$, and in the critical case exactly $=1$, is presented as a quantitative design rule for each building block—basic MLP, attention, normalization, and related components—so that the infinite limit converges. The paper specifically states that this explains why **pre-LayerNorm** Transformers, which can be shown to have bounded Lip, are stable in deep regimes, whereas **post-LayerNorm** variants can blow up [2604.24037].

Another implication concerns architecture search across scales. The paper states that having an analytic description of $f^*$—for instance in kernel or mean-field limits—opens the door to principled architecture search and hyperparameter tuning that are consistent across scales. In this reading, limit architecture is not only a theoretical endpoint but also a design prior.

## 6. Emergent intelligence, criticality, and related architecture-limit theories

Within the limit-theory paper, emergent intelligence is not treated as a vague increase in benchmark score. It is tied to the existence and behavior of the limit system, with the critical condition $\mathrm{Lip}(T)=1$ identified as the mathematical heart of emergence. The paper also reports empirical results corroborating these theoretical findings [2604.24037].

A nearby but distinct line of work is "The Deterministic Horizon: Impossibility Results as Design Specifications for Trustworthy AI Systems" [2605.23024]. That thesis proposes an architecture-only accuracy ceiling for decoder-only transformers, defined by an effective reasoning depth $\delta=m\cdot L$ and a **Deterministic Horizon** $d^*$, the largest depth at which per-step error remains below $1/2$. Its architectural scaling law is
$$
d^*=O\!\bigl(L\cdot \phi(d)\bigr),
\qquad
\phi(d)\in[\sqrt{\ln d},\,\ln d],
$$
with empirical fit
$$
d^*\approx \hat c\,\ln L\,\sqrt{\ln d},
\qquad
\hat c=2.74\pm 0.33.
$$
Across twelve architectures, the reported measured values lie between nineteen and thirty-one, and the probability of solving all steps correctly is bounded by a super-exponential decay beyond the horizon [2605.23024].

The two frameworks are not identical. The limit architecture theory studies the existence of an infinite-depth limiting system and the scaling of deviations from it, whereas the Deterministic Horizon studies an architecture-dependent ceiling on sequential reasoning depth. Nevertheless, a plausible connection is that both assign a primary role to architectural structure rather than to training alone. In both cases, architecture is treated as a source of mathematically characterizable limits: convergence and emergence in one framework, and reasoning-depth ceilings in the other.

For this reason, limit architecture occupies a distinctive place in current theory. It recasts emergent intelligence as the behavior of a limit system, ties that system to nonlinear Lipschitz operator theory, and yields explicit exponents for optimization, model-size, and sample scaling. Its central claim is not that infinite models are practically deployable, but that finite models become intelligible when viewed as approximations to a well-defined infinite-depth architecture.

Source: https://www.emergentmind.com/topics/limit-architecture