---
title: In-Context Neural Scaling Laws
url: https://www.emergentmind.com/topics/in-context-neural-scaling-laws
type: topic
---

# In-Context Neural Scaling Laws

In-context neural scaling laws are quantitative relations between inference-time adaptation performance and the resources that make such adaptation possible: the number of in-context examples, prompt length, usable context window, model depth and width, pretraining data, and sometimes training or test-time compute. The literature does not yet present a single universally accepted law. Instead, it offers several partially overlapping formalisms: Bayesian laws in which demonstrations update a posterior over latent tasks, solvable toy theories in which self-attention implements in-context optimization, empirical laws that jointly model downstream performance with training compute and provided context, and broader transformer theories that connect ICL emergence to depth, width, context length, and task structure [2410.16531].

## 1. Scope, variables, and problem formulations

The direct literature on in-context neural scaling laws uses different targets and different scaling variables. Some papers scale the probability of the next correct output as a function of the number of demonstrations. Others scale population risk in in-context regression as a function of depth, width, context length, and training time. Others model downstream task performance jointly in terms of training compute and prompt length. A more ambitious line states a transformer ICL test error bound that depends on model scale, training demonstrations, context length, and sequence length [2510.14919].

| Framework | Predicted quantity | Main variables |
|---|---|---|
| Bayesian ICL | expected next-example / next-token probability | \(n\), \(P_{X,m}\), \(p_m\), \(K\) |
| In-context regression | population risk \(\mathcal L\) | \(t\), \(N\), \(L\), \(P\) |
| Context-aware downstream scaling | aggregate task performance \(\mathcal P\) | \(C\), \(n_\text{pmt}\), \(n_\text{ctx}\) |
| Unified transformer ICL theory | ICL test error \(\epsilon\) | \(N\), \(D\), \(k\), \(n\) |

This heterogeneity is substantive rather than merely notational. In the Bayesian line, in-context learning is approximate posterior inference over a finite task family. In the in-context regression line, a deep linear self-attention model learns an in-context optimizer. In the context-aware empirical line, context is treated as an inference-time resource that interacts multiplicatively with training compute. In the unified transformer line, ICL emergence is tied to architectural scale and task hierarchy [2410.16531].

## 2. Bayesian laws for scaling with demonstrations

A direct probabilistic account models ICL as Bayesian inference over latent tasks \(T=\{T_1,\ldots,T_M\}\), with prior \(\rho\) and task-conditional likelihoods \(\delta\). Given a document \(D\), the posterior is
\[
p(T_m \mid D) = \frac{p(D\mid T_m)p(T_m)}{\sum_{m=1}^M p(D\mid T_m)p(T_m)},
\]
and the expected next-example / next-token probability under \(n\) in-context examples is
\[
\mathbb{E}_{o\sim X}[p(o \mid D)] = \frac{\sum_{m=1}^M (P_{X,m})^{Kn+1}\, p_m} {\sum_{m=1}^M (P_{X,m})^{Kn}\, p_m},
\]
where \(P_{X,m}\) is the expected probability of a sample from task \(X\) under task \(T_m\), \(p_m=p(T_m)\), and \(K\) is an ICL efficiency coefficient [2410.16531].

This law predicts saturation rather than indefinite improvement. As \(n\) grows, posterior mass concentrates on the task hypothesis best supported by the examples, so the curve approaches a task-conditional asymptote. That distinguishes it from unbounded power-law fits and makes prior-versus-evidence tradeoffs explicit. In this formalism, \(p_m\) controls the zero-shot or few-shot starting point, \(P_{X,m}\) controls curvature and asymptotes, and \(K\) controls how many effective Bayesian updates each nominal example contributes [2410.16531].

The framework is especially notable for controlled studies of many-shot jailbreaking. In synthetic GINC experiments, Bayesian variants matched or exceeded non-Bayesian baselines, and on extrapolation the scoring-wise Bayesian law achieved \(0.0467\) NRMSE compared with about \(0.0664\)–\(0.0668\) for the non-Bayesian baselines. In real-world instruction-tuned LLM experiments, Bayesian laws were competitive rather than uniformly dominant, with average NRMSE \(0.1603\) for Bayesian sampling-wise and \(0.1599\) for a logistic baseline. The substantive claim is therefore not only predictive accuracy, but interpretability: many-shot recovery of a suppressed behavior is modeled as likelihood evidence overcoming a prior disadvantage rather than as an opaque prompting effect [2410.16531].

## 3. Solvable in-context regression: depth, width, context, and time

A separate direct line studies in-context learning of linear regression in a deep linear self-attention model. A context contains \(P\) labeled examples and \(K\) masked evaluation points, and the context loss is
\[
\mathcal L(D)=\frac{1}{K}\sum_{\mu=P+1}^{P+K}(f_\mu-y_\mu)^2.
\]
Under alignment assumptions, the transformer reduces to a \(\Gamma\)-model in which the predictor acts like \(L\) steps of preconditioned gradient descent on the in-context regression problem. The core resource variables are depth \(L\), width \(N\), context length \(P\), batch size \(B\), and pretraining time \(t\), with compute
\[
C = t\, P^2 N^2 L.
\]
This makes width, depth, context, and training time distinct scaling axes rather than collapsing them into total parameter count [2510.01098].

The theory distinguishes three settings. In ISO, both covariates and tasks are isotropic; in FS, covariance structure is fixed across contexts; in RRS, covariances are randomly rotated across contexts. The difference is decisive. In ISO and FS, depth helps only when context is limited, because with large enough context a shallow model suffices or can memorize an effective preconditioner. In RRS, by contrast, covariance orientation changes across contexts, so the model cannot hard-code a universal anisotropic solver in its weights. Depth then remains useful even at infinite context length, because it is performing genuine iterative in-context computation rather than merely compensating for data scarcity [2510.01098].

In the RRS regime, the paper states a separable scaling law
\[
\mathcal L(t,N,L,P) \approx c_t\, t^{-\beta/(2+\beta)} + c_N\, N^{-\nu\beta} + c_L\, L^{-\beta} + c_P\, P^{-\nu\beta},
\]
together with a compute-optimal shape law
\[
L \propto N^\nu.
\]
The depth exponent depends only on the source exponent \(\beta\), whereas the width and context exponents depend on \(\nu\beta\). This is one of the clearest direct formulations of an in-context neural scaling law because the scaled task is itself ICL, the architectural variables include both width and depth, and the statistical variable \(P\) is the number of in-context examples [2510.01098].

## 4. Context-aware scaling and context-horizon laws

An empirical downstream line treats provided context as a first-class scaling variable. The proposed law for aggregate task performance is
\[
\begin{aligned}
\mathcal{P}(C, n_\text{pmt}, n_\text{ctx}) &= \Bigl[ 1 - \exp\Bigl(- A\, \Bigl(\frac{C}{C^{c}}\Bigr)^{\alpha} \Bigr) \Bigr] \\
&\times \Bigl[ 1 - \exp\Bigl(- B\, \Bigl(\frac{n_\text{pmt}}{n_\text{pmt}^c}\Bigr)^{\beta} \Bigr) \Bigr] \\
&\times \sigma\left(n_\text{pmt} - n_\text{ctx}\right),
\end{aligned}
\]
where the text describes the third factor as a penalty term for when \(n_\text{pmt} > n_\text{ctx}\). Here \(C\) is non-embedding training compute, \(n_\text{pmt}\) is prompt length, and \(n_\text{ctx}\) is the model’s context limit. The product structure encodes the claim that compute and context are complementary rather than additive, and the empirical results show low absolute prediction error on arithmetic reasoning, commonsense reasoning, and machine translation, together with reliable extrapolation to larger context [2510.14919].

This line is directly about downstream in-context performance rather than upstream language-model loss. It also makes task dependence explicit through fitted context exponents. Arithmetic reasoning yields \(\beta=0.56\), commonsense reasoning \(\beta=1.12\), and machine translation \(\beta=2.97\), which the paper interprets as different saturation regimes: arithmetic benefits from many demonstrations over a broader range, whereas commonsense reasoning and translation saturate quickly after the first few demonstrations [2510.14919].

A different but closely related theory derives scaling from language statistics rather than from prompt engineering. In the data-limited regime for autoregressive language models, the key quantities are pairwise token-correlation decay
\[
\|C(n)\|_{\mathrm{op}} \asymp n^{-\beta}
\]
and next-token conditional-entropy decay
\[
H_n - H_\infty \asymp n^{-\gamma}.
\]
These imply a data-dependent prediction horizon
\[
n^*(P)\asymp P^{1/(2\beta)},
\]
and, in the fast within-horizon learning regime,
\[
\mathcal{L}_{\mathrm{AR}}(P)-H_\infty \asymp P^{-\gamma/(2\beta)}.
\]
The same paper also predicts a scaling collapse for \(n\)-gram losses,
\[
\mathcal{L}_n(P)\asymp n^{-\gamma}\ell(P/n^{2\beta}).
\]
This is not a theory of few-shot ICL in the modern task-conditioning sense, but it is a direct theory of how usable context grows with data and how loss scales as longer effective context becomes exploitable [2602.07488].

## 5. Unified transformer theories of ICL emergence

A more ambitious transformer-specific framework states that ICL test error satisfies
\[
\epsilon(N, D, k, n) \leq C \left[\left(\frac{N_0}{N}\right)^{\alpha} + \left(\frac{D_0}{D}\right)^{\alpha} + \left(\frac{k_0}{k}\right)^{\gamma} + \left(\frac{n_0}{n}\right)^{\delta}\right],
\]
with
\[
\alpha = \frac{1}{2(h+1)}, \qquad \gamma = \frac{\beta}{2\beta + d_x}, \qquad \delta = \frac{1}{2}.
\]
Here \(h\) is the hierarchy depth of the task family, \(D\) is the number of training demonstrations, \(k\) is context length, and \(N=\Theta(Ld)\) is an effective model-size proxy. The framework further states that transformers can implement gradient-based meta-learning in their forward pass, with effective learning rate
\[
\eta_{\text{eff}} = \Theta(1/\sqrt{Ld}),
\]
and that ICL emergence follows a sigmoid-like transition with critical scale
\[
N_c = \Theta((k \cdot h)^{2(h+1)}).
\]
Within the same framework, the stated compute-allocation prescription is
\[
L^* \propto N^{2/3}, \qquad d^* \propto N^{1/3}.
\]
The central claim is therefore that model depth, width, context length, and training data participate in a common ICL scaling theory, and that the exponents are determined by task structure rather than being universal constants [2511.06232].

The mechanistic core of that theory is constructive. For function classes learnable by \(k\)-step gradient descent, the paper states that there exists a transformer with \(L=\Theta(k)\) and \(d=\Theta(\operatorname{poly}(1/\epsilon))\) such that forward-pass attention computes gradient-like updates and achieves error within \(\epsilon\) of the best \(k\)-step learner. This gives a direct bridge between architectural depth and the number of implicit in-context adaptation steps [2511.06232].

The same paper also reports synthetic-task evidence: measured model/data exponents close to the stated theory on linear regression, sparse linear regression, and decision trees; threshold-like emergence with critical scales that rise sharply with task hierarchy; and a strong depth advantage at fixed budget. At the same time, the paper details also note internal inconsistencies, including a mismatch between the stated \(\eta_{\text{eff}}=\Theta(1/\sqrt{Ld})\) and the supporting algebra, a tension between the corollary for depth-width scaling and the approximation term used elsewhere, and a discrepancy between the tabulated theoretical \(\alpha\) for \(h=2\) and the formula \(\alpha=1/(2(h+1))\). The framework is therefore significant as a synthesis, but not yet fully settled in its mathematical details [2511.06232].

## 6. Indirect foundations: duality, geometry, invariance, and data distribution

Several adjacent theories do not analyze ICL directly, but they supply reusable structures for thinking about in-context scaling. One field-theoretic random-feature regression model yields an exact planar result in which the optimal test loss scales as
\[
\langle \widehat L\rangle(\gamma^\star)\sim \left(\frac1N+\frac1T\right)^\alpha,
\]
with a corresponding optimal ridge law
\[
\gamma^\star \sim \left(\frac1N+\frac1T\right)^{\alpha-1},
\]
and an explicit \(N\leftrightarrow T\) duality between model size and sample count. In the noiseless case this duality forces identical scaling exponents with respect to \(N\) and \(T\); label noise breaks the symmetry. By analogy, this provides a clean capacity-versus-data template for in-context settings where context examples may play the role of temporary data and model parameters the role of stored capacity [2405.19398].

A different adjacent line derives transformer scaling laws on low-dimensional manifolds. Under a manifold hypothesis with intrinsic dimension \(d\) and \(\beta\)-Hölder targets, the paper states
\[
\mathbb E \|\hat T_n-f\|_{L^2(Q)}^2 \lesssim n^{-2\beta/(2\beta+d)},
\qquad
\inf_{T\in\mathcal T}\|T-f\|_{L^\infty(M)}^2 \lesssim N^{-2\beta/d}.
\]
This is not an ICL theorem, but it isolates an important geometric claim: scaling exponents are governed by intrinsic rather than ambient dimension. This suggests that heterogeneous in-context exponents across tasks may reflect different effective task or prompt manifolds rather than only different model sizes [2411.06646].

A third adjacent line studies invariance under data transformations. It states that bijective transformations preserve mutual information,
\[
I(T(X); Y) = I(X; Y),
\]
and therefore preserve scaling exponents, whereas non-bijective transformations change the law through an information-resolution variable
\[
\rho(T) = \frac{I(X';Y)}{I(X;Y)},
\]
yielding
\[
L(N, D, \rho) = \frac{A}{N^\alpha} + \frac{B}{D^\beta}\rho^{-\nu} + E + \kappa(1-\rho)^\mu.
\]
This is a training-time result rather than an ICL result. Still, it suggests that reversible prompt re-encodings could preserve in-context exponents, whereas truncation, lossy retrieval, or prompt compression could alter them through a context-resolution analogue of \(\rho\) [2605.07546].

A fourth adjacent theory roots scaling laws in a percolation model of the data distribution. It derives one regime with power-law-distributed discrete subtasks and another with a dominant manifold, and explicitly remarks that clustered data with many rare classes can drive emergent in-context learning in transformers. This suggests two possible ICL regimes: a retrieval-like regime over heavy-tailed latent subtasks, and a local interpolation regime over a connected task manifold. The paper stops short of a direct prompt-length theorem, but it makes the dependence of scaling exponents on data-distribution structure explicit [2412.07942].

## 7. Limitations and unresolved questions

The current state of in-context neural scaling laws is structurally plural rather than unified. Some theories are exact but highly stylized: deep linear self-attention on in-context regression, Gaussian teacher-student models, or random-feature kernels. Some are directly about prompt-conditioned behavior but only at the level of empirical curve fitting. Some are mechanistic but presently contain unresolved internal inconsistencies. Some are rigorous transformer scaling theories that are adjacent to ICL rather than directly about it [2510.01098].

A first limitation is task scope. Direct theories often analyze regression, synthetic latent task families, or stylized prompts rather than open-ended autoregressive language modeling. A second is architectural scope. Several results depend on linear attention, random features, NTK-style conditioning, or width proxies such as \(N=\Theta(Ld)\) rather than on full transformer parameterization. A third is statistical scope. Most theories assume i.i.d. demonstrations, latent task mixtures, or smooth structured tasks, whereas real prompts are often noisy, non-i.i.d., instruction-like, out-of-distribution, or adversarial. A fourth is the distinction between context use and in-context learning proper: some of the strongest context-horizon results concern autoregressive conditioning rather than few-shot task adaptation [2410.16531].

The central unresolved question is whether a single law can jointly account for parametric knowledge, prompt-conditioned adaptation, usable context horizon, retrieval or external memory, and test-time compute. The current literature supports several narrower conclusions: posterior-style evidence accumulation yields saturating demonstration curves; depth and width are not interchangeable in ICL; context and training compute interact rather than add; language statistics can determine how effective context grows with data; and data geometry or information preservation can alter exponents in principled ways. What remains missing is a broadly validated theory that simultaneously treats sequence structure, prompt semantics, transformer internals, and deployment-time interventions such as retrieval, summarization, and many-shot prompting [2510.14919].

Source: https://www.emergentmind.com/topics/in-context-neural-scaling-laws