---
title: Farseer Scaling Law for LLM Pretraining
url: https://www.emergentmind.com/topics/farseer-scaling-law
type: topic
---

# Farseer Scaling Law for LLM Pretraining

Searching arXiv for recent papers on Farseer scaling law and related extensions.
Farseer Scaling Law is a refined empirical scaling framework for large language model pretraining loss in which the loss surface is modeled as a function of model size \(N\) and training tokens \(D\), but unlike Chinchilla-style laws, the data-dependent term is itself \(N\)-dependent. In its general form, the law is written as
\[
L(N, D) = e^{a_3 N^{\gamma} + b_3} + e^{a_2 N^{\beta} + b_2}\; D^{-e^{a_1 N^{\alpha} + b_1}},
\]
and the original LLaMA-style study reported a concrete fit from approximately 1,000 LLMs trained across diverse scales and configurations, consuming roughly 3 million NVIDIA H100 GPU hours, with the stated aim of constructing a model loss surface \(L(N,D)\) that extrapolates more reliably than prior laws such as Chinchilla [2506.10972].

## 1. Origins and formal definition

Farseer emerged in the context of neural scaling laws that model validation or test loss as a smooth function of model size, data size, and compute. A canonical antecedent is the Chinchilla form
\[
L(N, D) = E + \frac{A}{N^a} + \frac{B}{D^b},
\]
with constant exponents \(a\) and \(b\), a constant irreducible loss \(E\), and an additive, separable dependence on \(N\) and \(D\) [2510.08702]. A related explicit program treated scaling laws as a forecasting system for converged loss, training steps, processed tokens, critical batch size, and full loss trajectories, and argued that the original OpenAI/Kaplan functional forms remain valid when scaling model size up to 33 billion, while their coefficients vary strongly with experiment setup [2403.06563].

The original Farseer paper reparameterized this landscape by positing a general decomposition
\[
L(N, D) = E + U(N) + V(D) + H(N, D),
\]
where \(U(N)\) is a model-size-only term, \(V(D)\) is a data-only term, and \(H(N,D)\) is an interaction term capturing coupled effects of model and data scale [2506.10972]. Its central empirical claim is that the combined data-dependent component behaves as a power law in \(D\), but with both coefficient and exponent depending on \(N\). This yields the general Farseer form
\[
L(N,D) = e^{a_3 N^{\gamma} + b_3} + e^{a_2 N^{\beta} + b_2}\; D^{-e^{a_1 N^{\alpha} + b_1}},
\]
which may also be written, in the notation quoted by later work, as
\[
L(N,D) = e^{s \cdot N^q + S} + e^{B \cdot N^b + Q} \cdot D^{-\,e^{A \cdot N^a + E}}.
\]
The distinguishing feature is that the exponent on \(D\) is no longer constant; it depends on \(N\), so the marginal benefit of additional data varies across model sizes [2510.08702].

For the main LLaMA-style family in the original study, the reported fitted law is
\[
L(N, D) = e^{-0.021 \cdot N^{0.169} -0.091}
+ e^{88.01 \cdot N^{-0.1} - 6.287}\cdot D^{-e^{-0.124 \cdot N^{0.123} + 0.424}},
\]
where \(N\) is the non-embedding parameter count, \(D\) is the number of training tokens, and \(L(N,D)\) is Bits Per Character on a high-quality validation set [2506.10972].

## 2. Differential piecewise fitting and construction of the loss surface

The fitting procedure in Farseer is based on a differential view of the loss surface rather than a direct global fit. The paper defines finite differences in the data and model directions,
\[
\Delta_D L(N,D) = L(N,D) - L(N,\lambda D), \qquad
\Delta_N L(N,D) = L(N,D) - L(\lambda N,D),
\]
with \(\lambda = \sqrt{2}\) as the step in the geometric grid [2506.10972]. Empirically, plotting \(\Delta_D L\) against \(D\) on log–log axes produces an approximately straight line across many model sizes, which motivates the hypothesis
\[
V(D)+H(N,D)=B(N)D^{-A(N)}+\varepsilon_R(N,D).
\]

The fitting then proceeds in three stages. First, for each fixed \(N\), the paper forms
\[
R_N(D)=L(N,D)-L(N,\lambda D)\approx B(N)(1-\lambda^{-A(N)})D^{-A(N)},
\]
and fits
\[
\log\bigl(\tilde{R}_N(D)\bigr)=\log(\hat{B}_N)-A_N\log(D)
\]
to recover discrete estimates \(\{A_N\}\) and \(\{B_N\}\). Second, it searches for simple continuous functions \(f_A(N;\theta_A)\) and \(f_B(N;\theta_B)\) using a small dictionary of transformations and an automated selection procedure. The best fits are reported as stretched exponentials,
\[
f_A(N;\theta_A)=\exp\!\bigl(a_1 N^{\alpha}+b_1\bigr), \qquad
f_B(N;\theta_B)=\exp\!\bigl(a_2 N^{\beta}+b_2\bigr).
\]
Third, after subtracting the fitted data-dependent term, the residual
\[
O(N,D)\triangleq L(N,D)-B(N)D^{-A(N)}
\]
is averaged across datasets for each \(N\), producing \(G(N)\), which is then fit by another stretched exponential of the form
\[
E+U(N)\simeq G(N)\approx \exp\!\bigl(a_3 N^{\gamma}+b_3\bigr)
\]
[2506.10972].

This construction is explicitly family-specific. The original study fixes an architecture family \(\mathbb{A}\) and data recipe \(\mathbb{D}\), treats \(L_{\mathbb{A},\mathbb{D}}(N,D)\) as the target surface, and trains approximately 1,000 LLaMA-style models with standardized hyperparameters and two main data recipes, Baseline and EN–ZH bilingual. The project states that all models, data, results, and logs are being open-sourced at `https://github.com/Farseer-Scaling-Law/Farseer`, together with the fitting code and the algorithms described in the appendix for Optimal Transformation Selection and Differential Piecewise Fitting [2506.10972].

## 3. Empirical behavior, extrapolation, and compute-optimality

The fitted Farseer surface is reported to be smooth and monotone in both axes. In the main study, the derivative with respect to \(D\) is
\[
\frac{\partial L}{\partial D}=-A(N)\,B(N)\,D^{-A(N)-1}<0,
\]
and the derivative with respect to \(N\) is described as negative everywhere based on numerical verification [2506.10972]. The model-size-only term
\[
U(N)=e^{-0.021N^{0.169}-0.091}
\]
acts as a capacity-controlled floor, while the data-dependent term
\[
B(N)D^{-A(N)}
\]
changes its slope as \(N\) changes, rather than obeying a single global exponent.

The paper’s principal empirical claim concerns extrapolation. When fitted on a grid of relatively small \(N\) and \(D\) and evaluated on six off-grid extrapolation targets, including a 25.1B model, Farseer is reported to achieve an average relative error of \(0.50\%\), while the Chinchilla scaling law exhibits an average relative error of \(2.68\%\), described in the paper as “a 433% increase” [2506.10972]. The same study reports that Farseer’s average relative error across the fitted range is 232% lower than Chinchilla’s, and that its error remains stable across the full \(N\) range, whereas Chinchilla’s error grows at the extremes.

Farseer is also used as a compute-allocation tool under the standard approximation
\[
C \approx 6ND,
\]
where \(C\) is total pretraining FLOPs [2506.10972]. Chinchilla’s analysis implies a roughly constant token-to-parameter ratio, often quoted around \(D/N \approx 20\), but Farseer recomputes the optimum by minimizing its richer \(L(N,D)\) surface under the compute constraint. The reported result is a steadily increasing optimal \(D/N\) as \(C\) grows, with larger-scale training requiring significantly more data per parameter than the older constant-ratio rule suggests. The paper further states that this rising \(D/N\) is qualitatively consistent with the training configurations of recent large LLMs such as Llama 3.1 and Qwen3 [2506.10972].

A later position paper fit Farseer data with fixed-exponent forms and reported, for the Farseer models, \(c_D = 918 \pm 6\), \(c_N = 331 \pm 3\), \(L_0 = 1.542 \pm 0.005\), a derived compute coefficient \(c_C = 1486 \pm 10\), and an optimal \(D/N = 21.3 \pm 0.5\), while also reporting free-exponent fits \(\alpha_N = 0.30 \pm 0.01\) and \(\alpha_D = 0.349 \pm 0.009\), both close to \(1/3\) [2606.25008]. This suggests an interpretive bridge between Farseer’s empirical surface and later universality arguments.

## 4. Domain-specific and architectural extensions

A major downstream use of Farseer has been in code LLM scaling. In a study of 117 code-only runs with model sizes from 0.2B to 3.8B non-embedding parameters and training tokens from 2B to 128B, the authors fit both a Chinchilla law and a Farseer law and report that the “more expressive Farseer law offers greater accuracy” [2510.08702]. Their code-specific fitted Farseer law is
\[
\begin{aligned}
L(N,D) &= \exp(-0.0047 \cdot N^{0.239} - 0.8188) \\
&\quad + \exp(62.8936 \cdot N^{-0.0614} - 14.0414)\cdot D^{-\exp(-0.0209 \cdot N^{0.1943} - 0.1826)},
\end{aligned}
\]
with \(N\) and \(D\) used as raw numeric values in billions. On those 117 runs, the reported mean relative error is \(0.82\permil\) for Farseer and \(1.03\permil\) for Chinchilla. On three larger validation points at fixed \(C=5.36\times 10^{21}\) FLOPs and \(D/N\in\{20,150,424\}\), the empirically best loss is at \(D/N=150\), which the paper identifies as code-optimal and far above the canonical NL ratio of about 20 [2510.08702]. The same work also studies two code–NL mixtures, 70% code + 30% NL and 30% code + 70% NL, and reports that NL helps in small-model or low-code-data regimes, while at \(N=0.94\)B pure code is better across all \(D_{\text{code}}/N\) ratios [2510.08702].

A separate extension interprets familial-model scaling as a Farseer-like generalization to a third axis, granularity \(G\). Familial models are defined as shared Transformer backbones with multiple early-exit heads, so that one training run produces \(G\) deployable sub-models. The proposed unified law is
\[
L(N, D, G)=\left(E+\frac{A}{N^\alpha}+\frac{B}{D^\beta}\right)G^\gamma,
\]
which reduces to the classical two-dimensional law when \(G=1\) [2512.23407]. In the representative fit reported in that work, the model-size exponent is \(\alpha=0.3006\), the data exponent is \(\beta=0.3514\), and the granularity exponent is \(\gamma=0.041\), which the paper describes as “extremely small.” Numerically, \(3^{0.041}\approx 1.04\) and \(8^{0.041}\approx 1.11\), so increasing the number of exits only modestly increases loss. The same study therefore argues that familial models preserve compute-optimal scaling behavior and that “train once, deploy many” is compatible with dense-model compute frontiers under an IsoFLOP design [2512.23407].

These extensions preserve the core Farseer idea—loss is not governed by a single global data exponent—but broaden the scope of the law. In code, the \(N\)-dependent data term is used to characterize a “more data-hungry regime.” In familial models, a Farseer-like perspective is used to add deployment granularity as a third scaling variable.

## 5. Related theoretical interpretations

Farseer itself is empirical, but a number of adjacent works have been used to interpret what a broader Farseer-like scaling framework might mean. One explicit scaling-law program estimates constants from models with only 1M–60M parameters and then predicts converged loss, minimum required training steps and processed tokens, critical batch size, and the full test-loss trajectory for models up to 33B parameters. Its central finite-batch trajectory equation is
\[
L(N, S, B)=\left(\frac{N_c}{N}\right)^{\alpha_N}+\left(\frac{S_c}{S}\right)^{\alpha_S}
\left(1+\frac{B_*}{B\cdot L(N,S,B)^{1/\alpha_B}}\right)^{\alpha_S},
\]
and the paper argues that this turns the OpenAI/Kaplan theory into a practical forecasting system for new training setups [2403.06563]. This suggests a broader interpretation of Farseer as a forecasting framework rather than only a two-variable fit.

Rigorous theory has also been developed in simplified settings. In quadratically parameterized linear regression, one paper derives an excess-risk scaling law under power-law spectral assumptions, with effective dimension
\[
D \asymp \min\Big\{ T^{1/\max\{\beta,(\alpha+\beta)/2\}},\, M\Big\},
\]
and a main upper bound
\[
\mathcal{R}_M(\mathbf{v}^T)-[\xi^2]
\lesssim \frac{1}{M^{\beta-1}}+\frac{\sigma^2 D}{T}+\left(\frac{D}{T}+\frac{1}{D^{\beta-1}}\mathbf{1}_{D<M}\right),
\]
together with improved rates over linear SGD in the regime \(\alpha>\beta\) [2502.09106]. A separate regression study states a multiple-regression scaling law of the form
\[
\E[L_R(V_n)] = \sigma^2 + \Theta\Big(\frac{1}{M^{a-1}}\Big) + \Theta\Big(\frac{1}{(N_{\rm eff}\gamma)^{(a-1)/a}}\Big),
\]
and extends analogous results to kernel regression under power-law spectra [2503.01314]. These works do not define Farseer, but they suggest a route by which fitted exponents may be linked to spectral decay, implicit regularization, and feature learning.

A more speculative interpretive layer is supplied by a 2026 position paper on neural scaling universality. That paper argues that current dense Transformer LLMs occupy a universality class with fixed exponents arising from generic mechanisms: one-third time scaling from Softmax nonlinearity, inverse-width scaling from representational superposition, and inverse-depth scaling from ensemble averaging of Transformer layers. It writes
\[
L(\tau,m,\ell)=\frac{c_\tau}{\tau^{1/3}}+\frac{c_m}{m}+\frac{c_\ell}{\ell}+L_0,
\]
and, at optimal shape and compute-optimality, derives
\[
L(D,N)=\frac{c_D}{D^{1/3}}+\frac{c_N}{N^{1/3}}+L_0,\qquad
L(C)=\frac{c_C}{C^{1/6}}+L_0,
\]
then fits Farseer data and reports exponents close to \(1/3\) and \(1/6\) [2606.25008]. This does not replace the original Farseer formalism, but it suggests that some of Farseer’s empirical behavior may be interpretable as coefficient variation within a fixed-exponent universality class.

## 6. Scope, limitations, and open questions

The original Farseer paper is explicit that its principal fitted law is family-specific. The main equation is calibrated on dense, LLaMA-style decoder-only Transformers, with the largest validation model at 25.1B parameters, two data recipes analyzed in detail, and Bits Per Character on a specially constructed validation set as the target metric [2506.10972]. The paper also states that Farseer assumes training is “appropriate,” in the sense of standardized hyperparameters and scale-aware heuristics, and that it remains an empirical law without a first-principles derivation.

The code adaptation also states clear limits. Its fit uses 117 points, one decoder-only Transformer family with SwiGLU, RoPE, and RMSNorm, model sizes only up to 3.8B parameters, and only two code–NL mixture ratios, 70/30 and 30/70 [2510.08702]. The authors note that extrapolation to frontier code models with tens of billions of parameters and much larger token budgets remains to be validated, and that the near-zero irreducible loss predicted by the fitted code-Farseer law is theoretically motivated rather than empirically verified at extreme scales.

The familial-model extension is likewise limited to models up to approximately 4B parameters, specific exit placements, and pretraining language-modeling loss as the optimization target [2512.23407]. Relay-style cooperative inference is discussed theoretically, but the paper reports no detailed experiments on dynamic routing or adaptive exiting. Its proposed generalization to other modalities and more complex dynamic architectures, including MoE plus familial models and higher-dimensional laws such as \(L(N,D,G,S,\dots)\), remains open.

A separate practical-scaling program argues that both Chinchilla and Farseer-like \(L(N,D)\) laws are calibrated mainly for data-rich, single-epoch pretraining and therefore do not represent baseline saturation, overfitting, or the distinction between unique data \(D\) and total examples seen \(T\). It proposes
\[
L(N, D, T)=E+(L_0-E)\frac{h}{1+h},\qquad
h=\frac{a}{N^\alpha}+\frac{b}{T^\beta}+c\frac{N^\gamma}{D^\delta},
\]
and validates this form on four multi-epoch experiments and five published LLM scaling-law grids, including Farseer’s [2605.09189]. That work argues that a general scaling framework should separate undercapacity, undertraining, and overfitting, and should support cost-aware allocation with explicit prices for data and compute. This suggests that the present scope of Farseer is strongest in the standard pretraining regime, whereas a broader theory of scaling may require an explicit third axis for total training exposure and a closed-form overfitting term.

Taken together, these limitations define the current status of Farseer Scaling Law. It is a refined and empirically successful \(L(N,D)\) law for dense LLM pretraining, with later domain-specific fits for code and a clear conceptual path toward higher-dimensional extensions. Its main unresolved questions concern theoretical derivation, robustness outside single-epoch data-rich regimes, transfer across architecture families, and the extent to which fitted exponents reflect genuine universality rather than the particular experimental envelopes in which Farseer has been calibrated.

Source: https://www.emergentmind.com/topics/farseer-scaling-law