Farseer Scaling Law for LLM Pretraining

• Farseer Scaling Law is an empirical framework that models LLM loss as a function of model size and training tokens with an N-dependent data exponent.
• It refines prior laws like Chinchilla by capturing the coupled effects of model and data scale, resulting in significantly lower extrapolation errors.
• The framework is applied to dense LLMs, code-specific models, and familial architectures, guiding compute-optimal training strategies.

Searching arXiv for papers on Farseer scaling law and related extensions. Farseer Scaling Law is a refined empirical scaling framework for LLM pretraining loss in which the loss surface is modeled as a function of model size $N$ and training tokens $D$, but unlike Chinchilla-style laws, the data-dependent term is itself $N$-dependent. In its general form, the law is written as

$$L(N, D) = e^{a_3 N^{\gamma} + b_3} + e^{a_2 N^{\beta} + b_2}\; D^{-e^{a_1 N^{\alpha} + b_1}},$$

and the original LLaMA-style study reported a concrete fit from approximately 1,000 LLMs trained across diverse scales and configurations, consuming roughly 3 million NVIDIA H100 GPU hours, with the stated aim of constructing a model loss surface $L(N,D)$ that extrapolates more reliably than prior laws such as Chinchilla (Li et al., 12 Jun 2025).

1. Origins and formal definition

Farseer emerged in the context of neural scaling laws that model validation or test loss as a smooth function of model size, data size, and compute. A canonical antecedent is the Chinchilla form

$L(N, D) = E + \frac{A}{N^a} + \frac{B}{D^b},$

with constant exponents $a$ and $b$, a constant irreducible loss $E$, and an additive, separable dependence on $N$ and $D$0 (Luo et al., 9 Oct 2025). A related explicit program treated scaling laws as a forecasting system for converged loss, training steps, processed tokens, critical batch size, and full loss trajectories, and argued that the original OpenAI/Kaplan functional forms remain valid when scaling model size up to 33 billion, while their coefficients vary strongly with experiment setup (Su et al., 2024).

The original Farseer paper reparameterized this landscape by positing a general decomposition

$D$1

where $D$2 is a model-size-only term, $D$3 is a data-only term, and $D$4 is an interaction term capturing coupled effects of model and data scale (Li et al., 12 Jun 2025). Its central empirical claim is that the combined data-dependent component behaves as a power law in $D$5, but with both coefficient and exponent depending on $D$6. This yields the general Farseer form

$D$7

which may also be written, in the notation quoted by later work, as

$D$8

The distinguishing feature is that the exponent on $D$9 is no longer constant; it depends on $N$0, so the marginal benefit of additional data varies across model sizes (Luo et al., 9 Oct 2025).

For the main LLaMA-style family in the original study, the reported fitted law is

$N$1

where $N$2 is the non-embedding parameter count, $N$3 is the number of training tokens, and $N$4 is Bits Per Character on a high-quality validation set (Li et al., 12 Jun 2025).

2. Differential piecewise fitting and construction of the loss surface

The fitting procedure in Farseer is based on a differential view of the loss surface rather than a direct global fit. The paper defines finite differences in the data and model directions,

$N$5

with $N$6 as the step in the geometric grid (Li et al., 12 Jun 2025). Empirically, plotting $N$7 against $N$8 on log–log axes produces an approximately straight line across many model sizes, which motivates the hypothesis

$N$9

The fitting then proceeds in three stages. First, for each fixed $$L(N, D) = e^{a_3 N^{\gamma} + b_3} + e^{a_2 N^{\beta} + b_2}\; D^{-e^{a_1 N^{\alpha} + b_1}},$$0, the paper forms

$$L(N, D) = e^{a_3 N^{\gamma} + b_3} + e^{a_2 N^{\beta} + b_2}\; D^{-e^{a_1 N^{\alpha} + b_1}},$$1

and fits

$$L(N, D) = e^{a_3 N^{\gamma} + b_3} + e^{a_2 N^{\beta} + b_2}\; D^{-e^{a_1 N^{\alpha} + b_1}},$$2

to recover discrete estimates $$L(N, D) = e^{a_3 N^{\gamma} + b_3} + e^{a_2 N^{\beta} + b_2}\; D^{-e^{a_1 N^{\alpha} + b_1}},$$3 and $$L(N, D) = e^{a_3 N^{\gamma} + b_3} + e^{a_2 N^{\beta} + b_2}\; D^{-e^{a_1 N^{\alpha} + b_1}},$$4. Second, it searches for simple continuous functions $$L(N, D) = e^{a_3 N^{\gamma} + b_3} + e^{a_2 N^{\beta} + b_2}\; D^{-e^{a_1 N^{\alpha} + b_1}},$$5 and $$L(N, D) = e^{a_3 N^{\gamma} + b_3} + e^{a_2 N^{\beta} + b_2}\; D^{-e^{a_1 N^{\alpha} + b_1}},$$6 using a small dictionary of transformations and an automated selection procedure. The best fits are reported as stretched exponentials,

$$L(N, D) = e^{a_3 N^{\gamma} + b_3} + e^{a_2 N^{\beta} + b_2}\; D^{-e^{a_1 N^{\alpha} + b_1}},$$7

Third, after subtracting the fitted data-dependent term, the residual

$$L(N, D) = e^{a_3 N^{\gamma} + b_3} + e^{a_2 N^{\beta} + b_2}\; D^{-e^{a_1 N^{\alpha} + b_1}},$$8

is averaged across datasets for each $$L(N, D) = e^{a_3 N^{\gamma} + b_3} + e^{a_2 N^{\beta} + b_2}\; D^{-e^{a_1 N^{\alpha} + b_1}},$$9, producing $L(N,D)$0, which is then fit by another stretched exponential of the form

$L(N,D)$1

(Li et al., 12 Jun 2025).

This construction is explicitly family-specific. The original study fixes an architecture family $L(N,D)$2 and data recipe $L(N,D)$3, treats $L(N,D)$4 as the target surface, and trains approximately 1,000 LLaMA-style models with standardized hyperparameters and two main data recipes, Baseline and EN–ZH bilingual. The project states that all models, data, results, and logs are being open-sourced at https://github.com/Farseer-Scaling-Law/Farseer, together with the fitting code and the algorithms described in the appendix for Optimal Transformation Selection and Differential Piecewise Fitting (Li et al., 12 Jun 2025).

3. Empirical behavior, extrapolation, and compute-optimality

The fitted Farseer surface is reported to be smooth and monotone in both axes. In the main study, the derivative with respect to $L(N,D)$5 is

$L(N,D)$6

and the derivative with respect to $L(N,D)$7 is described as negative everywhere based on numerical verification (Li et al., 12 Jun 2025). The model-size-only term

$L(N,D)$8

acts as a capacity-controlled floor, while the data-dependent term

$L(N,D)$9

changes its slope as $L(N, D) = E + \frac{A}{N^a} + \frac{B}{D^b},$0 changes, rather than obeying a single global exponent.

The paper’s principal empirical claim concerns extrapolation. When fitted on a grid of relatively small $L(N, D) = E + \frac{A}{N^a} + \frac{B}{D^b},$1 and $L(N, D) = E + \frac{A}{N^a} + \frac{B}{D^b},$2 and evaluated on six off-grid extrapolation targets, including a 25.1B model, Farseer is reported to achieve an average relative error of $L(N, D) = E + \frac{A}{N^a} + \frac{B}{D^b},$3, while the Chinchilla scaling law exhibits an average relative error of $L(N, D) = E + \frac{A}{N^a} + \frac{B}{D^b},$4, described in the paper as “a 433% increase” (Li et al., 12 Jun 2025). The same study reports that Farseer’s average relative error across the fitted range is 232% lower than Chinchilla’s, and that its error remains stable across the full $L(N, D) = E + \frac{A}{N^a} + \frac{B}{D^b},$5 range, whereas Chinchilla’s error grows at the extremes.

Farseer is also used as a compute-allocation tool under the standard approximation

$L(N, D) = E + \frac{A}{N^a} + \frac{B}{D^b},$6

where $L(N, D) = E + \frac{A}{N^a} + \frac{B}{D^b},$7 is total pretraining FLOPs (Li et al., 12 Jun 2025). Chinchilla’s analysis implies a roughly constant token-to-parameter ratio, often quoted around $L(N, D) = E + \frac{A}{N^a} + \frac{B}{D^b},$8, but Farseer recomputes the optimum by minimizing its richer $L(N, D) = E + \frac{A}{N^a} + \frac{B}{D^b},$9 surface under the compute constraint. The reported result is a steadily increasing optimal $a$0 as $a$1 grows, with larger-scale training requiring significantly more data per parameter than the older constant-ratio rule suggests. The paper further states that this rising $a$2 is qualitatively consistent with the training configurations of recent large LLMs such as Llama 3.1 and Qwen3 (Li et al., 12 Jun 2025).

A later position paper fit Farseer data with fixed-exponent forms and reported, for the Farseer models, $a$3, $a$4, $a$5, a derived compute coefficient $a$6, and an optimal $a$7, while also reporting free-exponent fits $a$8 and $a$9, both close to $b$0 (Liu et al., 23 Jun 2026). This suggests an interpretive bridge between Farseer’s empirical surface and later universality arguments.

4. Domain-specific and architectural extensions

A major downstream use of Farseer has been in code LLM scaling. In a study of 117 code-only runs with model sizes from 0.2B to 3.8B non-embedding parameters and training tokens from 2B to 128B, the authors fit both a Chinchilla law and a Farseer law and report that the “more expressive Farseer law offers greater accuracy” (Luo et al., 9 Oct 2025). Their code-specific fitted Farseer law is

$b$1

with $b$2 and $b$3 used as raw numeric values in billions. On those 117 runs, the reported mean relative error is $b$4 for Farseer and $b$5 for Chinchilla. On three larger validation points at fixed $b$6 FLOPs and $b$7, the empirically best loss is at $b$8, which the paper identifies as code-optimal and far above the canonical NL ratio of about 20 (Luo et al., 9 Oct 2025). The same work also studies two code–NL mixtures, 70% code + 30% NL and 30% code + 70% NL, and reports that NL helps in small-model or low-code-data regimes, while at $b$9B pure code is better across all $E$0 ratios (Luo et al., 9 Oct 2025).

A separate extension interprets familial-model scaling as a Farseer-like generalization to a third axis, granularity $E$1. Familial models are defined as shared Transformer backbones with multiple early-exit heads, so that one training run produces $E$2 deployable sub-models. The proposed unified law is

$E$3

which reduces to the classical two-dimensional law when $E$4 (Song et al., 29 Dec 2025). In the representative fit reported in that work, the model-size exponent is $E$5, the data exponent is $E$6, and the granularity exponent is $E$7, which the paper describes as “extremely small.” Numerically, $E$8 and $E$9, so increasing the number of exits only modestly increases loss. The same study therefore argues that familial models preserve compute-optimal scaling behavior and that “train once, deploy many” is compatible with dense-model compute frontiers under an IsoFLOP design (Song et al., 29 Dec 2025).

These extensions preserve the core Farseer idea—loss is not governed by a single global data exponent—but broaden the scope of the law. In code, the $N$0-dependent data term is used to characterize a “more data-hungry regime.” In familial models, a Farseer-like perspective is used to add deployment granularity as a third scaling variable.

5. Related theoretical interpretations

Farseer itself is empirical, but a number of adjacent works have been used to interpret what a broader Farseer-like scaling framework might mean. One explicit scaling-law program estimates constants from models with only 1M–60M parameters and then predicts converged loss, minimum required training steps and processed tokens, critical batch size, and the full test-loss trajectory for models up to 33B parameters. Its central finite-batch trajectory equation is

$N$1

and the paper argues that this turns the OpenAI/Kaplan theory into a practical forecasting system for new training setups (Su et al., 2024). This suggests a broader interpretation of Farseer as a forecasting framework rather than only a two-variable fit.

Rigorous theory has also been developed in simplified settings. In quadratically parameterized linear regression, one paper derives an excess-risk scaling law under power-law spectral assumptions, with effective dimension

$N$2

and a main upper bound

$N$3

together with improved rates over linear SGD in the regime $N$4 (Ding et al., 13 Feb 2025). A separate regression study states a multiple-regression scaling law of the form

$N$5

and extends analogous results to kernel regression under power-law spectra (Chen et al., 3 Mar 2025). These works do not define Farseer, but they suggest a route by which fitted exponents may be linked to spectral decay, implicit regularization, and feature learning.

A more speculative interpretive layer is supplied by a 2026 position paper on neural scaling universality. That paper argues that current dense Transformer LLMs occupy a universality class with fixed exponents arising from generic mechanisms: one-third time scaling from Softmax nonlinearity, inverse-width scaling from representational superposition, and inverse-depth scaling from ensemble averaging of Transformer layers. It writes

$N$6

and, at optimal shape and compute-optimality, derives

$N$7

then fits Farseer data and reports exponents close to $N$8 and $N$9 (Liu et al., 23 Jun 2026). This does not replace the original Farseer formalism, but it suggests that some of Farseer’s empirical behavior may be interpretable as coefficient variation within a fixed-exponent universality class.

6. Scope, limitations, and open questions

The original Farseer paper is explicit that its principal fitted law is family-specific. The main equation is calibrated on dense, LLaMA-style decoder-only Transformers, with the largest validation model at 25.1B parameters, two data recipes analyzed in detail, and Bits Per Character on a specially constructed validation set as the target metric (Li et al., 12 Jun 2025). The paper also states that Farseer assumes training is “appropriate,” in the sense of standardized hyperparameters and scale-aware heuristics, and that it remains an empirical law without a first-principles derivation.

The code adaptation also states clear limits. Its fit uses 117 points, one decoder-only Transformer family with SwiGLU, RoPE, and RMSNorm, model sizes only up to 3.8B parameters, and only two code–NL mixture ratios, 70/30 and 30/70 (Luo et al., 9 Oct 2025). The authors note that extrapolation to frontier code models with tens of billions of parameters and much larger token budgets remains to be validated, and that the near-zero irreducible loss predicted by the fitted code-Farseer law is theoretically motivated rather than empirically verified at extreme scales.

The familial-model extension is likewise limited to models up to approximately 4B parameters, specific exit placements, and pretraining language-modeling loss as the optimization target (Song et al., 29 Dec 2025). Relay-style cooperative inference is discussed theoretically, but the paper reports no detailed experiments on dynamic routing or adaptive exiting. Its proposed generalization to other modalities and more complex dynamic architectures, including MoE plus familial models and higher-dimensional laws such as $D$00, remains open.

A separate practical-scaling program argues that both Chinchilla and Farseer-like $D$01 laws are calibrated mainly for data-rich, single-epoch pretraining and therefore do not represent baseline saturation, overfitting, or the distinction between unique data $D$02 and total examples seen $D$03. It proposes

$D$04

and validates this form on four multi-epoch experiments and five published LLM scaling-law grids, including Farseer’s (Bryant et al., 9 May 2026). That work argues that a general scaling framework should separate undercapacity, undertraining, and overfitting, and should support cost-aware allocation with explicit prices for data and compute. This suggests that the present scope of Farseer is strongest in the standard pretraining regime, whereas a broader theory of scaling may require an explicit third axis for total training exposure and a closed-form overfitting term.

Taken together, these limitations define the current status of Farseer Scaling Law. It is a refined and empirically successful $D$05 law for dense LLM pretraining, with later domain-specific fits for code and a clear conceptual path toward higher-dimensional extensions. Its main unresolved questions concern theoretical derivation, robustness outside single-epoch data-rich regimes, transfer across architecture families, and the extent to which fitted exponents reflect genuine universality rather than the particular experimental envelopes in which Farseer has been calibrated.