Compute-Optimal Frontier

• Compute-optimal frontier is the set of model configurations that maximize performance within fixed compute (FLOP) constraints.
• The approach employs empirical scaling laws and constrained optimization (e.g., Lagrangian/KKT methods) to guide efficient resource allocation.
• It provides practical insights for balancing model size, data, and architectural choices to avoid compute wastage in different application domains.

A compute-optimal frontier, in the context of machine learning and statistical modeling, delineates the locus of model-design parameters that yield the highest achievable performance given a fixed (often stringent) compute constraint, typically measured in floating-point operations (FLOPs). This concept formalizes the trade-offs among model size, data usage, architectural choices, and other scaling factors under resource limitations, enabling principled allocation strategies that avoid compute wastage and underutilization. Unlike unconstrained scaling, which may yield significant inefficiency or suboptimal outcomes, optimizing explicitly over the compute frontier ensures that each marginal FLOP contributes maximally to downstream task performance, often yielding scaling laws, constrained optimization conditions, and design recipes tuned for empirical regimes of interest.

1. Mathematical Definition and General Formulation

Let $C_0$ denote a fixed compute budget (e.g., FLOPs per inference or per training run), and $f(x)$ a loss or negative performance function dependent on one or more scaling variables $x = (N_1, N_2, \ldots)$, such as model size, number of frames, or number of tokens. The compute-optimal frontier is the set of configurations

$$x^*(C_0) = \arg\min_{x} f(x) \quad \text{s.t.} \quad C(x) = C_0$$

where $C(x)$ is a (typically analytical) compute cost model. $f(x)$ is often a parametric scaling law, empirically fit to task performance, and may include interactive terms, data size dependence, and diminishing returns. Practical exploration typically involves parametric sweeps and grid-based search, followed by analytic or numerical optimization subject to hardware or architectural constraints.

In video vision-LLMs, for example, the inference compute cost is given by

$C(N,F,V) = 2F(MW + NV)$

for LM parameter count $N$, frames per example $F$, and tokens per frame $V$ (Wang et al., 24 May 2025).

2. Empirical Scaling Laws and Frontier Analytics

Compute-optimal frontiers are grounded in empirically observed scaling-laws that relate task loss/performance to model and data size. In LLMs, the canonical law is

$L(N,D) = E + A N^{-\alpha} + B D^{-\beta}$

for irreducible loss $E$, with exponents $(\alpha, \beta)$ and coefficients $(A, B)$ determined via fits across sweeps (Hoffmann et al., 2022, Ziarko et al., 2024, Cheng et al., 2024). For video VLMs, the additive-interact model is

$$f(N,F,V;n) = \sum_{k\in\{N,F,V\}} \alpha_k x_k^{-a_k} + \sum_{k\in\{N,F,V\}} \beta_k x_k^{b_k} n^{-d} + \xi n^{-d} + \varepsilon$$

with fitted $(a_k, b_k, d, \alpha_k, \beta_k, \xi)$ (Wang et al., 24 May 2025).

The scaling exponents directly dictate frontier elasticity, i.e., how optimal allocation shifts as compute or data size varies. Closed-form solutions result when analytic cost and loss models permit; otherwise, constrained numerical search is used.

3. Constrained Optimization, KKT Conditions, and Solution Methods

Optimizing along the compute frontier is a constrained optimization problem, often solved via Lagrangian or Karush-Kuhn-Tucker (KKT) conditions. For a generic parametric loss $f(x;n)$ and resource function $C(x)$, introduce the Lagrangian

$$\mathcal{L}(x, \lambda) = f(x; n) - \lambda(C(x) - C_0)$$

and set partial derivatives to zero. For video VLMs (Wang et al., 24 May 2025): $$\begin{aligned} \frac{\partial \mathcal{L}}{\partial N} &= -a_N \alpha_N N^{-a_N-1} + b_N \beta_N N^{b_N-1} n^{-d} - \lambda(2FV) = 0 \ \frac{\partial \mathcal{L}}{\partial F} &= -a_F \alpha_F F^{-a_F-1} + b_F \beta_F F^{b_F-1} n^{-d} - \lambda 2(MW + NV) = 0 \ \frac{\partial \mathcal{L}}{\partial V} &= -a_V \alpha_V V^{-a_V-1} + b_V \beta_V V^{b_V-1} n^{-d} - \lambda 2FN = 0 \ C(N,F,V) &= C_0 \end{aligned}$$ Practical solutions employ discrete grid search coupled with analytic sensitivity analysis. The resultant expressions yield compute-elasticity scalings, such as

$$N^*(C_0) \propto C_0^{0.30}, \quad F^*(C_0) \propto C_0^{0.20}, \quad V^*(C_0) \propto C_0^{0.50}$$

and data-size elasticities such as $$e_N \approx -0.22, e_F \approx +0.17, e_V \approx +0.79$$ (Wang et al., 24 May 2025).

4. Frontier Construction in Multi-Objective and Empirical Settings

In settings where objective trade-offs are explicit (e.g., fairness vs. relevance in recommender systems or accuracy vs. compute in SSL), the compute-optimal frontier is operationalized by Pareto efficiency: $$F = \{(c, a) \mid \forall m',\, C(m') \leq c \Rightarrow A(m') \leq a \}$$ with "non-dominated" points comprising the frontier (Li et al., 29 Sep 2025, Rampisela et al., 17 Feb 2025). Stepwise algorithms—sort by ascending compute and prune points that are accuracy-dominated at equal or lower compute—efficiently construct this frontier for discrete model families. In empirical accuracy–FLOP curves, this method determines the set of configurations that maximize achievable accuracy given a FLOP budget, as in the dominance of SALT over V-JEPA-2 (Li et al., 29 Sep 2025).

5. Empirical Guidelines and Practical Prescriptions

Principled descriptions of compute-optimal frontiers yield direct advice for model selection and scaling:

• Video VLMs: Always scale $(N, F, V)$ jointly—bottlenecks in any one dimension waste FLOPs. At low budgets ($\sim$2 TFLOPs) optimal models are small LMs ($N=1$B) but maximize $V$; at high budgets ($\sim$30–100 TFLOPs), allocate more to $N$ with moderate $F, V$. As finetuning data $n$ increases, shift compute from $N$ to $F$ and especially $V$ (Wang et al., 24 May 2025).
• Language/Protein Models: For given $C$, compute-optimal $N^*(C), D^*(C)$ obey scaling exponents; e.g., $N^*, D^* \propto C^{0.5}$ for LLMs (Hoffmann et al., 2022), $N^*, D^* \propto C^{\alpha'}, C^{\beta'}$ with known exponents for protein LMs (Cheng et al., 2024). Recipes specify: choose $N^*, D^*$ from scaling laws, invert $C = 6ND$.
• Self-supervised Video SSL: Always prefer frozen-teacher (SALT) over EMA-based distillation for compute-allocation; a minimal teacher, maximal student training is strictly optimal for compute–accuracy (Li et al., 29 Sep 2025).
• Embedding Models: Optimize over not just $(N,D)$ but fine-tuning method $M$ (full vs. LoRA etc.); the frontier switches from full at small $B$ to LoRA at large $B$ (Ziarko et al., 2024).

6. Frontier Shifts, Elasticity, and Data Regimes

The location and shape of the compute-optimal frontier shift systematically with data size, method, and domain:

• Data Size: As fine-tuning data increases, elasticity coefficients quantify allocation changes; e.g., in video VLMs, higher $n$ recommends decreasing $N$ and increasing $F, V$.
• Task Domain: Scaling exponents for loss vs. compute exhibit domain dependence: exponents for protein LMs differ markedly between CLM and MLM objectives (Cheng et al., 2024).
• Frontier Transitions: Methodological transitions (e.g., full fine-tuning to LoRA) are observed at critical compute thresholds (Ziarko et al., 2024).
• Diminishing Returns: IsoFLOP scaling curves flatten for both extreme $N$ or $D$, indicating diminishing returns outside the optimal regime.

7. Broader Context and Theoretical Significance

The compute-optimal frontier provides a systematic approach for balancing efficiency and performance under finite resources. Its theoretical basis extends to generalization-bounds: on the Chinchilla compute-optimal frontier, larger LLMs exhibit provably shrinking generalization gaps owing to fixed parameter-per-token ratios, with loss-variance and quantization error terms decaying with model scale (Finzi et al., 21 Apr 2025). In practice, the frontier framework unifies Pareto-optimality in multi-objective contexts (Rampisela et al., 17 Feb 2025, Singh et al., 2021), explicit optimization in nonparametric estimation (Nazin et al., 2014, Girard et al., 2011), and self-supervised learning regimes (Li et al., 29 Sep 2025).

By providing both analytic and empirical recipes for resource-constrained model optimization, compute-optimal frontier analysis is central to principled machine learning, informing everything from training and inference protocols to architectural design and resource allocation across modalities and domains.