Budget-Efficient Scaling Law Fitting

• The paper introduces a budget-aware framework that selects cost-effective experiments to accurately fit scaling laws while minimizing resource use.
• It uses robust statistical techniques, including multi-start optimization and uncertainty-guided acquisition, to balance information gain and cost.
• The method achieves near full-grid extrapolation performance using only about 10% of the total experimental cost, demonstrating its efficiency in LLM pretraining and scientific ML.

Budget-efficient scaling law fitting refers to a set of experimental and statistical methodologies for inferring predictive scaling laws—formulas that capture how model performance or loss varies with changes in parameter count, dataset size, compute, or other axes—while minimizing the total computational, monetary, or data-generation cost of the fitting process. These methods are motivated by the prohibitively high expense of exhaustive grid-sweep experiments at frontier model and dataset scales. Recent work systematically formalizes the budget-aware selection and optimization of pilot experiments to support reliable extrapolation under real-world constraints, providing both theoretical guarantees and validated best practices for diverse settings such as LLM pretraining, fine-tuning, data mixture optimization, and scientific ML (Li et al., 24 Apr 2026).

1. Formal Problem Definition and Motivation

The central task is to select and execute a subset of candidate experiments $\mathcal{S} \subseteq \mathcal{E}$, each with known cost $c_i$, so that the total cost does not exceed a fixed budget $B$: $\sum_{e_i \in \mathcal{S}} c_i \leq B$ Then, using results from $\mathcal{S}$, one fits a parametric scaling-law model $f(x; \theta)$ and extrapolates to predict performance in a high-cost "target region" $\mathcal{E}_{\text{tar}}$ (such as at largest model size or data scale), aiming to minimize extrapolation error: $$\text{error}(\mathcal{S}) = \left\| y_{\text{tar}} - f_{\mathcal{S}} \right\|_2^2$$ where $y_{\text{tar}}$ is the true performance vector in $\mathcal{E}_{\text{tar}}$, and $c_i$0 is the fitted model’s prediction (Li et al., 24 Apr 2026).

This formulation generalizes traditional scaling-law fitting (which is typically cost-agnostic and grid-oriented) into a constrained experimental design problem where the selection of which runs to perform is itself an optimization balancing information gain and resource cost.

2. Scaling Law Model Classes and Budget-Efficient Fitting Procedures

The parametric families $c_i$1 assumed in budget-efficient fitting typically mirror those validated by multi-decade empirical scaling studies:

• Power laws: $c_i$2
• Log-linear relations: $c_i$3
• Joint parameter–data laws: e.g., $c_i$4
• Multi-fidelity or mixture laws, for cases with multiple data fidelities or domain blends

Robust budget-efficient fitting leverages statistical procedures to maximize the value of each experiment:

• Warm start with low-cost experiments (cheapest $c_i$5 runs, $c_i$6 = # model parameters)
• Sequential active acquisition, using uncertainty-reduction metrics (detailed in §3)
• Mixture-of-basin posterior modeling for parametric uncertainty quantification
• Robust regression (e.g., Huber loss, multi-start L-BFGS, Gauss–Newton–Laplace for local posterior estimation)
• Batched acquisition with diversity criteria when parallel compute allows (Li et al., 24 Apr 2026, Choshen et al., 2024, Li et al., 12 Jun 2025)

Fitting typically includes bootstrapped uncertainty quantification and explicitly reports out-of-sample extrapolation errors (e.g., $c_i$7 over $c_i$8).

3. Active Experiment Selection via Uncertainty-Guided Acquisition

The key methodological advance is to treat the experiment allocation as a sequential decision process focused on reducing extrapolation uncertainty in the regions of greatest importance. Specifically, the approach:

• Maintains a mixture-of-basins posterior over scaling-law fits found via multi-start optimization
• Quantifies both intra-basin variance (local parameter uncertainty) and inter-basin variance (extrapolation disagreement among fit “modes”)
• For each candidate experiment $c_i$9, evaluates

$B$0

where $B$1 is estimated using the local linearized Fisher information, and $B$2 incorporates possible fit reweighting upon observing the outcome of $B$3 (Li et al., 24 Apr 2026).

The acquisition score is

$B$4

with $B$5 a modest cost-penalization (empirically $B$6). At each step, $B$7 is selected, run, and the process repeats until budget exhaustion. The method scales effectively to hundreds of candidates and has demonstrated near full-experiment extrapolation accuracy using only $B$810% of total cost (Table 1).

4. Empirical Validation and Baseline Comparisons

Benchmarks indicate that optimal budget-efficient experiment selection outperforms classical and heuristic approaches:

• Random, Cheapest-first, and Cost-weighted random methods prioritize cost minimization but ignore scientometric information value, often failing at tight budgets.
• Design-of-experiment criteria such as D-optimality (maximize $B$9) and V-optimality (var-reduction in prediction space) are competitive, but less robust when multimodal fits disagree in the high-cost region.
• All-data reference (full grid) provides an upper-bound but is typically infeasible under frontier compute budgets.
• The active, uncertainty-aware method consistently outperforms all heuristics at 1–5% budget and achieves parity with all-data reference by 10% budget (Li et al., 24 Apr 2026).

5. Extensions Across Domains and Scaling Law Regimes

Budget-efficient fitting methods have demonstrated applicability across diverse modeling scenarios:

• LLM pretraining and fine-tuning: Including power-law, Chinchilla, Farseer, and advanced joint forms for loss prediction under compute, token, and parameter constraints (Choshen et al., 2024, Li et al., 12 Jun 2025).
• Mixture-of-experts architectural scaling, where model capacity, FLOPs, and architectural shape (active vs. total parameters) must be jointly considered for budget-optimal designs (Wan et al., 23 Mar 2026).
• Data mixture optimization, where the budget-efficient selection of (mixture, scale) pairs is critical for downstream benchmark performance, with systematic sample allocation heuristics (e.g., "hourglass" allocation) providing provable improvements (Li et al., 9 Mar 2026).
• RL agent scaling laws and scientific ML with multi-fidelity data (e.g., CFD neural surrogates), where the optimal balance of low- and high-fidelity samples is solved analytically under the cost constraint (Setinek et al., 3 Nov 2025).
• Fine-tuning scaling under token and compute budgets with explicit modeling of performance as a function of dataset composition and model size. Active subsampling strategies (few_long, many_short, balanced) produce distinct best-fit exponents, offering fine control for practical scenarios (Lagasse et al., 9 May 2025).

6. Implementation Considerations and Best Practices

Efficient implementation requires careful attention to statistical modeling and computational engineering:

• Posterior representation should capture basin multiplicity and parameter correlation; multi-start local Gaussian approximations clustered in prediction space is the state-of-the-art.
• Acquisition step: Utilize efficient linear algebra (NumPy/SciPy) for Fisher/Hessian computation, and vectorized Jacobian evaluation for rapid scoring of all candidates.
• Hyperparameter tuning: The cost exponent $\sum_{e_i \in \mathcal{S}} c_i \leq B$0, warm-start pool size, and number of seed initializations $\sum_{e_i \in \mathcal{S}} c_i \leq B$1 for mixture modeling can be selected via fast ablations on proxy tasks.
• Batched execution: In high-throughput settings, batch top-k highest-score acquisitions and recompute posterior only every few steps to amortize fit/refinement cost.
• Code modularity: Separate posterior estimation, acquisition, experimental interface, and scaling-law formulations for maintainability.

7. Implications and Theoretical Significance

The emergence of budget-efficient scaling law fitting as an explicit experimental design and uncertainty quantification problem marks a methodological shift for resource-constrained research in deep learning:

• Scaling law experiments become optimal and adaptive, rather than fixed-grid, reducing pilot cost by an order of magnitude.
• Fitted laws and their uncertainties are more robust to overfitting, fit-misspecification, and multi-modal regression pathologies when using mixture-of-basin approaches.
• Accurate scaling law inference at small fractions of total budget enables efficient planning, improved scientific reproducibility, and democratization of LLM and related research to teams and organizations without exascale compute (Li et al., 24 Apr 2026).
• Methods are broadly compatible with classic and next-generation scaling forms (power-law, Chinchilla, Farseer, mixture, and data composition laws) and are extensible to multi-dimensional, combinatorial, and nonparametric regimes.

Budget-efficient scaling law fitting thus establishes a rigorous and highly scalable experimental framework for empirical scaling research across modern machine learning domains.