---
title: Scaling Laws for Fine-Tuning
url: https://www.emergentmind.com/topics/scaling-laws-for-fine-tuning
type: topic
---

# Scaling Laws for Fine-Tuning

Scaling laws for fine-tuning describe the predictable mathematical relationships that govern how neural model performance on downstream tasks improves with increasing model size, fine-tuning data volume, compute allocation, and—in multi-domain or transfer scenarios—the composition and alignment of fine-tuning datasets. Unlike scaling laws for pre-training, which typically focus on model and dataset size, fine-tuning scaling laws must explicitly account for the pre-trained foundation, data transfer dynamics, phase transitions in improvement, domain mixture composition, catastrophic forgetting, data poisoning susceptibility, complexity-driven task bottlenecks, and practical resource constraints. Theoretical and empirical studies across a diverse landscape of neural architectures (transformers, image models, speech models, mixture-of-experts, etc.) consistently reveal that fine-tuning improvements usually adhere to power-law or rectified power-law forms, but these can be modulated or even limited by factors distinct from those present during pre-training.

## 1. Mathematical Forms of Scaling Laws in Fine-Tuning

Fine-tuning scaling laws extend and generalize the classical power-law forms derived from pre-training. For generative and discriminative models across image, language, multimodal, and mathematical domains, loss $L$ is universally expressed as a sum of an irreducible term (usually data entropy or inherent error) plus a reducible power-law term dependent on model size $N$, data size $D$, or compute $C$:
$$
L(x) = L_{∞} + (x_0/x)^{\alpha_x}
$$
where $x$ stands for $N$, $C$, or $D$, $L_{∞}$ quantifies the entropy ("irreducible loss"), and $\alpha_x$ is a domain- and task-dependent scaling exponent [2010.14701].

For transfer and fine-tuning, the scaling law introduces "effective data transferred":
$$
D_t = k(D_F)^{\alpha} N^{\beta}
$$
where $D_F$ is the fine-tuning (target) dataset size, $N$ is the model parameter count, and exponents $\alpha$ and $\beta$ capture task alignment and model generality [2102.01293]. The observed loss is then largely determined by $D_F + D_t$ for the data regime of interest, shifting the data requirement lower for larger models and closer pre-training alignment.

Recent works introduce **rectified scaling laws for fine-tuning**, incorporating a "pre-learned data size" $D_l$ from the pre-trained model and capturing phase transitions:
$$
L(D) = \frac{B}{D_l + D^{\beta}} + E
$$
This form explains the transition from a slow "pre-power" phase (dominated by $D_l$) to a classic power-law regime (when $D \gg D_l$), ensuring accurate prediction even when only small fine-tuning datasets are available [2402.02314].

In multitask or transfer settings:
$$
L(p, f) = (A \cdot p^{-α} + G) \cdot f^{-β} + E
$$
where $p$ is pre-training steps/tokens, $f$ is fine-tuning dataset size, and $G$ is the **transfer gap**—the residual loss due to domain mismatch, which sets a lower bound on transfer efficiency [2408.16947]. More general forms also incorporate domain mixture weights $h$ as
$$
L(N, D, h) = E + \frac{1}{\sum_{i=1}^k C_i h_i^{γ_i} + A/N^{\alpha} + B/D^{\beta}}
$$
enabling prediction of optimal training mixtures [2507.09404].

## 2. Transfer, Effective Data, and Domain Alignment

A critical insight is that the benefit from pre-training is quantifiable as "effective data transferred", governed by a power law in both model size $N$ and fine-tuning data $D_F$:
$$
D_t = k\, D_F^{\alpha} N^{\beta}
$$
with $D_t$ dominating in the low-data regime. The exponents $\alpha$ and $\beta$ directly reflect the degree of **distributional alignment** and model generality. As $\alpha$ decreases (closer pre-training and fine-tuning distributions), pre-training provides more powerful data-multiplicative benefits. Scaling laws thus formalize the notion that larger and better aligned models "need less" task-specific fine-tuning data for comparable performance [2102.01293, 2408.16947].

However, **transfer gaps** $G$ can impose fundamental limits in the form of irreducible downstream loss, even with unlimited pre-training. When pre-training and target tasks are misaligned, scaling up pre-training yields diminishing returns, and fine-tuning data collection becomes comparatively more critical [2408.16947, 2402.04177]. This is particularly well-illustrated in machine translation scaling, where only models pretrained on distributionally matching languages exhibit monotonic and predictable BLEU score improvements [2402.04177].

## 3. Multidimensional Scaling: Compute, Data, Model Size, and Mixture Composition

Scaling laws provide practical guidance for optimizing model and training design under resource constraints. The **compute-optimal allocation** for training, given fixed total compute $C$, satisfies
$$
N_{\text{opt}} \propto C^\beta \quad \text{with} \quad \beta \approx 0.7
$$
Sublinear dataset scaling with model size ($D \sim N^{0.4}$) is thus optimal in this regime [2010.14701]. For fine-tuning, data composition—not just total tokens—is crucial: accounting for the number of examples $N$ and their mean length $L$ yields a dataset volume $V = N \times L$ whose contributions to downstream performance differ for the same total tokens under various sampling strategies [2505.06150].

In **Mixture-of-Experts (MoE)** and multitask models, scaling laws incorporate new architectural hyperparameters such as granularity $G$:
$$
L(N, D, G) = c + [(g/G^\gamma) + a]/N^\alpha + b/D^\beta
$$
Fine-grained tuning (raising $G$) often yields more efficient adaptation than traditional expert configurations [2402.07871].

Scaling laws for **optimal data mixtures** enable analytic determination of the best domain weights $h$ for any target task and compute budget, moving beyond trial-and-error for pretraining and fine-tuning mixtures [2507.09404].

## 4. Empirical Findings: Phase Transitions, Forgetting, Data Poisoning, and Complexity

Empirical studies reveal that fine-tuning scaling laws are not universally monotonic or linear. Notable phenomena include:

- **Pre-power and Power Phases**: Rectified scaling laws account for an initial regime where improvements are slow (controlled by pre-learned data size), transitioning to rapid power-law scaling as more fine-tuning data is added. This two-phase behavior is pronounced in low-data fine-tuning settings [2402.02314, 2502.12051].
- **Forgetting and Catastrophic Interference**: Fine-tuning large language models—especially with parameter-efficient methods (e.g., LoRA)—produces a **strong, inverse linear relationship between downstream loss and forgetting** of pretrained capabilities (knowledge, reasoning, safety), regardless of the number of fresh parameters trained. Both forgetting and fine-tuning loss scale as shifted power laws with respect to parameter count and update steps, quantifying "catastrophic forgetting" as an intrinsic scaling phenomenon [2401.05605].
- **Scaling Laws and Data Poisoning**: As model size increases, **vulnerability to data poisoning and jailbreak tuning scales upward**, with larger models learning harmful behaviors from minimal poisoned data at a faster rate. Regression analyses establish a positive scaling relationship between log-parameters and post-attack harmfulness scores, even at very low poisoning rates [2408.02946].
- **Complexity-Driven Limits**: Scaling laws can be governed by task-intrinsic complexity, as shown in combinatorial optimization (e.g., Traveling Salesman Problem). Fixed-capacity models exhibit superlinear increases in suboptimality with respect to problem size and complexity, revealing predictable, irreducible performance gaps that cannot be overcome by fine-tuning alone when the problem's solution or representation space scales "too fast" [2506.12932].

## 5. Domain-Specific Scaling Laws and Practical Implications

Scaling law forms and exponents differ across domains:

- In **contrastive (CLIP) and multimodal models**, zero-shot accuracy and retrieval scale as clean power-laws in compute and model size, but the actual effect sizes and scaling trends strongly depend on the particular data distribution and the alignment between source and target domains [2212.07143].
- In **speech recognition** (RescoreBERT), normalized word error rates improve as a joint power-law in both fine-tuning data and model size for pre-trained models, but only as a function of data size for non-pretrained models—quantifying the data-multiplicative effects of transfer [2306.15815].
- For **synthetic data**, scaling laws track those observed for real data, but improvements plateau earlier, and the number of tokens required to saturate performance is lower for large models [2503.19551].
- In **multi-task power system modeling**, scenario generalization follows a power law with data size, is largely insensitive to strong scaling of parameter count, and remains robust in multi-task environments—demonstrating predictability for practical limits of fine-tuning in highly structured, domain-specific applications [2503.20040].

Table: Prototypical Fine-Tuning Scaling Laws Across Domains

| Domain/Setting                  | Scaling Law Formulation                                                    | Notes/Interpretation                                                  |
|---------------------------------|---------------------------------------------------------------------------|-----------------------------------------------------------------------|
| Generic generative/fine-tuning  | $L(x) = L_{\infty} + (x_0 / x)^{\alpha_x}$                                | Loss approaches irreducible entropy; $\alpha_x$: scaling exponent     |
| Data transfer (low-data)        | $D_t = k D_F^{\alpha} N^{\beta}$                                          | Effective data from pretraining; $\alpha$: proximity, $\beta$: generality |
| Rectified fine-tuning           | $L(D) = B / (D_l + D^{\beta}) + E$                                        | Accounts for pre-power phase transition and pre-learned data          |
| Mixture-of-Experts              | $L(N, D, G) = c + [g/G^{\gamma} + a]/N^{\alpha} + b/D^{\beta}$            | Efficiency scales with granularity                                    |
| Transfer with gap               | $L(p, f) = (A p^{- \alpha} + G) f^{-\beta} + E$                           | $G$: transfer gap sets lower bound                                    |
| Data mixture optimization       | $L(N, D, h) = E + \frac{1}{\sum C_i h_i^{\gamma_i} + A/N^{\alpha} + B/D^{\beta}}$ | Mixture- and scale-optimal fine-tuning/pretraining                    |

## 6. Predictive Use, Model Selection, and Limitations

Scaling laws for fine-tuning enable:

- **Performance Forecasting**: From small-scale pilot runs, extrapolation using established scaling exponents can accurately predict large-scale model performance in both language and vision tasks, provided power-law fits are robust (goodness-of-fit $R^2 \gg 0.9$) [2202.06387].
- **Model/Data Trade-offs**: Law–backed analytic frameworks allow prediction of how to optimally allocate resources between increasing model size, expanding dataset size, collecting better-aligned data, or modifying domain mixtures to achieve a desired downstream loss under a fixed budget [2010.14701, 2507.09404].
- **Selection Algorithms**: The Accept-then-Stop (AtS) algorithm demonstrates that nature of the power-law and pre-power phases enables informed model selection (choose the candidate with the lowest extrapolated full-dataset loss) with orders-of-magnitude lower resource consumption [2402.02314].
- **Caveats and Deviations**: Not all domains, architectures, or objectives exhibit perfect adherence to simple scaling law forms. Phase transitions, irreducible domain gaps, catastrophic forgetting, compositionality bottlenecks, and problem complexity saturation are persistent challenges—suggesting that practitioners must calibrate scaling laws locally and monitor phase transitions during fine-tuning and deployment [2502.12051].

## 7. Data Quality, Annotation, and Ethical Considerations

Data quality and composition have scaling effects beyond simple size. For low-resource environments, the scaling law can be used as a yardstick for annotation quality: if model performance does not increase with model size for a given dataset, annotation revisions may be required [2405.02817]. The use of scaling law trends as both an annotation diagnostic and fine-tuning target metric provides a principled approach for robust dataset creation, especially under constraints of privacy, funding, and compute.

Fine-tuning scaling laws also highlight critical ethical risks, notably increased susceptibility to harmful behavior acquisition and safety degradation as models scale in size [2401.05605, 2408.02946], necessitating rigorous data curation, red-teaming, and safety benchmarking throughout large-scale fine-tuning.

---

In summary, scaling laws for fine-tuning provide a quantitative theoretical and empirical foundation for predicting, optimizing, and understanding the adaptation of large neural models to downstream tasks. The functional forms of these laws, along with their domain- and data-specific exponents, support principled model/data budgeting, optimal mixture selection, safety evaluation, and resource-efficient model selection. At the same time, deviations arising from forgetting, data quality issues, and complexity bottlenecks illustrate the necessity of adaptive, empirically validated fine-tuning protocols as model scales and application landscape continue to evolve.

Source: https://www.emergentmind.com/topics/scaling-laws-for-fine-tuning