---
title: Compute-Optimal Dataset Sizes
url: https://www.emergentmind.com/topics/compute-optimal-dataset-sizes
type: topic
---

# Compute-Optimal Dataset Sizes

Compute-optimal dataset sizes refer to the principled determination of training data volume that, jointly with model size, maximizes model quality for a fixed computational budget. These allocation laws, along with their associated workflow and practical recipes, are foundational for efficient large-scale training of language and protein models. They have recently been extensively revised, unified, and critiqued, with both empirical and information-theoretic approaches now in close agreement across modalities and tasks.

## 1. Basic Principles: The Compute-Budget Constrained Frontier

At the core of modern pretraining is the constraint that total computational cost (typically measured in floating-point operations, FLOPs) is first-order proportional to the product of model size (number of parameters, $N$) and dataset size (number of tokens or examples, $D$):

\[
C \propto N \times D
\]

Given a compute budget $C$, one seeks $(N^*, D^*)$ minimizing the final loss (proxy: bits-per-character, negative log-likelihood), such that $N D = C$. In classic scaling laws, one models loss surfaces as additive power laws:

\[
L(N, D) = A N^{-\alpha} + B D^{-\beta}
\]

and derives the efficient frontier by balancing the marginal benefits of parameter and data scaling. Optimizing under $C$ yields closed-form allocations for $N^*(C)$ and $D^*(C)$ [2203.15556][2304.03208][2410.16208].

## 2. The Chinchilla Law and Empirical Scaling in LLMs

The most influential empirical scaling law is the Chinchilla law of Hoffmann et al., validated by the Cerebras-GPT and Porian et al. studies [2203.15556][2304.03208][2406.19146]:

- **Optimal scaling:** Empirically, $N^*(C) \propto C^{0.5}$ and $D^*(C) \propto C^{0.5}$.
- **Constant “tokens-per-parameter” ratio:** The optimal frontier is achieved at $D^*/N^* \approx 20$ for the MassiveText and Pile datasets.
- **Explicit rule:**
  \[
  N^* = \sqrt{C/r},\quad D^* = r N^*
  \]
  where $r \approx 20$.

The optimal pair $(N^*, D^*)$ is robust to dataset type, optimizer, parametrization (μP), and minor implementation choices, provided total FLOPs are precisely counted (including final-layer FLOPs), and batch and learning rate scaling are properly tuned [2406.19146][2304.03208].

### Table 1: Empirical Chinchilla Law Parameters

| Source                | Exponent on $C$ ($N^*$) | Exponent on $C$ ($D^*$) | $D^*/N^*$ (tokens/param) |
|-----------------------|-------------------------|-------------------------|--------------------------|
| Hoffmann et al., 2022 |       0.5               |         0.5             |          20              |
| Porian et al., 2024   |      0.498              |        0.498            |          21              |
| Cerebras-GPT, 2023    |       0.5               |         0.5             |          20              |


Under these laws, increasing $N$ without proportional data scaling leads to overfitting and suboptimal loss; scaling $D$ without growing $N$ produces diminishing returns.

## 3. Unified Scaling Laws and Degeneracy

“More Compute Is What You Need” [2404.19484] reports that for transformer LLMs, model performance depends primarily on total computation $C \approx N D$, **independent** of the specific split between parameters and tokens. The main empirical fit,

\[
\mathrm{BPC}(N,D) = \alpha\log(ND) + \beta
\]

with $\alpha = -0.031$, $\beta = 0.572$, and $R^2 \sim 0.95$ (across 20+ models), indicates that all pairs $(N,D)$ with $N D = C$ achieve the same loss. This produces a **degenerate optimum**—any aspect ratio $\lambda$ in $(N, D) = (\lambda C^{1/2}, C^{1/2}/\lambda)$ attains identical predicted performance.

**Practical implication:** Optimal $(N, D)$ allocation should reflect secondary desiderata:
- Small $N$, large $D$: preferable for inference efficiency and latency.
- Large $N$, small $D$: required if high-quality data is exhausted.

Under data exhaustion, only further model scaling improves performance [2404.19484].

## 4. Domain-Specific Scaling: Protein LMs and Skill-Dependence

### Protein Language Models

Empirical studies in protein language modeling [2406.07249][2411.02142] show qualitatively different exponents from text LLMs:

- **Sublinear model scaling:** $N^*(C) \propto C^{0.27}$, $D^*(C) \propto C^{0.71}$.
- **Plateau regime:** For encoder-only pLMs, once a single epoch of unique tokens is seen, further data contributes negligible gains.
- **Loss-scaling for CLM/MLM:** Closed forms
  \[
  T^*_{\rm CLM}(C,N) \propto C^{0.58} N^{-0.42},\quad T^*_{\rm MLM}(C,N) \propto C^{0.75} N^{-0.25}
  \]
  indicate that for masked LMs optimal data sizes may exceed realistically available unique tokens.

### Skill-Dependent Scaling Laws

Recent work demonstrates that **compute-optimal dataset size is skill dependent** [2503.10061]. For example, knowledge-based QA and code generation have optimal $D^*(C)$ scaling exponents of $0.61$ and $0.66$, respectively (from $N^*(C)\propto C^{0.39}$, $C^{0.34}$). This reflects that code and reasoning evaluation rewards larger datasets relative to model size, while knowledge QA is relatively more capacity-bound.

## 5. Information-Theoretic and Random Graph Foundations

Probabilistic and information-theoretic analyses [2212.01365][2410.01243] reproduce the observed scaling laws:

- For broad neural architectures, minimax bounds on cross-entropy error derive asymptotically linear data-to-parameter scaling: $N^* \propto k\,M^*$, $k \sim \ln C$ growing slowly in compute.
- In semantic graph-based formulations, compute-optimal scaling arises from LDPC iterative decoding and matching the critical threshold for coverage in a bipartite “concept–text” network, yielding $N^*, D^* \sim C^{1/2}$ [2410.01243].

These theories predict emergent phenomena—such as performance plateaus and sharp appearance of new skills—arising when giant components (“skills”) percolate in the random graph as $C$ increases.

## 6. Compute-Optimal Data Curation and Finetuning

When the data collection process involves significant computational cost, as in expensive data selection or synthetic data generation, compute-optimal strategies must account for both selection and training cost [2408.16737][2410.16208]:

- For synthetic data, repeated sampling from a weaker, cheaper generator (lower per-example FLOPs) yields greater coverage/diversity and better downstream scores for fixed budget, provided false positive rate is controlled. The transition to sampling from a strong generator is only favorably crossed when these benefits are demonstrably saturated.
- In data selection for finetuning, the optimal dataset size is $N^*_m = C_{\rm total}/(c^{(\rm train)} + c^{(\rm sel)}_m)$, where $c^{(\rm sel)}_m$ is the per-token selection cost of method $m$. Simple methods (BM25, Embed) are almost always compute-optimal at moderate budgets; high-cost selectors (perplexity, gradients) only become worthwhile when the train-to-selector model size ratio is $\gtrsim5\times$ or $10\times$.

### Table 2: Compute-Optimal Data Generation Strategies

| Setting                    | Optimal Allocation                              | Condition                               |
|----------------------------|-----------------------------------------------|-----------------------------------------|
| Synthetic data (weak/strong)| All budget to weaker generator (WC)           | Coverage/diversity WC$>$SE for rescaled $k$ |
| Data selection (finetuning) | Max tokens with lowest-cost selector           | High-cost methods only optimal at scale |

## 7. Practical Recommendations and Caveats

- For LLMs: The fixed-ratio Chinchilla law ($D/N\approx20$) holds robustly for pretraining, provided compute is accurately accounted and hyperparameters are tailored per scale [2304.03208][2406.19146].
- For specialized domains (e.g., protein LMs): Exponents may differ; do not blindly import NLP scaling laws, but measure domain-specific loss surfaces and account for loss plateaus and data limits [2406.07249][2411.02142].
- Skill composition: Compute-optimal dataset/parameter split is task-dependent. When optimizing for multiple skills, composite validation sets must represent real-world priorities; otherwise, the optimum may shift by up to $50\%$ in parameter count due to validation set misspecification [2503.10061].
- Data quality and single-epoch limits: In regimes where unique data is exhausted, further gains require larger $N$ and increased compute, with little benefit from repeating tokens.
- Extrapolation: Scaling laws are empirical and generally validated up to scales of $C\sim10^{23}$ FLOPs; extrapolation beyond this, or to new domains/architectures, should be approached with caution.

---

**References**:
- [2203.15556] Training Compute-Optimal Large Language Models
- [2304.03208] Cerebras-GPT: Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Cluster
- [2406.19146] Resolving Discrepancies in Compute-Optimal Scaling of Language Models
- [2404.19484] More Compute Is What You Need
- [2212.01365] An Information-Theoretic Analysis of Compute-Optimal Neural Scaling Laws
- [2410.01243] An Information Theory of Compute-Optimal Size Scaling, Emergence, and Plateaus in Language Models
- [2406.07249] Are Protein Language Models Compute Optimal?
- [2411.02142] Training Compute-Optimal Protein Language Models
- [2410.16208] Compute-Constrained Data Selection
- [2503.10061] Compute Optimal Scaling of Skills: Knowledge vs Reasoning
- [2408.16737] Smaller, Weaker, Yet Better: Training LLM Reasoners via Compute-Optimal Sampling

Source: https://www.emergentmind.com/topics/compute-optimal-dataset-sizes