--- title: Compute-Optimal Model Size url: https://www.emergentmind.com/topics/compute-optimal-model-size type: topic --- # Compute-Optimal Model Size A compute-optimal model size is the parameter count \(N^*(C)\) that minimizes task loss (or maximizes downstream utility) subject to a fixed compute budget \(C\) (measured in floating-point operations, FLOP) for pre-training, fine-tuning, or inference. This concept is grounded in empirical and theoretical scaling laws for parameter–data tradeoffs in neural networks, and it underpins efficient resource allocation for large-scale models in natural language processing, computer vision, protein modeling, embedding specialization, deep reinforcement learning, and test-time scaling. ## 1. Canonical Compute-Optimal Scaling Laws Classical compute-optimal scaling arises from empirical power-law fits to the relationship between generalization loss \(L(N, D)\), model size \(N\), and data quantity \(D\), subject to a fixed-compute constraint \(C \approx \kappa N D\). The most widely used ansatz for language modeling loss is \[ L(N,D) \approx L_{\infty} + \frac{A}{N^{\alpha}} + \frac{B}{D^{\beta}} \] with constants \(\alpha, \beta > 0\) determined by grid sweeps over \(N\) and \(D\). The optimal allocation of compute is determined by Lagrangian minimization under this constraint, leading to the Chinchilla scaling law [2203.15556, 2406.19146, 2410.01243]: \[ N_{\rm opt}(C) \propto C^{\beta / (\alpha + \beta)}, \quad D_{\rm opt}(C) \propto C^{\alpha / (\alpha + \beta)} \] Empirical fits (e.g., Hoffmann et al., 2022) yield \(\alpha \approx 0.34\), \(\beta \approx 0.28\), so that \[ N_{\rm opt}(C) \propto C^{0.45}, \quad D_{\rm opt}(C) \propto C^{0.55} \] In practice, this yields an approximately equal allocation of compute to model size and data in contemporary large language models, with finer corrections explained by input dimensionality, data quality, and optimizer particulars [2410.01243, 2406.19146, 2212.01365, 2501.02068]. ## 2. Analytical Derivation and Corrections The closed-form solution to the compute-optimal trade-off is obtained via Lagrange multipliers: \[ \mathcal{L}(N, D, \lambda) = \frac{A}{N^{\alpha}} + \frac{B}{D^{\beta}} + \lambda(\kappa N D - C) \] Setting partial derivatives to zero yields \[ N^* = \left[\frac{\alpha A}{\beta B \kappa^{\beta} C^{-\beta}}\right]^{1/(\alpha+\beta)} \] with the scaling exponent \(\beta/(\alpha+\beta)\) [2410.01243, 2212.01365]. Recent corrections to the original Kaplan law (exponent \(\sim 0.88\))—accounting for the last-layer head cost, warmup proportional to \(N\), and optimal batch size and AdamW \(\beta_2\)—yield exponents (\(b\)) converging to the Chinchilla value of \(b = 0.50\) [2406.19146]. Practical formulas are \[ N^*_\text{opt}(C) \approx 0.091\,C^{0.50} \] across a wide range of budgets \(10^{17} \leq C \leq 10^{25}\) FLOPs. ## 3. Domain Adaptations and Empirical Extensions Domain specialization, data quality, and modality significantly modulate the optima and the scaling exponents: - **Domain-specialized LMs**: Lower data-quality constant \(B\) shifts \(N^*\) upward for fixed \(C\) [2501.02068]. Legal, medical, finance: as model size increases, specialization offers greater compute efficiency, e.g. the 14B-legal model attains its loss floor at \(4.3\times\) lower compute than a general-domain model. - **Protein language modeling**: CLM and MLM objectives on protein sequences yield exponents differing from NLP: e.g., for MLM, \(N^*\propto C^{0.776}\), \(D^*\propto C^{0.230}\) [2411.02142, 2406.07249]. The optimal allocation is more parameter-dominated for MLMs (smaller vocabulary, high compositional complexity). - **Vision Transformers**: For shape-optimized ViTs, the total parameter count scales as \(N(C)\propto C^{0.4}\), obtained by fitting error power laws to width, depth, and MLP-size, then combining exponents [2305.13035]. Optimal trade-offs are modality- and architecture-specific and require empirical calibration for each domain, but all obey a parametric frontier of the form \(N^*\propto C^{s}\). ## 4. Compute-Optimal Model Size in Fine-Tuning and Transfer Specialized tasks such as contrastive embedding model training exhibit their own compute-optimal laws, incorporating the spectrum of fine-tuning methods: - **Embedding models**: For given compute, optimal model size \(N^*\) depends on the choice of fine-tuning (full vs. partial/LoRA). Below a crossover (\(\sim 9.06\times 10^{16}\) FLOP), full fine-tuning is optimal; above, LoRA with rank 32–128 is preferred [2406.04165]. - The optimum \(N^*\) grows sublinearly with \(C\); at high budgets, LoRA and full fine-tuning yield almost identical optima, dictated by the task's empirical IsoFLOP curves. Bias-only fine-tuning is never optimal. Empirical recipe for text embedding models under a fixed \(C\): 1. For \(C \leq 9\times 10^{16}\) FLOP: full fine-tuning with largest feasible \(N\). 2. For \(C > 9\times 10^{16}\) FLOP: use LoRA with high rank, largest feasible \(N\). 3. Saturate \(C\) with \(D\) via \(6ND = C\) (full) or the appropriate fine-tuning cost formula (LoRA). ## 5. Compute-Optimal Model Size at Test-Time and End-to-End Budgets Classical scaling laws focus on pretraining, but modern deployments optimize for total (pretrain+inference) compute or inference-specific FLOPs: - **Train-to-Test (T²) Scaling**: Introduction of test-time sampling (pass@\(k\)), where inference cost scales as \(C_{\rm inf}=2Nk\), shifts the pretraining optimum. When accounting for fixed inference compute, the compute-optimal \(N^*, D^*, k^*\) jointly minimize loss or maximize accuracy: \[ N^* \propto C_{\rm train}^{\beta/Z}\,C_{\rm inf}^{\gamma/Z}, \quad D^* \propto C_{\rm train}^{\alpha/Z}\,C_{\rm inf}^{\gamma/Z}, \quad k^* \propto C_{\rm train}^{-\beta/Z}\,C_{\rm inf}^{(\alpha+\beta)/Z} \] with \(Z = \alpha+\beta+\gamma\), typical exponents \(\alpha \approx 0.7\), \(\beta \approx 0.3\), \(\gamma \approx 0.5\) [2604.01411]. - **Test-time scaling and inference**: Smaller, overtrained models, plus many test samples, outperform large models at fixed inference budget [2408.00724, 2502.06703, 2604.01411]. Empirically, \(N^*\propto C_{\rm inf}^{0.84}\) for inference-only scaling [2408.00724]. Empirical observations from T² and test-time scaling: as soon as inference compute is included, the optimal region shifts toward smaller models trained much longer (high tokens/parameter), supporting aggressive test-time sampling or search. | Paradigm | \(N^*(C)\) scaling | \(D^*(C)\) scaling | Key context | | ------------- | ---------------------| ----------------------|-----------------------------| | Chinchilla LM | \(C^{0.5}\) | \(C^{0.5}\) | Language modeling [2203.15556, 2406.19146] | | ViT | \(C^{0.4}\) | n/a (data unlimited) | Computer vision [2305.13035] | | pLM MLM | \(C^{0.776}\) | \(C^{0.230}\) | Protein masked LM [2411.02142] | | Inference-opt | \(C^{0.84}\) | - | LLM inference, test-time [2408.00724] | | T² end-to-end | \(C^{0.2\textrm{–}0.5}\) | see formulas | Train-to-test, many \(k\) [2604.01411] | ## 6. Unified Laws: Limits, Anomalies, and Practical Considerations Attempts to collapse model performance onto total compute, decoupling data and parameter allocation, lead to different "unified" scaling laws: - **Unified Law (Guo 2024)**: Some recent works fit BPC as a function of \(\log C\) only, with no unique optimum in (N, D); any allocation with \(ND=C\) yields identical compression [2404.19484]. Compute-optimal split then requires an external constraint (inference cost, data supply) to pick \(N^*\) (e.g., for inference-optimality, push \(N\) minimal, \(D = C/N\)). - These regimes highlight boundaries of the classic two-variable scaling law, with implications for exabyte-scale and web-limited data scenarios. **Practical guidelines and caveats**: - Proper accounting of "head" parameters and scale-dependent learning rate/batch (\(\beta_2\)) tuning are essential for correct exponents [2406.19146]. - For domain-specialized settings with high-quality data, exponents shift to favor larger \(N\) at fixed \(C\) [2501.02068]. - Under strong test-time constraints or aggressive sampling/search protocols, optimal \(N^*\) can be order(s) of magnitude smaller than pure pretraining scaling suggests, with significant overtraining (tokens per parameter many times higher than canonical ratios) [2604.01411]. ## 7. Cross-Domain Generalization and Future Directions Compute-optimal scaling laws have been extended to a range of architectures (ViT, RL), domains (protein, law, medicine), fine-tuning protocols, and both training and inference regimes, as summarized in the literature above. Ongoing and open topics: - Understanding exponents' dependence on data complexity and modality (e.g., protein MLM vs. CLM vs. text). - Joint optimization of pretraining and deployment (test-time search, sample-efficient inference). - Constant-factor improvements via optimizer/batch/hyperparameter scaling. - Information-theoretic characterizations adding \(1/\sqrt{C}\) corrections and sharp thresholds based on semantic graph analogies [2410.01243]. The resulting body of work provides practitioners and theorists with explicit, empirically validated recipes for determining \(N^*(C)\) under a wide spectrum of practical constraints, enabling rigorous design of compute-scaled models across the spectrum of modern machine learning systems. Source: https://www.emergentmind.com/topics/compute-optimal-model-size