Compute-Optimal Model and Data Allocation

• Compute-optimal model and data allocation is a framework that optimally divides a finite compute budget between model parameters and training data to maximize performance.
• It compares foundational laws like Chinchilla’s proportional scaling with unified log-linear models that allow flexible splits under fixed compute constraints.
• The article also explores applications in inference, distributed computing, reinforcement learning, and wireless systems, with empirical validations and task-specific adaptations.

Compute-optimal model and data allocation refers to the principled division of a finite computational budget between a model's size (number of parameters, $N$) and the amount of data used for training (number of tokens, $D$) or inference, in order to maximize empirical or theoretical performance. Modern research has extended this concept beyond standard pretraining to data selection, adaptive model shaping, distributed computing, RL, inference-time allocation, and resource-constrained wireless systems. This article systematically surveys core laws, methodologies, and practical rules for compute-optimal allocation in both foundational and emerging scenarios.

1. Foundational Laws: From Chinchilla to One-Dimensional Scaling

Traditional approaches to compute-optimal allocation in transformer-based language modeling posit that, for a fixed compute budget $C$, both model size $N$ and dataset size $D$ should be increased in lock-step. The classic “Chinchilla rule” is derived from empirical two-dimensional power-law fits of the form

$$L(N, D) \approx A N^{-\alpha} + B D^{-\beta} + L_\infty$$

subject to $C \sim N D$ FLOPs. Minimizing loss under this constraint yields

$$N^* \propto C^{\beta/(\alpha+\beta)}\,,\quad D^* \propto C^{\alpha/(\alpha+\beta)}$$

For LLMs, exponents are generally $\alpha\approx\beta\approx0.5$, leading to $N^*\sim D^*\sim C^{1/2}$ and $D$0, requiring proportional doubling of $D$1 and $D$2 as $D$3 increases (Hoffmann et al., 2022).

However, "More Compute Is What You Need" (Guo, 2024) provides strong empirical evidence for a single-parameter scaling law: $D$4 with $D$5. Here, model compression (bits-per-character, BPC) depends almost exclusively on total compute $D$6, not the split between $D$7 and $D$8. Any $D$9 satisfying $C$0 achieves the same final compression. This unified law implies no intrinsic trade-off: for fixed $C$1, practitioners are free to select any $C$2 according to inference constraints.

2. Compute Allocation Strategies, Data Regimes, and Inference Efficiency

The unified log-linear law (Guo, 2024) motivates two distinct regimes:

• Data-rich regime: When abundant high-quality web data is available ($C$3), minimizing $C$4 reduces inference cost while maximizing $C$5 absorbs the remaining compute. Thus, optimal allocation is achieved by using the smallest $C$6 compatible with latency/throughput requirements, dedicating maximum possible compute to more data.
• Data-exhaustion regime: Once $C$7 saturates available web-scale data (typically $C$8 T tokens), only increasing $C$9 (and thus $N$0) yields further gains.

Crucially, inference cost scales poorly with $N$1, but $N$2 impacts only training. The result is that, for deployment-oriented pretraining, shifting compute towards larger $N$3 allows both optimal loss and improved inference efficiency. In contrast, the Chinchilla law ($N$4) constrains $N$5 to a constant and cannot fully leverage this regime without violating the isocompute constraint (Hoffmann et al., 2022, Guo, 2024).

3. Derivation and Empirical Validation

The unified log-linear law arises from regression of BPC on $N$6, fit to open-source transformer models (e.g., Llama 1/2/3, Falcon, Qwen, DeepSeek, Yi, Mistral). Models such as Llama 3, trained with $N$7 more data than Chinchilla-recommended, nevertheless lie perfectly on the same log-linear trend, refuting any requirement for special balancing. Outliers—models trained with extreme $N$8 ratios—do not systematically deviate from the fit, supporting the thesis that $N$9 alone governs final compression (Guo, 2024).

4. Comparative Perspective: Power-Law Versus Log-Linear Scaling

Allocation Law	C-Dependence	D/N Constraint	Optimum Structure
Chinchilla (Power-Law)	$D$0	$D$1	Equal scaling: $D$2 and $D$3 both increase as $D$4
Unified (Log-Linear) (Guo, 2024)	$D$5	$D$6 unconstrained	Any $D$7 with $D$8, allowing arbitrary splits

Unlike Chinchilla and other two-term power-law models (Hoffmann et al., 2022, Nayak et al., 2024), the log-linear law empirically dominates for current generation transformers in practical compute ranges. However, it remains to be tested in extreme asymptotic settings.

5. Skills, Validation, and Sensitivity to Task

Skill-specific analysis demonstrates that optimal allocation is sensitive to downstream evaluation mix. Empirical scaling exponents differ sharply between, e.g., knowledge-based QA (capacity-hungry, $D$9) and code generation (data-hungry, $$L(N, D) \approx A N^{-\alpha} + B D^{-\beta} + L_\infty$$0), with up to 50% swing in optimal $$L(N, D) \approx A N^{-\alpha} + B D^{-\beta} + L_\infty$$1 depending on validation set composition (Roberts et al., 13 Mar 2025). Thus, correct allocation must be informed by application-specific priorities, and practitioners must align validation metrics accordingly. If the target is "satisficing" across heterogeneous skills, mixing pretraining datamixes (e.g., $$L(N, D) \approx A N^{-\alpha} + B D^{-\beta} + L_\infty$$2 code:knowledge) aligns optimal $$L(N, D) \approx A N^{-\alpha} + B D^{-\beta} + L_\infty$$3 for critical subskills.

6. Compute-Optimal Allocation in Broader Contexts

Beyond language modeling, compute-optimal allocation extends to diverse settings:

• Data selection under compute constraints: The joint minimization of data-selection and training compute reveals that, at small budgets, lexically or embedding-based scoring dominates, while computationally heavy selection (perplexity or gradient) is only optimal when the training model is much larger (at least $$L(N, D) \approx A N^{-\alpha} + B D^{-\beta} + L_\infty$$4 or $$L(N, D) \approx A N^{-\alpha} + B D^{-\beta} + L_\infty$$5) than the scorer (Yin et al., 2024). Compute should be prioritized for cheaper selection and more tokens until this threshold is crossed.
• Inference compute allocation: OSCA algorithmically divides a fixed sample budget across a set of LLM inference configurations (model, temperature, language, etc.) using a convex program, leading to order-of-magnitude compute savings over best single configurations (Zhang et al., 2024).
• RL: Compute-optimal scaling in value-based deep RL involves allocating compute among model size, update-to-data ratio, and batch size. The empirical trade-off is governed by phenomenon such as TD-overfitting; optimal allocation power-laws are derived for Pareto-efficient sample and compute use (Fu et al., 20 Aug 2025).
• Distributed computing: Universal, deterministic allocation of data and subfunction loads (as a $$L(N, D) \approx A N^{-\alpha} + B D^{-\beta} + L_\infty$$6-uniform hypergraph edge partitioning) achieves order-optimal communication and computation scaling, e.g., $$L(N, D) \approx A N^{-\alpha} + B D^{-\beta} + L_\infty$$7 communication cost and bounded load imbalance, via interweaved clique design (Maheri et al., 9 Jan 2026).
• Wireless/federated resource allocation: In edge scenarios, joint model adapter compression and local compute allocation is solved via fractional programming and KKT updates under system/energy/delay constraints (Wang et al., 2024). Dinkelbach’s method yields superlinear convergence to the utility/consumption optimum.

7. Caveats, Assumptions, and Limitations

The unified computational scaling law (Guo, 2024) is strictly empirical for transformer-based autoregressive LLMs. Deviations may occur:

• For architectures other than transformers, or with task-specific objective functions outside next-token prediction.
• At extreme values of $$L(N, D) \approx A N^{-\alpha} + B D^{-\beta} + L_\infty$$8 or $$L(N, D) \approx A N^{-\alpha} + B D^{-\beta} + L_\infty$$9, where the log-linear law has not been tested.
• When data quality or heterogeneity is significant: higher-quality data can shift optimal allocations (as shown by DeepSeek, 2024).
• BPC is not a comprehensive measure of downstream or emergent abilities.
• The $C \sim N D$0-based compute proxy omits model depth, sequence length, optimizer and pipeline parallelization factors.

Regardless, for practical compute budgets and modern open-source LMs, allocating compute primarily based on the total product $C \sim N D$1, guided by inference constraints and data availability, is the new empirical optimum. The previous practice of fixed $C \sim N D$2 balancing is now superseded except under data exhaustion or specialized skill-optimized scenarios.

---

References:

• "More Compute Is What You Need" (Guo, 2024)
• "Training Compute-Optimal LLMs" (Hoffmann et al., 2022)
• "An Information Theory of Compute-Optimal Size Scaling, Emergence, and Plateaus in LLMs" (Nayak et al., 2024)
• "Compute-Constrained Data Selection" (Yin et al., 2024)
• "Scaling LLM Inference with Optimized Sample Compute Allocation" (Zhang et al., 2024)
• "Compute Optimal Scaling of Skills: Knowledge vs Reasoning" (Roberts et al., 13 Mar 2025)
• "Compute-Optimal Scaling for Value-Based Deep RL" (Fu et al., 20 Aug 2025)
• "Universal and Asymptotically Optimal Data and Task Allocation in Distributed Computing" (Maheri et al., 9 Jan 2026)
• "Resource Allocation and Secure Wireless Communication in the Large Model-based Mobile Edge Computing System" (Wang et al., 2024)