Compute-Optimal Design: Principles & Applications

• Compute-optimal design is a framework for efficiently allocating computational resources in experiments and ML training to maximize predictive and statistical efficiency.
• It leverages theoretical scaling laws and algorithmic approaches, including multiplicative updates, cutting-plane methods, and gradient flows, to optimize design under constrained compute budgets.
• Practical recommendations involve empirically extracting scaling exponents and using advanced solvers and parallelism to ensure efficient resource allocation in diverse domains like deep learning and reinforcement learning.

Compute-optimal design refers to the principled allocation of computational resources—either in experimental design or machine learning model training—with the explicit objective of maximizing performance per unit compute. It encompasses theoretical prescriptions, scaling laws, and algorithmic approaches for selecting design points, model sizes, or hyperparameters such that, under fixed or constrained budgets of compute (often measured in FLOPs for machine learning, or number of model evaluations for experimental design), the resulting statistical or predictive efficiency is maximized. The concept applies across classical optimal experimental design, deep learning, and modern large-scale AI models, each with distinct mathematical frameworks and practical methodologies.

1. Mathematical Foundations and Classical Optimal Design

Compute-optimal design originated in the context of optimal experimental design, where the aim is to select a set of experimental conditions (design points and weights) to optimize an information-theoretic or risk-based criterion (e.g., D-optimality, A-optimality), typically under constraints on sample size or cost.

Given a finite candidate set $X = \{x_1, \ldots, x_N\} \subset \mathbb{R}^d$ and regression vectors $f(x_i) \in \mathbb{R}^p$, an approximate design consists of nonnegative weights $w = (w_1, \ldots, w_N)$ with $\sum_i w_i = 1$. The information matrix is $M(w) = \sum_{i=1}^N w_i f(x_i) f(x_i)^\top$.

• D-optimality: maximize $\log \det M(w)$ (minimize the generalized volume of confidence ellipsoid for linear parameter inference).
• A-optimality: minimize $\operatorname{tr} (M(w)^{-1})$ (minimize mean variance of parameter estimates).

Efficient computational algorithms for approximate D- and A-optimal designs involve multiplicative weight updates based on sensitivity functions. For D-optimality, the update is $$w_i^{(h)} \leftarrow w_i^{(h-1)} \cdot (D_i^{(h-1)}/p)$$ where $D_i^{(h-1)} = f(x_i)^\top [M^{(h-1)}]^{-1} f(x_i)$. Monotonic convergence is guaranteed for D-optimality, and proven rates to the continuous optimum exist under high-resolution discretizations (Duan et al., 2021).

This framework extends to continuous designs, nonlinear models, T-optimality (model discrimination), c-optimal, compound Bayes-risk, and criterion-robust design problems—each with specific optimization structures (semi-infinite programming, cutting-plane methods, mixed-integer linear or conic programming, etc.) (Duan et al., 2018, Burclova et al., 2015, Harman et al., 2017, Harman et al., 2023, Sagnol et al., 2013, Mogalle et al., 2022).

2. Compute-Optimal Allocation in Modern AI Training

The concept of compute-optimal design generalizes beyond experimental design to the scaling of deep neural models, where one seeks to optimally allocate total compute $C$ between parameter count $f(x_i) \in \mathbb{R}^p$0 and data size $f(x_i) \in \mathbb{R}^p$1. The joint tradeoff empirically obeys a two-term scaling law for loss (or perplexity):

$f(x_i) \in \mathbb{R}^p$2

Given a fixed compute budget $f(x_i) \in \mathbb{R}^p$3, the optimal allocation, derived by Lagrangian optimization, yields

$f(x_i) \in \mathbb{R}^p$4

This scaling law has been empirically validated for masked diffusion LLMs, vision transformers, and protein LLMs, among others (Chao et al., 17 Mar 2026, Alabdulmohsin et al., 2023, Serrano et al., 2024). Notably, the exponents $f(x_i) \in \mathbb{R}^p$5 are extracted from regression over joint sweeps in $f(x_i) \in \mathbb{R}^p$6 and $f(x_i) \in \mathbb{R}^p$7, and $f(x_i) \in \mathbb{R}^p$8 captures compute per forward-backward pass.

For instance, in MDM-Prime-v2 diffusion models, the exponents are measured as $f(x_i) \in \mathbb{R}^p$9 and $w = (w_1, \ldots, w_N)$0, prioritizing dataset scale slightly over model scale for loss minimization at fixed compute (Chao et al., 17 Mar 2026). SoViT vision transformers use similar scaling exponents for width, depth, and MLP dimensions to derive compound scaling for "shape-optimized" models (Alabdulmohsin et al., 2023). For protein LLMs, exponents $w = (w_1, \ldots, w_N)$1, $w = (w_1, \ldots, w_N)$2 reveal that modestly increasing model size while strongly increasing data size achieves compute-optimality, with large models showing rapidly diminishing returns (Serrano et al., 2024).

3. Algorithms for Compute-Optimal Experimental Design

A range of algorithmic paradigms address the compute-optimal selection of designs or models:

• Multiplicative algorithms: Iterative updates based on local sensitivities, typically for D- or A-optimality, with guarantees of monotonic convergence and proven rates under mild assumptions (Duan et al., 2021, Duan et al., 2018).
• Cutting-plane methods: Infinite-dimensional linear or convex programming, with successive refinement via the most violated constraints, applicable to "robust" or multi-criterion design (Burclova et al., 2015).
• Gradient flows: Use of continuous-time dynamical systems (e.g., log-determinant gradient flow) to reach the optimal design, with strong convergence rate results (Piazzon, 2022).
• Screening and sparsity: Quadratic Lasso equivalence and screening rules identify inessential candidates, accelerating large-scale c-optimality problems and enabling dynamic pruning in iterative solvers or homotopy computations (Sagnol et al., 2023, Harman et al., 31 Aug 2025).
• Adaptive surrogate-based search: Bayesian optimization with Gaussian process surrogates to adaptively explore continuous design spaces at lower Jacobian-evaluation cost, particularly effective for high-dimensional and nonlinear models (Seufert et al., 2021).
• Branch-and-bound/minorant-heavy solvers: Modern mixed-integer (conic or linear) programming and projected Newton frameworks accelerate exact D-, A-, I-, and G-optimal design under fixed sample constraints (Harman et al., 2023, Sagnol et al., 2013, Liang et al., 2024).
• Domain-specific translate/relax: For compound-Bayes or random-coefficient criteria, problem restructuring (e.g., to A-optimality in augmented spaces) enables use of SOCP/MISOCP solvers (Harman et al., 2017).

These technical advancements permit large-scale, constraint-rich, and nonlinear designs to be computed under stringent resource budgets, often with provable optimality or a-priori bounds.

4. Compute-Optimal Design in Deep Reinforcement Learning

Compute-optimal design principles have also been explicitly extended to value-based deep RL, where the axes of model capacity $w = (w_1, \ldots, w_N)$3, update-to-data (UTD) ratio $w = (w_1, \ldots, w_N)$4, and batch size $w = (w_1, \ldots, w_N)$5 introduce distinct trade-offs. The key result is a resource allocation formula:

$w = (w_1, \ldots, w_N)$6

with $w = (w_1, \ldots, w_N)$7 samples required to reach return $w = (w_1, \ldots, w_N)$8, and $w = (w_1, \ldots, w_N)$9 required compute. The sample efficiency is empirically modeled as involving saturating power laws in $\sum_i w_i = 1$0 and $\sum_i w_i = 1$1, while batch size is constrained by TD-overfitting phenomena—small models overfit for large $\sum_i w_i = 1$2, while larger models can accommodate larger batch sizes without loss in generalization (Fu et al., 20 Aug 2025).

Practical optimization involves selecting $\sum_i w_i = 1$3 and $\sum_i w_i = 1$4 per derived power-law scalings, subject to upper limits on $\sum_i w_i = 1$5 to prevent overfitting. RL-specific effects (non-stationary TD-targets, replay buffer dynamics) necessitate these nuanced prescriptions, sharply diverging from pure supervised compute scaling.

5. Practical Recommendations and Implementation Considerations

Across regimes, certain general recommendations for compute-optimal design emerge:

• Extract exponents empirically: For new architectures or problem domains, fit scaling exponents $\sum_i w_i = 1$6 from small- or medium-scale grid searches (Chao et al., 17 Mar 2026, Alabdulmohsin et al., 2023, Ziarko et al., 2024).
• Allocate compute via explicit formulas: Compute-optimal $\sum_i w_i = 1$7 and $\sum_i w_i = 1$8 are set as $\sum_i w_i = 1$9, $M(w) = \sum_{i=1}^N w_i f(x_i) f(x_i)^\top$0; in transformer shape scaling, each dimension is scaled by its empirically fit power (Alabdulmohsin et al., 2023).
• Automate sparse screening: For large design spaces or sample sets, employ Lasso reformulations or variance-based pruning to reduce candidates prior to expensive global search or MISOCP (Sagnol et al., 2023, Harman et al., 31 Aug 2025).
• Leverage fast solvers: Apply advanced optimization algorithms (projected Newton, efficient QP-exchange, mixed-integer linear/conic solvers) as backends for exact design computation at moderate or high dimension (Liang et al., 2024, Harman et al., 2023, Sagnol et al., 2013).
• Exploit parallelism: Many sensitivity or validation computations decouple across points or batches, favoring parallel‐first implementations for scalability (Duan et al., 2021, Harman et al., 31 Aug 2025).
• Monitor known pitfalls: Diminishing returns at scale, data/model mismatch, or domain-specific overfitting phenomena require continual empirical monitoring and validation (Serrano et al., 2024, Fu et al., 20 Aug 2025).

In machine learning applications (embedding models, LLMs, RL agents), practical recipes specify when to switch between full fine-tuning versus parameter-efficient approaches, and which hyperparameters (LoRA rank, pool size, architectural "shape") to select at a given compute budget (Ziarko et al., 2024, Chao et al., 17 Mar 2026).

6. Frontier Applications and Empirical Impact

The compute-optimal paradigm has had measurable impact across domains:

• Statistical Models: Exact or approximate optimal designs for complex regression, model discrimination, and nonlinear settings, solved efficiently at previously inaccessible scales (Duan et al., 2021, Duan et al., 2018, Mogalle et al., 2022).
• Deep Language/Vision Models: State-of-the-art perplexity or accuracy from models up to $M(w) = \sum_{i=1}^N w_i f(x_i) f(x_i)^\top$1 smaller by principled compound scaling derived from compute-optimal laws; closure of historic efficiency gaps between architectures (e.g., masked diffusion vs. autoregressive LMs) (Chao et al., 17 Mar 2026, Alabdulmohsin et al., 2023).
• Protein LLMs: Demonstrated that much smaller models, with sufficiently large token sets, converge to the same loss plateau as billion-parameter baselines, reshaping compute budgeting in bioinformatics (Serrano et al., 2024).
• Deep RL: First explicit compute-vs-sample budgeting rules for value-based Q-learning and actor-critic, providing batch and UTD bounds to maximize policy quality under hardware or interaction cost constraints (Fu et al., 20 Aug 2025).
• Practical Workflows: Widely used open-source packages (e.g., PNOD.jl, OptimalDesignComputation, Python qlasso) implement these methods for broad scientific and engineering use (Liang et al., 2024, Piazzon, 2022, Sagnol et al., 2023).

Compute-optimal design enables practitioners to target the efficient frontier of statistical or ML performance for given constraints, systematically exploiting advances in theory, empirical scaling laws, and optimization technology.