Compound Model Scaling

Updated 3 April 2026

Compound model scaling is a framework that jointly allocates parameters, compute, and data across multiple axes to achieve optimal performance.
It applies compound scaling laws derived from empirical estimation to balance model depth, width, input resolution, and modality-specific data.
This approach enables precise trade-off analysis and resource allocation in applications ranging from vision and language to molecular and multimodal systems.

Compound model scaling refers to a spectrum of model design and scaling strategies that systematically allocate computational resources, parameter budgets, and data across multiple axes—such as depth, width, input resolution, modalities, or system modules—to achieve optimal performance under real-world constraints. Instead of scaling a single dimension, compound scaling frameworks jointly consider several architectural and/or data axes, incorporating their interactions into principled scaling laws. This approach governs modern large-scale models across domains from vision and language to multimodal, molecular, and inference systems, as well as modular compound systems involving multiple submodels or pipelines. Rigorous compound scaling permits precise trade-off analysis and enables practitioners to achieve near-optimal efficiency, generalization, and performance in diverse compute and deployment environments.

1. Formulation of Compound Scaling Laws

The archetype for compound model scaling is the generalized multimodal scaling law, which extends classical univariate scaling relations to account for interaction between model size, modality-specific data, and data compression:

$L(N,\{D_m\}) = A N^{-\beta} + \sum_m B_m (c_m D_m)^{-\gamma_m} + \varepsilon$

where:

$N$ is the total number of model parameters (shared or modular),
$D_m$ is the size of the training corpus for modality $m$ ,
$c_m$ is the bits-per-token (compression) for modality $m$ ,
$A, B_m, \beta, \gamma_m$ are fitted constants and scaling exponents,
$\varepsilon$ captures irreducible error/noise.

This law unifies the familiar power-law performance gains from parameter scaling with those attainable via data scaling for each modality, modulated by their representational efficiency. The fit of this law to empirical results is typically strong (e.g., $R^2 \approx 0.98$ ) and constants are readily estimated from grid experiments (Sun et al., 2024).

For architectural settings beyond transformers, the compound scaling principle manifests as multi-axis scaling laws involving depth $(d)$ , width $N$ 0, input resolution $N$ 1, or parameter count $N$ 2 and train token budget $N$ 3. EfficientNet, for example, uses

$N$ 4

where $N$ 5 is the scaling coefficient (Tan et al., 2019). In transformer LLMs, more complex nested scaling laws account for width, depth, total parameters, and tokens, as in:

$N$ 6

empirically fitted on large suites spanning wide architectural regimes (McLeish et al., 7 Feb 2025).

In specialized application domains such as molecular modeling, the compound scaling law for cross-entropy loss is expressed as

$N$ 7

where $N$ 8 is parameter count, $N$ 9 is training tokens, fit separately for each molecular representation (Xu et al., 30 Jan 2026).

2. Empirical Estimation and Structural Interpretation

Compound scaling laws generalize single-axis laws (e.g. $D_m$ 0) and require careful empirical estimation of prefactors and exponents via controlled sweeps over multiple dimensions. For multimodal models, $D_m$ 1, $D_m$ 2, $D_m$ 3, $D_m$ 4 are estimated by regressions over grids of parameter budgets and modality-specific datasets, with compression factors $D_m$ 5 accounting for the efficiency of the tokenization scheme (e.g., text $D_m$ 6, image $D_m$ 7) (Sun et al., 2024).

Typical exponents for mainstream modalities are:

Modality	$D_m$ 8 (bits/token)	$D_m$ 9	$m$ 0
text	1	0.095	0.45
audio	5	0.120	0.60
image	10	0.110	0.55
video	20	0.140	0.80

Interpreting the exponents, $m$ 1 captures marginal performance returns from parameter scaling (typically $m$ 2), while $m$ 3 quantifies the return from scaling modality $m$ 4's effective corpus (higher $m$ 5 yields higher value per data unit). The compression $m$ 6 sets the relative cost of acquiring effective data.

For transformer architectures, the nested reciprocal law in (McLeish et al., 7 Feb 2025) exposes stepwise saturation: width dominates at small scales (large $m$ 7), depth ( $m$ 8) and parameter ( $m$ 9) scaling only yield substantial gains after width scaling saturates, and token scaling ( $c_m$ 0) flattens out at very large scales.

3. Optimization and Allocation Recipes

Compound model scaling enables principled allocation of compute, parameter, and data resources:

Multimodal allocation: Given a total parameter and data budget, and a target loss/accuracy, compute-optimal division is obtained via Lagrangian minimization. The resulting proportional allocation for each axis follows

$c_m$ 1

where $c_m$ 2 is determined by the performance constraint (Sun et al., 2024).

Architecture grid search: For width, depth, and parameter combined scaling (as in Gemstones), practitioners evaluate the compound loss law over a grid of $c_m$ 3 (with $c_m$ 4 and $c_m$ 5 set by budget) and select the physically viable optimum (McLeish et al., 7 Feb 2025).
Compute-optimal scaling: For any setting where $c_m$ 6 (FLOPs budget), as in molecular LMs, the optimal allocation is

$c_m$ 7

$c_m$ 8

(Xu et al., 30 Jan 2026). In vision models, EfficientNet's balanced scaling ensures receptive field, capacity, and resolution grow in lockstep for optimal trade-offs (Tan et al., 2019).

Resource-constrained deployment: When parameter count is constrained (e.g., on mobile devices), prioritization is adaptive to the returns and compression. For instance, allocate more to text and audio if high-resolution vision is prohibitively expensive (Sun et al., 2024).

4. Compound Scaling in System-Level and Modular Contexts

Compound scaling also governs compound inference systems, ensemble methods, and modular/multi-LLM pipelines:

Compound inference systems: In majority-vote or filter-vote LLM ensembles, performance as a function of call count $c_m$ 9 follows a non-monotonic scaling law due to distribution of instance difficulty:

$m$ 0

where $m$ 1, $m$ 2 characterize "easy" and "hard" items; $m$ 3 is the binomial tail. There exists a closed-form for optimal $m$ 4 maximizing performance (Chen et al., 2024).

Compound AI systems / pipelines: In modular systems (e.g., multi-agent debate, self-refine), end-to-end accuracy is monotonic in module-level improvements, and assigning the best available submodel per module yields globally optimal allocation, efficiently found via the LLMSelector algorithm ( $m$ 5 calls for $m$ 6 modules, $m$ 7 model choices) (Chen et al., 20 Feb 2025).

5. Multimodal and Heterogeneous Data–Model Co-scaling

In industrial or search/recommendation settings, compound scaling laws generalize to joint data–model co-design, where performance exhibits positive synergy between model scale and heterogeneous, high-fidelity data:

$m$ 8

with cross-term $m$ 9. Model scaling saturates if not matched by expansion in (properly constructed) sample diversity and label richness (e.g., via ES $A, B_m, \beta, \gamma_m$ 0 in UniScale); conversely, data scaling without architecture adaptation induces negative transfer or plateaus (Yu et al., 25 Mar 2026).

Complex architectures such as HHSFT are required to efficiently fuse and exploit heterogeneous sources as sample capacity grows, demonstrating non-linear, accelerating scaling returns in business metrics when data and model axes are grown together.

6. Architectural Compound Scaling: Vision and Beyond

In vision, compound scaling balances depth, width, and input resolution (EfficientNet, Fast Compound Scaling) for optimal accuracy–FLOPs trade-offs. Uniform compound scaling maintains the scaling constraint ( $A, B_m, \beta, \gamma_m$ 1), while Fast Compound Scaling interpolates towards width-dominant scaling $A, B_m, \beta, \gamma_m$ 2 for memory-bound deployment, minimizing activation growth $A, B_m, \beta, \gamma_m$ 3 with $A, B_m, \beta, \gamma_m$ 4 the scale (Dollár et al., 2021). Practical choices of exponents and relative scaling depend strongly on the deployment constraint (FLOPs, memory, wall-time) and target hardware profile.

7. Domain-Specific and Systemic Compound Scaling Frameworks

Compound scaling laws are domain-adaptive; e.g., in molecular LMs, fragment-based representations yield higher data exponent ( $A, B_m, \beta, \gamma_m$ 5) and lower irreducible loss than atom-level SMILES, so optimal parameter–token allocation shifts with representation (Xu et al., 30 Jan 2026). In mixture-of-experts (MoE) architectures, joint triads of FLOPs per token, active parameter count, and total parameters must be simultaneously constrained; scaling laws arise from algebraic reduction and rank-preserving dimensionality reduction, with optimality bands widening at large scale, granting deployment flexibility (Wan et al., 23 Mar 2026).

8. Limitations, Sensitivities, and Practical Considerations

Compound scaling prescriptions are sensitive to underlying hyperparameter choices (learning rate, initialization scale, tokenization granularity), experimental ranges, and representation choices (McLeish et al., 7 Feb 2025). The precise exponents and coefficients must be refitted for any major deviation in training recipe, architecture, or data pipeline; over/under-training penalties are typically mild only near the scaling optimum. At extreme scales or when introducing new architectures/modalities, empirical evaluation across axes is necessary, since extrapolation outside the fitted regime is not guaranteed to hold.

By unifying multiple resource and data axes, compound model scaling provides a robust and extensible foundation for optimal model and system design across deep learning domains and deployment modalities. It quantitatively prescribes the joint trade-offs underlying contemporary state-of-the-art architectures, ensemble pipelines, and multimodal inference systems. For precise implementation, practitioners are advised to follow empirically-fitted scaling laws, continually revalidating against actual performance on the intended data and operational context.

Key references: (Sun et al., 2024, Tan et al., 2019, McLeish et al., 7 Feb 2025, Dollár et al., 2021, Wang et al., 25 Feb 2025, Xu et al., 30 Jan 2026, Yu et al., 25 Mar 2026, Chen et al., 20 Feb 2025, Chen et al., 2024, Wan et al., 23 Mar 2026).