Compound Scaling Protocol

• Compound scaling protocol is a method that systematically scales neural network dimensions—depth, width, and resolution—using power-law principles to meet resource constraints.
• It generalizes single-axis scaling with grid-search tuned multipliers, providing closed-form rules or empirical recipes for both convolutional and transformer architectures.
• The approach balances trade-offs among statistical efficiency, computational cost, and hardware limitations, informing strategies for training and compound inference.

A compound scaling protocol is a principled methodology for simultaneously scaling multiple architectural dimensions of a neural network—such as depth, width, and input resolution—under a computational or resource budget, with the aim of maximizing efficiency or target performance. The protocol generalizes the paradigm of single-axis scaling and, depending on domain and model class, can generate closed-form scaling laws, empirical recipes, or optimization algorithms for effective network growth. Compound scaling achieves state-of-the-art efficiency for both convolutional and transformer architectures and has extensions to compound inference via multiple calls in LLM-based systems.

1. Mathematical Framework of Compound Scaling

The compound scaling formulation for convolutional architectures seeks to trade accuracy for resources by optimizing the tuple $(d, w, r)$—depth (layers), width (channels), and resolution (input size). The canonical formulation, introduced in "EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks" (Tan et al., 2019), casts network scaling as: $$\max_{d, w, r} \quad \mathrm{Accuracy}(N(d, w, r)) \quad\text{s.t.}\quad \mathrm{FLOPS}(N(d, w, r))\leq\text{budget}$$ The core observation is that convolutional network FLOPs scale as $d\cdot w^2\cdot r^2$. Compound scaling proposes a power-law expansion from a baseline, parametrized by a compound coefficient $\phi$ and three scaling multipliers $(\alpha, \beta, \gamma)$: $$d=\alpha^{\phi},\quad w=\beta^{\phi},\quad r=\gamma^{\phi}$$ with $\alpha\beta^2\gamma^2\approx 2$ (per $\phi=1$ doubling FLOPs), and typically $\alpha\geq1, \beta\geq1, \gamma\geq1$. The protocol comprises a two-stage process: grid search $(\alpha, \beta, \gamma)$ for $$\max_{d, w, r} \quad \mathrm{Accuracy}(N(d, w, r)) \quad\text{s.t.}\quad \mathrm{FLOPS}(N(d, w, r))\leq\text{budget}$$0 on a small baseline model, then stretching to arbitrary $$\max_{d, w, r} \quad \mathrm{Accuracy}(N(d, w, r)) \quad\text{s.t.}\quad \mathrm{FLOPS}(N(d, w, r))\leq\text{budget}$$1 and rounding dimensions as appropriate.

For transformers, as in "Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers" (Tay et al., 2021), dimensions include layer count, hidden size, FFN size, and head count. No closed-form law is derived; instead, empirical exponents for layer count and width are obtained for each compute regime: $$\max_{d, w, r} \quad \mathrm{Accuracy}(N(d, w, r)) \quad\text{s.t.}\quad \mathrm{FLOPS}(N(d, w, r))\leq\text{budget}$$2

2. Classical vs. Fast Compound Scaling Variants

Uniform compound scaling, as per EfficientNet, evenly distributes scaling among dimensions using the constraint $$\max_{d, w, r} \quad \mathrm{Accuracy}(N(d, w, r)) \quad\text{s.t.}\quad \mathrm{FLOPS}(N(d, w, r))\leq\text{budget}$$3, leading to $$\max_{d, w, r} \quad \mathrm{Accuracy}(N(d, w, r)) \quad\text{s.t.}\quad \mathrm{FLOPS}(N(d, w, r))\leq\text{budget}$$4 for a $$\max_{d, w, r} \quad \mathrm{Accuracy}(N(d, w, r)) \quad\text{s.t.}\quad \mathrm{FLOPS}(N(d, w, r))\leq\text{budget}$$5-fold FLOPs increase. In practice,

• Depth scaling: $$\max_{d, w, r} \quad \mathrm{Accuracy}(N(d, w, r)) \quad\text{s.t.}\quad \mathrm{FLOPS}(N(d, w, r))\leq\text{budget}$$6 (activation count grows linearly)
• Width scaling: $$\max_{d, w, r} \quad \mathrm{Accuracy}(N(d, w, r)) \quad\text{s.t.}\quad \mathrm{FLOPS}(N(d, w, r))\leq\text{budget}$$7 (sublinear activation growth but lower accuracy)
• Uniform compound: $$\max_{d, w, r} \quad \mathrm{Accuracy}(N(d, w, r)) \quad\text{s.t.}\quad \mathrm{FLOPS}(N(d, w, r))\leq\text{budget}$$8 (balances both)

The fast compound scaling strategy proposed in "Fast and Accurate Model Scaling" (Dollár et al., 2021) introduces a one-parameter family interpolating between pure width and full compound: $$\max_{d, w, r} \quad \mathrm{Accuracy}(N(d, w, r)) \quad\text{s.t.}\quad \mathrm{FLOPS}(N(d, w, r))\leq\text{budget}$$9 with the key result that for $d\cdot w^2\cdot r^2$0 (editor: "dWr protocol"), accuracy matches classic compound, while activations—and thus runtime on memory-bound hardware—only scale as $d\cdot w^2\cdot r^2$1, compared to $d\cdot w^2\cdot r^2$2 for classic compound (Dollár et al., 2021).

3. Empirical Protocols and Scaling Recipes

EfficientNet's procedure entails:

1. Single grid search for $d\cdot w^2\cdot r^2$3 on the baseline network at $d\cdot w^2\cdot r^2$4, enforcing $d\cdot w^2\cdot r^2$5.
2. For a target $d\cdot w^2\cdot r^2$6, compute $d\cdot w^2\cdot r^2$7.
3. Adjust dimensions (rounding for compatibility), retrain, and apply regularization (e.g., increased dropout with depth).

For transformers, the protocol from (Tay et al., 2021) is:

1. Empirically sweep $d\cdot w^2\cdot r^2$8 at multiple total compute budgets.
2. Fit exponents $d\cdot w^2\cdot r^2$9 for each budget regime to trace out the fine-tuning Pareto frontier.
3. Select models "deep then wide," using widths that preserve per-head key dimension in the $\phi$0--$\phi$1 range.

In compound inference, as framed in "Are More LLM Calls All You Need? Towards Scaling Laws of Compound Inference Systems" (Chen et al., 2024), the protocol is:

1. Fit parameters $\phi$2 from a small pilot (e.g., $\phi$3, $\phi$4 items).
2. Predict accuracy for any $\phi$5 using the closed-form model.
3. Solve analytically for $\phi$6 (optimal call count) using the scaling law $\phi$7 where $\phi$8 solves:

$\phi$9

ensuring no over-allocation of computational resources.

4. Non-monotonic Behavior and Analytical Laws in Compound Inference

The classical intuition that increasing the number of aggregate model calls (e.g., majority voting over LLM outputs) monotonically increases system performance is violated when the test set comprises a mixture of “easy” and “hard” instances. The analytical law for majority-vote accuracy for binary tasks parameterized by $(\alpha, \beta, \gamma)$0 is: $(\alpha, \beta, \gamma)$1 where $(\alpha, \beta, \gamma)$2 is the incomplete beta function (Chen et al., 2024).

The expected accuracy curve $(\alpha, \beta, \gamma)$3 can show monotonic, U-, or inverse-U-shaped behavior depending on $(\alpha, \beta, \gamma)$4. Non-monotonicity arises due to the tradeoff: increasing $(\alpha, \beta, \gamma)$5 helps "easy" queries (with $(\alpha, \beta, \gamma)$6) but hurts performance on "hard" queries (with $(\alpha, \beta, \gamma)$7) due to over-confident aggregation. The point of maximal accuracy is computable in closed-form (see above), enabling precise resource allocation.

Empirically, fitting parameters on small pilot sets predicts $(\alpha, \beta, \gamma)$8 for $(\alpha, \beta, \gamma)$9 with mean squared error $$d=\alpha^{\phi},\quad w=\beta^{\phi},\quad r=\gamma^{\phi}$$0–$$d=\alpha^{\phi},\quad w=\beta^{\phi},\quad r=\gamma^{\phi}$$1 (Chen et al., 2024).

5. Applications, Guidelines, and Pareto-Efficiency

Compound scaling protocols have been validated in both vision and LLMs. EfficientNet demonstrates that harmoniously increasing depth, width, and resolution yields state-of-the-art accuracy for ImageNet and other transfer tasks, with up to $$d=\alpha^{\phi},\quad w=\beta^{\phi},\quad r=\gamma^{\phi}$$2 fewer parameters and $$d=\alpha^{\phi},\quad w=\beta^{\phi},\quad r=\gamma^{\phi}$$3 less FLOPs for comparable accuracy compared to prior models (Tan et al., 2019). Table 1 from (Tay et al., 2021) summarizes transformer Pareto-optimal configurations across different compute regimes:

Model	$$d=\alpha^{\phi},\quad w=\beta^{\phi},\quad r=\gamma^{\phi}$$4	$$d=\alpha^{\phi},\quad w=\beta^{\phi},\quad r=\gamma^{\phi}$$5	Params	Flops	SG	GLUE	SQuAD
Small$$d=\alpha^{\phi},\quad w=\beta^{\phi},\quad r=\gamma^{\phi}$$6	24	512	193M	10T	71.1	83.1	85.5
Base$$d=\alpha^{\phi},\quad w=\beta^{\phi},\quad r=\gamma^{\phi}$$7	36	768	621M	29T	75.6	85.3	87.8
Large$$d=\alpha^{\phi},\quad w=\beta^{\phi},\quad r=\gamma^{\phi}$$8	36	1024	1.1B	50T	79.3	87.2	89.2
XL$$d=\alpha^{\phi},\quad w=\beta^{\phi},\quad r=\gamma^{\phi}$$9	32	1024	3.8B	169T	79.9	86.9	89.5

Guidelines for practitioners include: grid-search scaling ratios once per baseline (EfficientNet), select region-appropriate exponents for transformer shape knobs (Tay et al., 2021), prioritize width for memory-limited deployment (set $\alpha\beta^2\gamma^2\approx 2$0 in fast compound scaling (Dollár et al., 2021)), and in compound inference, always fit difficulty mixture parameters before scaling model calls (Chen et al., 2024).

6. Trade-offs, Limitations, and Hardware Considerations

Compound scaling balances statistical efficiency with practical hardware constraints. With higher $\alpha\beta^2\gamma^2\approx 2$1, activation memory and runtime may rise sub-linearly with FLOPs under fast compound scaling, which offers wall-time reductions on modern GPU/TPU hardware (Dollár et al., 2021). A trade-off exists: while width-centric scaling is memory-efficient, insufficient depth or spatial extent can reduce representational capacity and ultimately accuracy.

All compound scaling protocols require careful rounding heuristics for architectural compatibility (e.g., channel counts divisible by group counts, input size aligned to network stride), especially for highly regularized or group-convolutional architectures (Tan et al., 2019); optimal heads-per-dimension settings are necessary for transformers (Tay et al., 2021). Over-scaling may yield diminishing returns or unstable training.

A plausible implication is that compound scaling should always be tuned to the operational regime (e.g., vision vs. language, pre-training vs. inference, available FLOPs, or active memory constraints), as no universal exponents or ratios optimize all regimes. Empirical pilot calibration is thus essential for robust adoption.