---
title: Per-Module Hyperparameter Optimisation
url: https://www.emergentmind.com/topics/per-module-hyperparameter-optimisation
type: topic
---

# Per-Module Hyperparameter Optimisation

Per-module hyperparameter optimisation is a methodology in modern deep learning and optimisation wherein hyperparameters are tuned individually for distinct architectural modules or algorithmic components, rather than being shared globally across the whole model or algorithm. This paradigm accommodates the heterogeneous behavior and scaling properties of diverse network or algorithm segments—such as transformer blocks, subnetworks, optimizer modules, or augmentation blocks—yielding improved performance and efficiency. Recent advances validate systematic approaches for both searching and transferring per-module hyperparameters, especially under rapid scaling in model capacity, data, and compute budgets [2512.22382][2304.14766][2202.11841][2102.12905].

## 1. Motivation for Per-Module Hyperparameter Optimisation

Traditional global hyperparameter regimes enforce a single learning-rate, weight-decay, or optimizer setting across all parameter groups of a network. In modular architectures (transformers, CNNs, attention blocks, federated and multi-subnetwork models), modules differ substantially in gradient scale, curvature, effective capacity, and data exposure. Empirical findings show that enforcing uniform hyperparameter values can lead to sub-optimal outcomes by orders of magnitude, with distinct modules favoring sharply different configurations for stability and convergence rate [2512.22382].

A plausible implication is that as models are scaled in width, depth, batch size, or training duration, optimal per-module settings increasingly diverge from a one-size-fits-all rule. Empirical evidence further demonstrates that per-module tuning at proxy (small) scale can yield substantial training speedups and generalization improvements when transferred to much larger models [2512.22382][2202.11841].

## 2. Parameterisations and Scaling Laws

State-of-the-art per-module hyperparameter transfer leverages the unified Complete$^{(d)}$ parameterisation, which co-adapts module settings along four axes: width ($N$), depth ($L$), batch size ($B$), and duration ($D$) [2512.22382]. This parameterisation prescribes that any per-module hyperparameter $\zeta_{m,\ell}$ at a new scale is computed as:

$$
\zeta_{m,\ell}(N, L, B, D) = \zeta^{type}_m \cdot \zeta^{depth}_\ell(L) \cdot SDE(B, D) \cdot CP_m(N, L)
$$

where:

- $\zeta^{type}_m$ optimizes module-type multipliers at base scale,
- $\zeta^{depth}_\ell(L)$ is a linearly-interpolated depth multiplier,
- $SDE(B, D)$ implements the square-root batch-size/duration reparameterisation,
- $CP_m(N, L)$ applies explicit scaling laws (such as for learning rates, initialization scales, residual multipliers) per module type (see Table 1 in [2512.22382]).

Each module—such as embeddings, QKV projections, MLP blocks, LayerNorms—is assigned its own learning-rate multiplier ($\eta_{m,\ell}$), weight-decay ($\lambda_{m,\ell}$), AdamW parameters, initialization scale ($\sigma_{m,\ell}$), and, for residual blocks, a residual scaling factor ($s_{m,\ell}$).

Transfer to larger models combines base-scale optimizers with explicit formulaic rescalings (e.g., $\eta_{hidden} \gets \eta_b \cdot N \cdot L \cdot \sqrt{B/D}$) for each module and depth group, without requirement for re-tuning at scale [2512.22382].

## 3. Search Algorithms and High-Dimensionality

The search space for per-module hyperparameter optimisation often reaches 80+ dimensions due to the combination of module types and depths. The response surfaces exhibit “invex” geometry (i.e., the absence of spurious local minima) but contain sharp cliffs where runs diverge [2512.22382]. Gaussian Process surrogates become ineffective due to nonstationarity and high failure rates in random search.

A robust approach applies trust-region random search in log-space. Starting from global-tuned hyperparameters, the algorithm samples perturbations within a fixed radius, repeatedly shrinking the radius upon consecutive failures. Each candidate configuration is trained for a full run; up to 5,000 parallel trials are feasible on modern hardware [2512.22382].

For multi-subnetwork architectures, additional strategies include:

- Divide-and-Conquer Bayesian Optimisation (DCBO): recombine well-tuned subnetworks from previous top-performing full models, leveraging exponential candidate pool size without incurring full re-training costs [2202.11841].
- Subnetwork-Adaptive Bayesian Optimisation (SABO): allocate search budget to subnetworks according to their empirically estimated influence on end-to-end loss, favouring exploration of weak links [2202.11841].

## 4. Transfer and Validation Protocols

Per-module hyperparameters optimised at small scale transfer to large-scale models via explicit scaling laws—global multipliers remain unchanged, depth-wise multipliers are interpolated, and formulaic rescalings ensure invariants when adjusting width, depth, batch, and duration [2512.22382].

In applied settings, network and data are partitioned (e.g., as in [2304.14766]), with each block updated on its own shard and out-of-training-sample (OOTS) criteria applied for hyperparameter validation. This marginal-likelihood inspired criterion efficiently optimises hyperparameters (including data-augmentation, regularization, dropout rates) in a single interleaved run, sidestepping the need for separate validation sets or repeated runs.

For algorithmic frameworks such as Modular CMA-ES, per-module tuning comprehensively assesses contributions of new modules by comparing baseline vs extended search spaces, analyzing activation frequencies and interplay among modules, and evaluating performance via metrics like Area Over the ECDF (AOC) and Expected Running Time (ERT) [2102.12905].

## 5. Empirical Performance Gains and Benchmarks

Experimental results validate the effectiveness of per-module hyperparameter optimisation:

- Transformer models: per-module tuned hyperparameters reach target loss $2.3\times$ faster than best global hyperparameters, with large-scale transfer yielding $27$% speed-ups (i.e., $1.32\times$ faster) at identical final loss [2512.22382].
- Large Language Models (7.2B params): pre-training accelerated by $\approx 25$%, improving downstream benchmarks for language, coding, and reasoning [2512.22382].
- Multi-subnetwork deep models (CNN/DFNN): DCBO achieves mean speedups of $4.8\times$ to $23.6\times$, accuracy gains up to $3.5$% and MSE reductions up to $4.4$ vs single-model TPE-based Bayesian optimisation [2202.11841].
- Neural network partitioning: OOTS-driven validation lowers error rates by $3$–$5$ pp in low-data regimes and achieves competitive performance on MNIST, CIFAR-10, TinyImageNet, and federated learning benchmarks [2304.14766].
- Modular optimisers: extending module options in Modular CMA-ES yields up to $97$% improvement in AOC on specific BBOB functions, with clear dependencies among module settings [2102.12905].

## 6. Methodological Guidelines and Analysis

Canonical guidelines for per-module hyperparameter optimisation include:

- Decompose architectures into orthogonal modules; encode both categorical (activation/switches) and continuous (scaling) hyperparameters per module [2102.12905].
- Construct manageable search spaces, leveraging base-scale tuning for global and type-wise multipliers, and continuous interpolation for depth multipliers [2512.22382].
- For racing-based tuners (e.g., irace), allocate budget per module group, and replicate assessment runs for robust validation [2102.12905].
- For multi-subnetwork architectures, prefer DCBO for maximum speed-up, optionally augmenting with SABO when subnetwork relevance is highly imbalanced [2202.11841].
- Always conduct independent verification of elite configurations, and analyze module activation frequencies and interplay via statistical tests and visualizations [2102.12905].
- Use robust aggregate metrics, such as Area Over the ECDF, and quantile bands for ERT, to score distributed performance across targets [2102.12905].

## 7. Limitations and Future Directions

While current per-module hyperparameter optimisation strategies demonstrate substantial empirical gains, several constraints and open questions remain:

- DCBO/SABO approaches have primarily been validated with TPE-based Bayesian Optimisation; generalization to GP-based BO, population-based training, or RL-driven NAS requires further study [2202.11841].
- Effectiveness is established for CNN/DFNN subnetworks; extensions to attention mechanisms, RNNs, graph networks, or arbitrary architectures entail additional complexity in loss estimation [2202.11841].
- Partitioning schemes require careful choice of $K$ to balance granularity of OOTS signals and sufficient representation per block [2304.14766].
- Empirical speedups are sensitive to implementation details (e.g., weight freezing, framework specifics); transfer may require adaptation in non-TensorFlow environments [2202.11841].
- For module assessment in optimisers, high run-to-run variance and stochasticity motivate multiple repeated tuner runs and conservative configuration budgets [2102.12905].

A plausible implication is that the future of per-module hyperparameter optimisation may involve automated proxy-driven importance rankings, hierarchical priors, and multi-fidelity or early-stopping budget control to expedite convergence and scalability in increasingly modular deep learning and optimisation landscapes.

Source: https://www.emergentmind.com/topics/per-module-hyperparameter-optimisation