Adaptive Doubling Strategy in HPC HPO

• Adaptive Doubling Strategy is a scheduling principle that promotes promising trials by doubling GPU count while increasing training epochs.
• It integrates asynchronous successive halving with resource-adaptive doubling to efficiently reallocate fixed HPC resources, achieving runtime reductions up to 1.9×.
• The strategy enhances model performance in computer vision, CFD, and additive manufacturing tasks through optimized batch scaling and early configuration elimination.

Adaptive doubling strategy, in the sense developed for large-scale hyperparameter optimization on high-performance computing systems, denotes a scheduling principle in which promising trials are promoted not only to higher time fidelity but also to higher space fidelity by doubling the number of workers or GPUs assigned to them. The most explicit formulation is the Resource-Adaptive Successive Doubling Algorithm (RASDA), which combines an ASHA-style asynchronous successive halving process in training time with a resource-adaptive successive doubling process in data-parallel workers, under a fixed total GPU allocation $R_{\max}$ (Aach et al., 2024). In this formulation, poor configurations are eliminated early, their GPUs are released immediately, and promoted configurations are relaunched from checkpoints with larger world size, larger effective batch size, and a learning-rate warmup; empirically, this yields up to $1.9\times$ runtime reduction relative to ASHA while maintaining or improving final solution quality on workloads from computer vision, computational fluid dynamics, and additive manufacturing (Aach et al., 2024).

1. Conceptual definition and problem setting

The adaptive doubling strategy arises in bandit-based hyperparameter optimization with multi-fidelity training. Each configuration $x \in X$ is treated as an arm, and an evaluation is a partial training run at fidelity $r$, such as a number of epochs or a number of workers/GPUs. The optimization objective is to minimize validation loss $L(\theta(x))$, or equivalently maximize accuracy, under a finite compute budget $B$, while using early stopping to discard weak arms and promotion to allocate more resources to promising ones (Aach et al., 2024).

A defining feature of the RASDA formulation is its use of two resource types. Time fidelity is the number of epochs or iterations per trial. Space fidelity is the number of workers or GPUs assigned to a trial via data-parallel training. Instead of treating fidelity exclusively as training duration, RASDA couples time-wise successive halving with space-wise successive doubling, so that later-stage trials both train longer and train on more devices (Aach et al., 2024).

The intended deployment environment is a static-allocation HPC job. A fixed total number of GPUs $R_{\max}$ is reserved in advance, and the scheduler reallocates GPUs within that job without changing $R_{\max}$. Dynamic cloud elasticity is explicitly not assumed. This matters because the method is designed for workloads such as ImageNet ($\sim 146$ GB), additive manufacturing 3D video ($\sim 60$ GB), and CFD flows ($1.9\times$0 TB), where data movement, checkpoint I/O, and interconnect saturation are first-order constraints (Aach et al., 2024).

Within this setting, the phrase “adaptive doubling” refers to doubling workers at promotion events, not to a gambling-style progression. A related but distinct example is “Doubly Adaptive Social Learning,” where “doubly adaptive” refers to two concurrent adaptation stages in online belief formation rather than successive resource doubling (Carpentiero et al., 24 Apr 2025). This suggests that, in current arXiv usage, the phrase is context-sensitive and should be interpreted from the surrounding algorithmic framework.

2. Formalization of time and space fidelities

RASDA formalizes rung-based progression in time and doubling in space. For time fidelity, rung milestones follow a geometric schedule:

$1.9\times$1

with $1.9\times$2 and $1.9\times$3, where $1.9\times$4 is the maximum number of epochs (Aach et al., 2024). More generally, with scaling factor $1.9\times$5,

$1.9\times$6

The paper states that $1.9\times$7 is typical (Aach et al., 2024).

Successive elimination proceeds under a reduction factor $1.9\times$8, again typically $1.9\times$9. If $x \in X$0 denotes the number of active trials at rung $x \in X$1, then

$x \in X$2

Promotion is determined by a performance quantile threshold $x \in X$3 that selects the best $x \in X$4 trials (Aach et al., 2024).

The space-fidelity rule is the core adaptive doubling mechanism:

$x \in X$5

subject to

$x \in X$6

Here $x \in X$7 is the number of workers or GPUs assigned to configuration $x \in X$8 at rung $x \in X$9, and $r$0 is the cap on per-trial parallelism (Aach et al., 2024). The scheduler therefore promotes fewer trials and simultaneously gives each promoted trial more GPUs, while remaining within the fixed global worker pool.

The data-parallel training model makes the effect of doubling explicit. With $r$1 workers and per-worker batch size $r$2, the effective batch is

$r$3

Workers compute local gradients $r$4, which are averaged as

$r$5

The epoch-time model is

$r$6

and parallel efficiency is

$r$7

This expresses the governing trade-off: more workers reduce pure compute time but increase communication cost through NCCL synchronization (Aach et al., 2024).

3. Algorithmic realization in RASDA

RASDA is defined as “ASHA-style asynchronous successive halving in time + resource-adaptive successive doubling in space” (Aach et al., 2024). Each trial begins at rung $r$8 with $r$9 epochs and $L(\theta(x))$0 workers under PyTorch Distributed Data Parallel with NCCL backend. When it reaches a rung milestone $L(\theta(x))$1, it reports a metric $L(\theta(x))$2, and the scheduler ranks completed trials in that rung asynchronously (Aach et al., 2024).

Promoted trials are treated as elastic relaunches rather than in-place resizes. For a promoted trial $L(\theta(x))$3 at rung $L(\theta(x))$4, the target worker count is

$L(\theta(x))$5

If enough GPUs are free, the trial is relaunched with $L(\theta(x))$6, its checkpoint is restored, the DDP process group is reinitialized, the effective batch size is increased, and a learning-rate warmup is applied before continuing to the next rung. If enough GPUs are not free, the trial enters a waiting list until eliminated trials release resources (Aach et al., 2024).

The asynchronous design is central. There are no global barriers, decisions are made when metrics arrive, and stragglers do not block promotion. Because promotions occur asynchronously, the exact fraction eliminated at a rung may differ from $L(\theta(x))$7, although worker allocation remains bounded by $L(\theta(x))$8 (Aach et al., 2024). This differs from synchronous successive halving and preserves high resource utilization in heterogeneous runtime conditions.

Checkpointing is part of the operational design. Trials checkpoint every $L(\theta(x))$9 epochs, with the paper giving $B$0 as an example to reduce I/O. Promoted trials relaunch from the latest checkpoint. Hardware failures trigger trial restart and rejoin the rung while allocation remains elastic within $B$1 (Aach et al., 2024). This framing makes adaptive doubling an HPC scheduler design as much as an HPO method.

The implementation described in the paper uses Ray Tune’s ASHA for time-wise halving together with ray.tune.schedulers.ResourceChangingScheduler for space-wise doubling. Promotion-triggered resource changes reinitialize torch.distributed.init_process_group, rebuild DataLoader shards for the new world_size, reload model and optimizer state, and continue until the next milestone (Aach et al., 2024).

4. Optimization rationale and relation to earlier schedulers

The design rationale given for adaptive doubling is that HPC systems admit two levels of parallelism. First, many configurations can be evaluated concurrently. Second, promising configurations can be accelerated through data-parallel training once weaker ones have been terminated. In the paper’s formulation, terminating poor trials frees GPUs that are then reassigned to good trials, reducing the wall-clock time required to fully train the best configuration (Aach et al., 2024).

A further rationale is implicit batch-size scheduling. Because promotion increases $B$2, it also increases $B$3. The paper states that literature and its experiments indicate that increasing batch size later in training, when statistical efficiency declines, can maintain or improve generalization if learning-rate warmups and scaling rules are applied. For SGD, the linear scaling rule with warmup is

$B$4

and for ADAM, following Malladi et al., the square-root rule is

$B$5

again with warmup (Aach et al., 2024). Promotion therefore changes both systems throughput and optimization dynamics.

Relative to prior schedulers, RASDA preserves the bandit elimination principle of Successive Halving and Hyperband but adds worker doubling at promotion. It also differs from SH and Hyperband by using asynchronous execution, like ASHA, rather than synchronous brackets or global elimination phases (Aach et al., 2024).

The relationships can be summarized as follows.

Method	Time fidelity	Space fidelity
Successive Halving / Hyperband	Fixed milestones, elimination	Not added at promotion
ASHA	Asynchronous halving in time	Not added at promotion
RASDA	ASHA-style asynchronous halving	Successive doubling in workers/GPUs

The paper also contrasts RASDA with HyperSched, Rubberband, and SEER. Those systems are characterized as cloud-oriented elastic schedulers optimizing deadlines or cost with dynamic cluster sizes, whereas RASDA assumes a static HPC allocation and aims at obtaining the best-quality model as fast as possible under fixed total GPUs (Aach et al., 2024). A plausible implication is that adaptive doubling is particularly well matched to batch-scheduled supercomputing environments where unused GPUs inside an allocation are an avoidable opportunity cost.

5. Empirical performance and scaling behavior

The empirical evaluation spans two systems: JURECA-DC-GPU, with 192 nodes and $B$6 NVIDIA A100 40 GB per node with 2 IB links per node, and JUWELS BOOSTER, with 936 nodes and $B$7 NVIDIA A100 40 GB per node with 4 IB links per node (Aach et al., 2024).

Three application domains are reported. The CV task uses ResNet50 on ImageNet on 64 GPUs for 40 epochs with $B$8 and milestones $B$9. The AM task uses a SwinTransformer reconstructing laser power and speed from high-speed LPBF video on 128 GPUs for 20 epochs with milestones $R_{\max}$0. The CFD task uses a convolutional autoencoder on actuated TBL flows on 128 GPUs for 40 epochs with milestones $R_{\max}$1 (Aach et al., 2024).

The reported results are direct evidence for the adaptive doubling strategy.

Workload	Runtime: ASHA vs RASDA	Quality outcome
CV	31,637 s vs 18,502 s	RASDA slightly better accuracy
AM	5,784 s vs 3,803 s	RASDA better val/test MSE
CFD	19,487 s vs 10,242 s	RASDA better val/test MSE and relative error

For CV, training accuracy is reported as $R_{\max}$2 vs $R_{\max}$3, validation accuracy as $R_{\max}$4 vs $R_{\max}$5, and test accuracy as $R_{\max}$6 vs $R_{\max}$7, with RASDA faster by $R_{\max}$8 (Aach et al., 2024). For AM, validation MSE is $R_{\max}$9 vs $R_{\max}$0 and test MSE is $R_{\max}$1 vs $R_{\max}$2, with $R_{\max}$3 runtime reduction (Aach et al., 2024). For CFD, validation MSE is $R_{\max}$4 vs $R_{\max}$5, test MSE is $R_{\max}$6 vs $R_{\max}$7, and test relative reconstruction error is $R_{\max}$8 vs $R_{\max}$9, with $\sim 146$0 runtime reduction (Aach et al., 2024).

The scaling study extends to 1,024 GPUs on JUWELS BOOSTER. Weak scaling efficiency is reported as $\sim 146$1 on 6-epoch ImageNet runs up to 1,024 GPUs (Aach et al., 2024). In the 1,024-GPU CFD HPO experiment, 64 configurations start at 16 GPUs per trial over 5–20 epochs, yielding approximately $\sim 146$2 validation MSE, approximately $\sim 146$3 test MSE, and approximately $\sim 146$4 relative error. The paper states that this improves quality by $\sim 146$5–$\sim 146$6 relative to the 128-GPU run and describes the result as the first application of systematic HPO to a terabyte-scale scientific dataset in the literature (Aach et al., 2024).

These results support two distinct conclusions. First, adaptive doubling reduces time-to-quality relative to halving-only ASHA. Second, its rung-based batch enlargement does not merely preserve final quality but can improve it, which the paper attributes to implicit batch-size scheduling with appropriate learning-rate scaling and warmup (Aach et al., 2024).

6. Practical use, limitations, and terminological boundaries

The paper provides explicit guidance for parameterization. The initial number of configurations $\sim 146$7 should saturate early-rung parallelism without overwhelming storage; the example given is 32 configurations on 64 GPUs with 2 GPUs per trial. Geometric milestones with $\sim 146$8 are recommended, and $\sim 146$9 is stated as typical, although smaller $\sim 60$0, such as $\sim 60$1, can reduce premature elimination under noisy metrics (Aach et al., 2024). Base and maximum workers per trial, $\sim 60$2 and $\sim 60$3, should be chosen with respect to interconnect bandwidth, communication overhead, and generalization behavior at large batch sizes.

Several operational constraints are emphasized. A global worker pool must enforce $\sim 60$4. Promotions should be enqueued until GPUs free up, and checkpointing should not be too frequent in order to avoid thrashing; the example checkpoint interval is every 5 epochs. Communication overhead can be mitigated through PyTorch DDP with NCCL, gradient bucketing, compute-communication overlap, mixed precision, and, when necessary, gradient accumulation, although the paper notes that gradient accumulation changes optimization dynamics (Aach et al., 2024). For the data pipeline, DALI or highly tuned PyTorch DataLoaders, shared-memory caching, high-bandwidth file systems, and prefetching are recommended.

The listed pitfalls define the boundary conditions of effective use. Very large batches can degrade generalization; the proposed mitigation is progressive increase only at later rungs, together with learning-rate warmup, regularization, and data augmentation. Asynchrony reduces straggler blocking, but hardware heterogeneity can bias promotion order. I/O bottlenecks can arise if too many trials hit rung boundaries simultaneously or if checkpointing is too frequent (Aach et al., 2024).

The stated limitations are also specific. RASDA is bandit-inspired but does not provide formal regret or optimality guarantees for joint time-plus-space fidelity scheduling. Performance depends on the quality of early metrics, so $\sim 60$5 and rung spacing need task-specific tuning. The paper identifies multi-resource extensions, such as adding memory per GPU, CPU threads, or data fraction, and multi-objective optimization over accuracy and time or cost, as future directions (Aach et al., 2024). It also highlights the question of optimal timing for batch scaling, suggesting adaptive policies based on Gradient Noise Scale or loss curvature as a future avenue (Aach et al., 2024).

Finally, the term should be distinguished from similarly named concepts in other literatures. “Doubly Adaptive Social Learning” introduces two adaptive stages—SGD-based model tracking and adaptive belief updates with forgetting—but is not a worker-doubling scheduler (Carpentiero et al., 24 Apr 2025). Likewise, work on double-team defense in basketball frames “doubling” as a tactical action in an RL policy rather than a resource-allocation rule (Wang et al., 2018). In the HPC and HPO literature, adaptive doubling most precisely denotes successive doubling of data-parallel workers attached to promoted trials, as instantiated by RASDA (Aach et al., 2024).