ASHA: Asynchronous Successive Halving

• ASHA is a hyperparameter optimization algorithm that employs asynchronous successive halving to allocate resources efficiently and terminate underperforming runs early.
• It removes synchronization barriers in distributed environments, allowing dynamic promotion of configurations and near-linear scalability across parallel workers.
• ASHA’s extensions, such as MO-ASHA, integrate geometry-based and scalarization techniques to manage multi-objective trade-offs in high-performance computing.

The Asynchronous Successive Halving Algorithm (ASHA) is a parallelizable, early-stopping hyperparameter optimization (HPO) algorithm designed to efficiently allocate computational resources to promising model configurations and terminate suboptimal runs early. It extends classical Successive Halving to asynchronous, lock-free distributed settings, achieving near-optimal wall-clock efficiency, and has been generalized for multi-objective optimization. ASHA forms the core of several state-of-the-art HPO systems for both academic research and production ML platforms (Li et al., 2018, Schmucker et al., 2021, Aach et al., 2024).

1. Foundations and Motivation

Successive Halving (SHA) begins by evaluating a large number of random hyperparameter configurations on a minimal computational budget (e.g., epochs or data samples). After evaluation, the worst configurations are terminated, and only a fraction (commonly $1/\eta$ for reduction factor $\eta$) are promoted to continue with increased budgets. This culling-and-promotion continues until the maximum budget $R$ is reached. The drawback of classical SHA in distributed environments is its synchronization barrier: each stage ("rung") waits for all jobs to finish before further promotion, leading to significant wall-clock inefficiency due to stragglers or lost jobs.

ASHA addresses this by completely removing synchronization barriers: as soon as any job completes, it is evaluated for promotion independent of other jobs, and compute resources are reassigned without waiting for a full cohort at any rung. This enables aggressive early stopping, fine-grained resource allocation, and near-linear scalability with the number of parallel workers. ASHA is applicable when the hyperparameter space is large, per-run cost is significant, and the system architecture supports distributed, asynchronous execution (Li et al., 2018, Aach et al., 2024).

2. Core Algorithmic Structure

The core parameters of ASHA are the minimum resource $r_0$, maximum resource $R$, and reduction factor $\eta>1$. The number of rungs $K$ is set as $K=\lfloor\log_\eta(R/r_0)\rfloor$. At each rung $k=0,\ldots,K$, a configuration $\theta$ is evaluated on resource $r_0\eta^k$. Promotion decisions are made as soon as $\eta$ or more configurations have completed at a rung: among them, only the top fraction $1/\eta$ (by minimum loss or error) are eligible for promotion to the next rung.

In the asynchronous loop, whenever a worker becomes free, it is assigned either (i) the oldest promotable configuration from the lowest-possible rung, or (ii) a new random configuration at the base rung if no promotions are possible. Thus, the scheduler is constantly filling all compute resources with productive work without global synchronization (Schmucker et al., 2021, Li et al., 2018, Aach et al., 2024).

3. Theoretical Properties and Complexity

ASHA's asynchronous design achieves convergence to the same set of (Pareto-)optimal configurations as synchronous SHA, under mild conditions, with notable wall-clock advantages in parallel/distributed settings. Let $T(R)$ be the time required to train one configuration fully; then ASHA can promote a configuration through all rungs in approximately

$$\left(\sum_{i=0}^K \eta^{-i}\right)T(R) \leq \frac{\eta}{\eta-1}T(R) \approx 2T(R)$$

when there are sufficient workers to saturate all rungs. This represents nearly ideal parallel efficiency up to the maximal width of the lowest rung (Li et al., 2018, Aach et al., 2024).

The algorithm's core operations—searching for promotable configurations, managing rung lists, and maintaining asynchronous state—are computationally lightweight. Checkpointing and exact promotion sequence logging are practical measures for reproducibility and efficient production-level deployment (Li et al., 2018).

4. Multi-Objective ASHA (MO-ASHA)

MO-ASHA generalizes ASHA from single-objective to multi-objective settings where the goal is to estimate the Pareto front of a vector-valued objective $f(\theta)\in\mathbb{R}^n$ (to be minimized). The algorithms provide alternative candidate-selection mechanisms for promotion:

• Scalarization schemes (Random-weights, ParEGO, Golovin) project multi-objective results into a scalar via weight vectors sampled from the simplex, promoting configurations scoring best across sampled directions.
• NSGA-II (non-dominated sorting + crowding distance) assigns each configuration to Pareto fronts and ranks them by crowding distance within the fronts.
• EpsNet (ε-net covering) covers the evaluated set by greedily selecting maximally separated points from each non-dominated front.

Key definitions include weak/strict Pareto domination and the dominated hypervolume (volume beneath the Pareto front relative to a given reference point). These selection policies are modular within ASHA, and empirical results show that geometry-based selectors (NSGA-II, EpsNet) consistently outperform scalarization approaches, especially for non-convex fronts and when the objective scales are heterogeneous. Scalarization can entirely overlook nonconvex regions of the Pareto front (Schmucker et al., 2021).

5. Large-Scale and High-Performance Applications

ASHA has demonstrated scaling to hundreds or thousands of parallel workers, with empirical studies reporting linear speedup up to the maximum width of the lowest rung. For example, on neural architecture search benchmarks with 16–500 workers, ASHA finds low-error architectures in up to $10\times$ less wall-clock time than serial or less-parallelized methods (Li et al., 2018). With scientific workloads on terabyte-scale datasets, ASHA's efficiency remains robust, though newer extensions such as Resource-Adaptive Successive Doubling Algorithms (RASDA) can outperform ASHA by up to $1.9\times$ in runtime on up to 1,024 GPUs, while retaining or improving final solution quality (Aach et al., 2024).

A comparative table highlighting results from large-scale empirical studies:

Setting	Method	Runtime Speed-up	Best Model Quality
ResNet50/ImageNet/64GPU	ASHA	1×	0.6688 (test acc.)
ResNet50/ImageNet/64GPU	RASDA	1.71×	0.6766 (test acc.)
SwinTransf./AM/128GPU	ASHA	1×	0.0554 (test MSE)
SwinTransf./AM/128GPU	RASDA	1.52×	0.0516 (test MSE)
CAE/CFD/128GPU	ASHA	1×	4.42×10⁻⁶ (test MSE)
CAE/CFD/128GPU	RASDA	1.90×	2.40×10⁻⁶ (test MSE)

RASDA, built on ASHA, expands resource allocation to include both "halving in time" (iteration budget) and "doubling in space" (number of workers per trial) at each promotion, exploiting gradient noise scale theory and improving utilization in HPC contexts (Aach et al., 2024).

6. Empirical Evaluation and Benchmarks

ASHA and MO-ASHA have been evaluated on diverse and challenging benchmarks:

• Neural Architecture Search (NAS-201): ASHA and MO-ASHA recover near-true Pareto fronts (validation error vs. inference time) faster and with smaller wall-clock budgets compared to synchronous baselines or single-objective methods.
• Algorithmic Fairness (UCI Adult, MLP): Geometry-based MO-ASHA selectors achieve equal or superior results under multiple fairness constraints (statistical parity, equalized odds), outperforming both scalarization-based selectors and state-of-the-art fairness solvers.
• Transformer LLMs (WikiText-2): EpsNet-based MO-ASHA finds trade-offs between perplexity and inference time, particularly under limited compute (Schmucker et al., 2021).

Empirical guidance recommends reduction factor $\eta=3$–5, small minimum budget $r_0$ for early filtering, normalizing objectives to $[0,1]$, and geometry-based selectors over scalarization when the number of objectives is small or when the Pareto surface is nonconvex.

7. Recommendations and Practical Considerations

Recommended deployment scenarios for ASHA:

• Large HPO spaces where the number of configurations greatly exceeds available workers.
• Expensive training per configuration, and full-cluster distributed setups.
• Highly heterogeneous job completion times, where lock-free scheduling minimizes idle time.

ASHA's practical implementations (e.g., Determined AI) expose only high-level controls to the user ($n$, $R$), with aggressive but robust defaults for $\eta$ and early-stopping sensitivity. Checkpointing and promotion sequence logging allow for full reproducibility and production-grade robustness (Li et al., 2018).

For MO-ASHA, geometry-based selectors (ε-net, NSGA-II) are recommended when $n \leq 3$; for $n > 3$, NSGA-II's crowding distance is preferred. Scalarization requires a sufficient variety of weight vectors to achieve reasonable Pareto coverage, but is not robust to nonconvexities (Schmucker et al., 2021).

ASHA remains the standard asynchronous foundation for early-stopping parallel HPO, and its generalizations define new baselines for scalable multi-objective hyperparameter search in modern ML and scientific computing (Li et al., 2018, Schmucker et al., 2021, Aach et al., 2024).