Successive Halving in Hyperparameter Optimization

• Successive Halving is a multi-fidelity strategy that evaluates many configurations with minimal resources and progressively allocates larger budgets to top performers.
• The method relies on early performance as a proxy for later outcomes, but may prematurely eliminate slow starters due to noisy or delayed learning curves.
• Enhancements like Hyperband, ASHA, and predictive variants expand SH’s applicability by balancing resource allocation, reducing idle time, and addressing multi-objective challenges.

Successive Halving (SH), also called Sequential Halving in the bandit literature, is a multi-fidelity pruning strategy that allocates a small resource budget to many candidates, evaluates them, and repeatedly concentrates larger budgets on a shrinking set of survivors. In hyperparameter optimization, the resource can be epochs, training steps, fraction of data, wall-clock time, or even computational workers; in best-arm identification, it is a fixed sampling budget distributed across elimination rounds. The common structure is an iterative coupling of ranking, elimination, and increased resource allocation, with the aim of improving search efficiency under a fixed budget (Geissler et al., 2024, Sagers et al., 2024).

1. Canonical formulation

In the standard hyperparameter-optimization formulation, SH starts from an initial population of configurations and a minimum resource per configuration. With initial number of configurations $n_0$, initial resource $r_0$, halving rate $\eta > 1$, and round index $t = 0,1,\dots,T$, the canonical schedule is

$$n_t = \left\lfloor \frac{n_0}{\eta^t} \right\rfloor,\qquad r_t = r_0 \cdot \eta^t.$$

At round $t$, each of the $n_t$ configurations is trained for $r_t$ resource units, ranked by a performance metric, and only the top $\lfloor n_t/\eta \rfloor$ survive to the next round. The total budget is

$B = \sum_{t=0}^{T} n_t \cdot r_t,$

and termination occurs when only one configuration remains or when the budget is exhausted (Geissler et al., 2024).

A widely used special case is $r_0$0, which gives literal “halving.” In the formulation emphasized for black-box and hyperparameter optimization, SH begins with a large population, often $r_0$1 instances, trains or evaluates all survivors at each stage with increased resource, and retains only the configurations that beat the population median at that stage. After $r_0$2 stages, $r_0$3 out of $r_0$4 remains (Kumar et al., 2018).

This schedule has a characteristic computational profile. Since $r_0$5, the per-round cost is approximately constant across rounds, so SH redistributes compute from breadth to depth without sharply increasing round cost (Lin et al., 20 Aug 2025). In the bandit setting, the same elimination logic appears in fixed-budget best-arm identification: with $r_0$6 arms and total sample budget $r_0$7, SH allocates samples uniformly within each round, halves the active set by empirical means, and repeats until one arm remains (Sagers et al., 2024).

2. Statistical rationale, assumptions, and failure modes

SH is motivated by the assumption that low-fidelity evaluations are noisy but informative proxies for high-fidelity outcomes. In machine learning this usually means that partial training can predict eventual performance well enough to justify pruning. Under this assumption, SH aggressively explores many candidates at small budgets and then exploits by concentrating resources on promising survivors (Aach et al., 2024).

The approach is therefore strongest when performance improves with resource in expectation and when early rankings correlate reasonably with final rankings. In Monte-Carlo Tree Search, this logic is recast as simple-regret minimization at the root: if only the final recommendation matters, eliminating empirically weak arms can be preferable to cumulative-regret-driven sampling rules such as UCB1 (Sagers et al., 2024). In neural architecture search and hyperparameter optimization, the same reasoning underlies multi-fidelity evaluation, early stopping, and promotion across rungs (Schmucker et al., 2021).

The central limitation is myopic pruning. Standard SH ranks configurations using current intermediate performance values, so it can prematurely eliminate “slow starters” that begin poorly but would later become optimal. This risk is amplified when learning curves exhibit warmup, delayed improvement, plateau phases, or non-stationary behavior (Lin et al., 20 Aug 2025). Aggressive reduction factors increase this risk further: larger $r_0$8 prunes more aggressively, whereas smaller $r_0$9 smooths decisions at higher total cost (Lin et al., 20 Aug 2025).

Other failure modes are setting-dependent. In multi-objective optimization, Pareto dominance can flip across fidelities, so a configuration that appears attractive at low resource may be dominated later (Schmucker et al., 2021). In noisy objectives, weak early signals can make rank-based elimination unstable. In fixed-budget bandit SH, an additional practical limitation is that the total budget must be predetermined, because the number of rounds and per-round allocation depend explicitly on that budget (Sagers et al., 2024).

3. Scheduler family: Hyperband, ASHA, and resource-adaptive scaling

SH is the core scheduling primitive for a broader family of methods. Hyperband wraps SH in multiple brackets with different initial population sizes and per-configuration budgets, thereby spanning a spectrum of schedules and hedging against a poor single choice of breadth versus depth. In contrast to SH’s single schedule, Hyperband balances “how many configurations to try” against “how much resource to spend per configuration,” and is noted for desirable theoretical guarantees about anytime performance under a fixed total budget (Kumar et al., 2018).

ASHA removes the synchronization barrier of classical SH. Rather than waiting for all configurations in a rung to finish, it promotes configurations asynchronously as soon as workers are free, while multiple rungs remain active simultaneously. This reduces straggler effects and improves wall-clock efficiency, especially when evaluation times vary (Schmucker et al., 2021). On high-performance computing systems, the same asynchronous mechanism can be combined with data-parallel training. “Resource-Adaptive Successive Doubling” (RASDA) augments ASHA by doubling the number of workers assigned to promoted trials, so promotions simultaneously increase temporal fidelity and spatial parallelism. On ImageNet, additive manufacturing, and computational fluid dynamics workloads, RASDA was reported to be up to $\eta > 1$0 faster than ASHA while maintaining or surpassing solution quality, and weak scaling showed parallel efficiency above $\eta > 1$1 up to $\eta > 1$2 GPUs (Aach et al., 2024).

The main scheduler variants can be summarized as follows.

Variant	Modification of SH	Reported consequence
Hyperband	Multiple SH brackets with different $\eta > 1$3	Hedges a poor single schedule
ASHA	Asynchronous promotion across rungs	Lower wall-clock idle time
RASDA	Promotion plus worker doubling	Up to $\eta > 1$4 runtime improvement
MO-ASHA	Pareto-based promotion instead of scalar ranking	Better Pareto fronts in less time

These variants preserve SH’s basic elimination-and-promotion structure, but they change the systems assumptions under which it operates. Hyperband broadens scheduling choices, ASHA removes synchronization, and RASDA treats computational workers themselves as a fidelity dimension rather than merely an execution substrate (Aach et al., 2024).

4. Learned, non-uniform, and predictive extensions

A major line of work replaces or augments SH’s direct ranking rule. “Successive Halving and Classification” (SHAC) keeps the core idea of culling half the candidates per stage, but it eliminates resource-based halving of individual configurations. Instead, it trains a cascade of binary classifiers to reject undesirable regions of the search space. With classifiers $\eta > 1$5 and acceptance region

$\eta > 1$6

new points are sampled from a prior $\eta > 1$7 by rejection sampling and evaluated only if they pass the cascade. Each classifier is trained on stage data binarized by the median objective value, so the accepted region shrinks by approximately a factor of $\eta > 1$8 per stage. SHAC is scale-invariant because the labeling depends only on relative ordering within a stage, not on the absolute scale of the objective, and the paper uses XGBoost with $\eta > 1$9 trees, $t = 0,1,\dots,T$0, and a classifier-adoption rule requiring $t = 0,1,\dots,T$1-fold cross-validation accuracy of at least $t = 0,1,\dots,T$2 (Kumar et al., 2018). Empirically, SHAC consistently beats RS-2X across synthetic, hyperparameter, and many architecture-search settings, and is competitive with more specialized NAS methods (Kumar et al., 2018).

A different departure is non-uniform supervision. In “RANK-NOSH,” non-uniform successive halving (NOSH) maintains a hierarchical pyramid of architectures trained to different budgets rather than a uniform fixed-budget population. Candidates that are not promoted remain in the pool and continue to provide supervision; previously terminated candidates may even be re-promoted if they later rank highly relative to new proposals. A train-free level-0 based on proxies such as synflow “mag” can cheaply filter candidates before any training. Because labels are censored and budget-dependent, the predictor is trained by pairwise ranking rather than regression on final accuracies. On NAS-Bench-201, DARTS, and NAS-Bench-101, the method reduced search cost by about $t = 0,1,\dots,T$3 while achieving competitive or better results relative to prior predictor-based NAS methods (Wang et al., 2021).

Predictive guidance can also be inserted directly into SH. “Successive Halving with Learning Curve Prediction via Latent Kronecker Gaussian Processes” replaces myopic rung ranking by uncertainty-aware predictions of final performance from partial learning curves. The latent Kronecker GP uses a separable kernel over hyperparameters and time,

$t = 0,1,\dots,T$4

with covariance

$t = 0,1,\dots,T$5

and ranks candidates by an “expected wins” criterion computed from predictive means and variances of final-window performance. This directly targets SH’s slow-starter problem. However, the study found that predictive SH, although competitive, was not Pareto optimal relative to simply investing the same compute into standard SH, because the predictive model requires fully observed learning curves as training data. The practical implication given is that the approach becomes more attractive when historical curve repositories already exist (Lin et al., 20 Aug 2025).

5. Multi-objective and energy-aware formulations

Classical SH assumes a scalar ranking metric, but the same halving logic extends to vector-valued objectives. “Multi-objective Asynchronous Successive Halving” (MO-ASHA) generalizes ASHA by replacing scalar ranking with Pareto-aware promotion rules. Within each rung, candidates are selected either by non-dominated sorting plus NSGA-II crowding distance or by non-dominated sorting plus EpsNet farthest-point selection. Promotion therefore reflects the geometry of the current Pareto-front approximation rather than a scalarized score. Across neural architecture search, algorithmic fairness, and language-model optimization, Pareto-front-based promotion consistently outperformed scalarization-based multi-fidelity HPO in wall-clock time, and the resulting schedulers established new baselines for scalable multi-objective HPO (Schmucker et al., 2021).

Energy-aware SH changes the objective rather than the fidelity structure. “Spend More to Save More” (SM2) augments SH with an exploratory pretraining stage that performs one epoch over approximately $t = 0,1,\dots,T$6 of the dataset, measures live energy consumption on the target hardware, probes learning-rate stability with a cyclical schedule, and prunes configurations using the multi-objective score

$t = 0,1,\dots,T$7

where $t = 0,1,\dots,T$8 is normalized performance, $t = 0,1,\dots,T$9 is normalized energy per epoch, and $$n_t = \left\lfloor \frac{n_0}{\eta^t} \right\rfloor,\qquad r_t = r_0 \cdot \eta^t.$$0 encodes the largest stable learning rate. The implementation integrates Carbontracker $$n_t = \left\lfloor \frac{n_0}{\eta^t} \right\rfloor,\qquad r_t = r_0 \cdot \eta^t.$$1 with NVIDIA SMI and executes configurations sequentially because SMI reports only total GPU power rather than per-process attribution. In experiments on ResNet-18/CIFAR-10, an LSTM on an Energy-Household dataset, and a Transformer on WikiText2, SM2 reduced energy by $$n_t = \left\lfloor \frac{n_0}{\eta^t} \right\rfloor,\qquad r_t = r_0 \cdot \eta^t.$$2–$$n_t = \left\lfloor \frac{n_0}{\eta^t} \right\rfloor,\qquad r_t = r_0 \cdot \eta^t.$$3 relative to performance-only selection with $$n_t = \left\lfloor \frac{n_0}{\eta^t} \right\rfloor,\qquad r_t = r_0 \cdot \eta^t.$$4. The ResNet and LSTM cases matched final performance, whereas the Transformer showed about a $$n_t = \left\lfloor \frac{n_0}{\eta^t} \right\rfloor,\qquad r_t = r_0 \cdot \eta^t.$$5 percentage-point perplexity trade-off between $$n_t = \left\lfloor \frac{n_0}{\eta^t} \right\rfloor,\qquad r_t = r_0 \cdot \eta^t.$$6 and $$n_t = \left\lfloor \frac{n_0}{\eta^t} \right\rfloor,\qquad r_t = r_0 \cdot \eta^t.$$7 (Geissler et al., 2024).

These extensions make explicit that “successive halving” is not tied to a single notion of utility. The halving step can be governed by Pareto dominance, energy-aware composite scores, or other structured ranking rules, provided that lower-fidelity observations remain sufficiently informative for promotion decisions (Schmucker et al., 2021).

6. Broader applications, terminology, and conceptual generalizations

Outside hyperparameter optimization, the term appears in several related but distinct settings. In best-arm identification for Monte-Carlo Tree Search, Sequential Halving is used at the root to minimize simple regret rather than cumulative regret. The fixed-budget version requires a predetermined iteration budget, so “Anytime Sequential Halving” was proposed as an interruptible variant that repeatedly performs minimal SH passes: sample each surviving arm $$n_t = \left\lfloor \frac{n_0}{\eta^t} \right\rfloor,\qquad r_t = r_0 \cdot \eta^t.$$8 times, halve by empirical mean, double $$n_t = \left\lfloor \frac{n_0}{\eta^t} \right\rfloor,\qquad r_t = r_0 \cdot \eta^t.$$9, and restart from the full arm set once a pass ends. On synthetic multi-armed bandit problems and ten board games, the anytime version was competitive with both fixed-budget Sequential Halving and UCB1-based baselines (Sagers et al., 2024).

In differentiable optimization, “Successive Halving Top-k” uses a tournament-style soft selection to approximate the non-differentiable top-$t$0 operator. At each round, scores are sorted, paired, and merged by pairwise boosted softmax, so the number of candidates shrinks geometrically until $t$1 remain. This avoids repeated full-length softmax evaluations and yields a continuous relaxed selection mask derived from pathwise survival probabilities. The paper reports lower computational cost and better approximation quality, measured by normalized Chamfer Cosine Similarity, than an iterative softmax baseline, especially for larger $t$2 (Pietruszka et al., 2020).

A more distant usage occurs in stochastic geometry. In the iterated Rényi parking process, car length is repeatedly halved across stages, and the interval is re-jammed at each new scale. There, “successive halving” refers to geometric halving of object size rather than budget pruning. The uncovered length decays exponentially with limiting ratio

$t$3

so the process exhausts the interval asymptotically (Mackey et al., 2016).

Across these uses, the essential abstraction is recursive elimination under a geometric schedule. In HPO and NAS, what is halved is the active set of configurations; in differentiable top-$t$4, it is the candidate list under soft pairwise tournaments; in Rényi parking, it is the physical scale of the objects themselves. The algorithmic family is therefore best understood not as a single procedure, but as a general scheme in which geometric reduction is coupled to a fidelity, scoring, or scale update (Kumar et al., 2018).