DeepHyper: Scalable HPO for HPC

Updated 4 July 2026

DeepHyper is an open-source Python framework designed for hyperparameter and black-box optimization in computationally expensive scientific workloads.
It employs advanced methods such as asynchronous Bayesian optimization, random-forest surrogates, and manager–worker execution models to efficiently guide search.
DeepHyper integrates with full-scale workflows like HydraGNN and PhysNet, orchestrating large-scale distributed trials to optimize complex objectives.

Searching arXiv for DeepHyper and related application papers to ground the article. DeepHyper is an open-source scalable Python package and optimization framework originally developed for hyperparameter optimization of machine learning methods and later used as a substrate for asynchronous model-based search, Bayesian optimization, and multiobjective optimization over expensive black-box workflows in scientific computing and HPC (Dorier et al., 2022, Madireddy et al., 2019, Sun et al., 2024). In the literature, DeepHyper is typically not the domain model or training stack itself: it sits above application codes such as HydraGNN, PhysNet, Fourier neural operators, storage-service workflows, or neuromorphic recurrent networks, proposes configurations, launches evaluations in parallel, and uses returned objective values to guide further search (Pasini et al., 22 May 2026, Pasini et al., 15 Apr 2026, Park et al., 2022).

1. Definition and operational scope

DeepHyper is used in published work as an HPO and black-box optimization layer for computationally expensive workloads rather than as a general-purpose model-training framework. In the smart-grid OPF study, it is explicitly “the mechanism used for large-scale hyperparameter optimization of heterogeneous graph neural networks inside a broader HydraGNN-based workflow,” while HydraGNN remains “the model/training system” and DeepHyper is confined to the model-selection stage on Frontier (Pasini et al., 22 May 2026). The same division of labor appears in the atomistic graph foundation model workflow, where DeepHyper is the “orchestration and search layer” upstream of HydraGNN production runs, and in the molecular-property framework, where it provides hyperparameter tuning inside a larger stack that also includes training, inference, interpretability, and experiment logging (Pasini et al., 15 Apr 2026, Park et al., 2022).

This operational scope gives DeepHyper a characteristic position in scientific-ML pipelines. A typical study defines a search space, maps each sampled configuration to a full training or simulation job, and returns a scalar or vector objective to the optimizer. Published examples range from distributed GNN training over millions of heterogeneous graphs, to autotuning storage-service parameters for coupled HPC workflows, to searching over local plasticity rules in neuromorphic learning systems (Pasini et al., 22 May 2026, Dorier et al., 2022, Madireddy et al., 2019).

2. Optimization methods documented in the literature

Published applications describe several distinct DeepHyper search modes. In smart-grid OPF surrogate modeling, the method is specified as “DeepHyper Centralized Bayesian Optimization with an Upper Confidence Bound acquisition function,” with the minimum validation loss across completed epochs used as the HPO objective rather than the final-epoch loss (Pasini et al., 22 May 2026). In HPC storage autotuning, DeepHyper provides asynchronous Bayesian optimization with a manager–workers loop, a random-forest surrogate, lower confidence bound ranking, and a constant-liar strategy for multi-point proposal generation; the objective is transformed to $f_{\text{DH}}(x) = -\log(\text{runtime}(x))$ because DeepHyper maximizes by default (Dorier et al., 2022).

Other studies expose additional search backends. The neuromorphic architecture-optimization paper uses DeepHyper’s asynchronous model-based search with a random-forest surrogate and a hedging strategy over acquisition functions, searching jointly over a categorical rule family $\Psi \in \{\text{GMR,MCR,NSCR,LMSR,SLR,GUR,NSCoR,MOR}\}$ and continuous parameters $\alpha,\beta_1,\beta_2,\beta_3$ (Madireddy et al., 2019). The ocean-dynamics study uses DeepHyper for centralized parallel Bayesian optimization with a tree-based surrogate, $q$ UCB multipoint acquisition, and randomized scalarization for two-objective search over negative validation MSE and validation ACC (Sun et al., 2024).

The objective interface is correspondingly heterogeneous. In PhysNet optimization, the reported objective is the negated validation error because “DeepHyper tries to maximize the objective of search” (Park et al., 2022). In OPF GNN selection, the optimized quantity is the best validation MSE-like supervised loss reached during a 10-epoch budget (Pasini et al., 22 May 2026). In storage-service autotuning, the search target is end-to-end workflow runtime (Dorier et al., 2022). This diversity shows that DeepHyper is used as a general black-box optimizer over workflow-defined metrics rather than over a fixed built-in loss.

3. Execution model and distributed systems behavior

Application papers commonly describe DeepHyper through centralized or manager–worker execution patterns. In the HEPnOS study, the manager maintains the search state and surrogate model, while worker nodes asynchronously execute workflow instances and return results as soon as they complete (Dorier et al., 2022). In the OPF and atomistic HydraGNN studies, each DeepHyper trial is itself a large distributed training job, and new proposals are launched as resources free up; in the exascale atomistic paper, “New trials were proposed and launched as soon as running trials completed” (Pasini et al., 22 May 2026, Pasini et al., 15 Apr 2026).

A later systems paper characterizes DeepHyper more broadly as one of the Python frameworks that “support decentralized operation by coordinating workers through a shared database,” alongside Optuna, Hyperopt, and Orion (Becker et al., 19 Jun 2026). That description is architectural rather than benchmark-based: the same paper explicitly states that it does not provide a direct empirical comparison with DeepHyper (Becker et al., 19 Jun 2026). Taken together, the literature presents DeepHyper as a framework that supports asynchronous, parallel optimization and is routinely embedded in HPC execution environments, while specific applications may expose that capability through centralized Bayesian optimization campaigns or analogous manager–worker abstractions (Dorier et al., 2022, Becker et al., 19 Jun 2026).

Operationally, DeepHyper is often coupled to experiment-management functions adjacent to HPO. The OPF workflow reports trial-level fault isolation and CSV-based result saving and warm-starting (Pasini et al., 22 May 2026). The molecular-property framework reports checkpointed search that can be resumed with a --resume flag and automatic logging to Weights & Biases (Park et al., 2022). These are not generic workflow-orchestration claims for every use case, but they are recurrent patterns in published deployments.

4. Representative application domains

DeepHyper has been used across scientific ML, HPC systems tuning, and neural architecture/configuration search.

Domain	DeepHyper role	Representative scale or target
OPF surrogate modeling on HydraGNN	Six architecture-specific HPO campaigns over `hidden_dim`, `num_conv_layers`, and `learning_rate`	Three million heterogeneous graph instances; 512 Frontier nodes per HPO run; 32 nodes per trial; 387 dispatched trials (Pasini et al., 22 May 2026)
Exascale atomistic GFMs on HydraGNN	Backbone-specific campaigns for EGNN, SchNet, DimeNet, MACE, PaiNN, and PNAEq	1,024 Frontier nodes per HPO run; 256 nodes per trial; 574 explored configurations; 181 successful 20-epoch trials (Pasini et al., 15 Apr 2026)
HPC storage-service autotuning	Asynchronous BO with TVAE-guided transfer learning	20 parameters; 128 Theta nodes as workers; 51,830 workflow evaluations across 115 CSV files (Dorier et al., 2022)
Neuromorphic dynamic learning	AMBS over learning-rule family and rule coefficients	128 Theta nodes per dataset; separate runs for MNIST and FashionMNIST (Madireddy et al., 2019)
Molecular-property prediction	Centralized Bayesian HPO within an end-to-end AI framework	Detailed PhysNet tuning on QM9 HOMO and ZPVE; deployment on ThetaGPU and Delta (Park et al., 2022)
Ocean-dynamics forecasting	Multiobjective HPO/NAS over preprocessing, FNO architecture, and training	20 Polaris nodes with 4 Nvidia A100 GPUs per node (Sun et al., 2024)

Across these domains, the evaluated workload is usually expensive enough that each DeepHyper suggestion corresponds to a full application execution rather than a lightweight proxy. In the OPF and atomistic HydraGNN studies, one trial is a large multi-node distributed GNN run (Pasini et al., 22 May 2026, Pasini et al., 15 Apr 2026). In HEPnOS autotuning, each evaluation launches the full storage service and application workflow (Dorier et al., 2022). In ocean modeling, each trial trains an FNO under early stopping on GPU clusters (Sun et al., 2024). This common pattern explains why DeepHyper appears most frequently in HPC-oriented papers rather than in small-scale notebook-style tuning studies.

5. Search spaces and objective design

A recurring feature of DeepHyper use is explicit search over mixed discrete, continuous, and categorical spaces. In the OPF campaign, the full reported search space is hidden_dim in $[32,256]$ , num_conv_layers in $[2,6]$ , and learning_rate log-uniform in $[10^{-5},10^{-2}]$ , with mpnn_type fixed within each of six architecture-specific runs (Pasini et al., 22 May 2026). In the HEPnOS autotuning paper, the search space spans 20 parameters across the storage service, loader, and application, with 3 ordinal, 8 categorical, and 9 range-based variables, totaling about $1.5\times 10^{23}$ distinct configurations (Dorier et al., 2022). In the neuromorphic study, the categorical choice of plasticity rule is flattened together with continuous coefficients, even though different rules consume different subsets of $\beta_1,\beta_2,\beta_3$ (Madireddy et al., 2019).

Scientific-ML uses show the same pattern. The PhysNet study tunes agb, amp, batch size, epochs, gradient clip, learning rate, optimizer, and weight decay, with discrete variables sampled from uniform distributions and continuous variables from normal distributions with or without log scale (Park et al., 2022). The ocean FNO paper partitions the decision vector as $p=(p_D,p_N,p_T)$ , covering preprocessing (padding, padding_type, coord_feat), architecture (num_FNO, num_latent_feat, num_modes, projection settings, activation choices), and training (alpha, optimizer, lr, weight_decay, batch_size) (Sun et al., 2024).

The objective functions are equally application-specific. Storage tuning maximizes $\Psi \in \{\text{GMR,MCR,NSCR,LMSR,SLR,GUR,NSCoR,MOR}\}$ 0 (Dorier et al., 2022). PhysNet tuning maximizes the negated validation error (Park et al., 2022). The OPF GNN campaign minimizes the minimum validation loss reached during short-horizon training, specifically to cope with oscillatory training and early overfitting (Pasini et al., 22 May 2026). The ocean FNO workflow separates inner-loop training loss from outer-loop HPO objectives: model training uses a weighted combination of MSE and negative ACC, while DeepHyper performs two-objective search over validation MSE and validation ACC via randomized scalarization (Sun et al., 2024). This suggests that DeepHyper is best understood as an optimizer over externally defined evaluation functions rather than over a standardized internal model interface.

6. Documented outcomes, limitations, and reproducibility

The outcomes attributed to DeepHyper are typically selection or autotuning results rather than end-to-end scientific claims by themselves. In OPF surrogate modeling, the HPO campaign identified HeteroSAGE and HeteroHEAT as the leading architectures, with compact models of about 1.6M and 1.7M parameters and “essentially identical” best validation losses; it also exposed large architecture-dependent failure rates, including only 2 valid HeteroPNA trials out of 115 dispatched (Pasini et al., 22 May 2026). In exascale atomistic modeling, DeepHyper campaigns over six backbones promoted a PaiNN configuration that became the lead model and was the only promoted backbone to reach 100 epochs within fixed 2,048-node production runs (Pasini et al., 15 Apr 2026).

In workflow autotuning, the most explicit quantitative claim is the HEPnOS study’s report that VAE-guided DeepHyper achieved more than $\Psi \in \{\text{GMR,MCR,NSCR,LMSR,SLR,GUR,NSCoR,MOR}\}$ 1 search speedup over random search, compared with $\Psi \in \{\text{GMR,MCR,NSCR,LMSR,SLR,GUR,NSCoR,MOR}\}$ 2 to $\Psi \in \{\text{GMR,MCR,NSCR,LMSR,SLR,GUR,NSCoR,MOR}\}$ 3 without transfer learning, while also comparing favorably to GPtune and HiPerBOt in resource utilization and parallelization (Dorier et al., 2022). In PhysNet tuning, DeepHyper-guided search outperformed naive random selection on both QM9 targets in a 20-model comparison, reducing min_loss from 6.632 to 1.604 for HOMO and from 1.751 to 0.923 for ZPVE (Park et al., 2022). In ocean forecasting, DeepHyper-selected FNO configurations improved one-step prediction and, more importantly, produced substantially more stable autoregressive rollouts up to about 30 days than the default Modulus baseline (Sun et al., 2024).

The literature also documents recurrent limitations. Application papers often omit the exact best hyperparameter tuples, DeepHyper version, launch scripts, random seeds, or full software environment; the OPF and exascale atomistic HydraGNN studies are especially explicit about these gaps (Pasini et al., 22 May 2026, Pasini et al., 15 Apr 2026). Several studies rely on short-horizon proxy budgets—10 epochs in OPF, 20 epochs on 10% of each dataset in the atomistic campaign, 30 epochs with aggressive stopping in ocean FNO search—which means DeepHyper is optimizing a truncated proxy for final training quality rather than the full training regime (Pasini et al., 22 May 2026, Pasini et al., 15 Apr 2026, Sun et al., 2024). And although DeepHyper is often chosen because of asynchronous parallelism and scalability, some papers do not report wall-clock search overheads, acquisition diagnostics, or direct head-to-head comparisons against alternative optimizers (Madireddy et al., 2019, Park et al., 2022).

These limits shape the most defensible generalization. DeepHyper is well documented as an HPC-capable optimization framework for expensive black-box evaluations, and the application record shows it functioning as an effective model-selection or autotuning layer across smart grids, materials discovery, storage systems, molecular modeling, ocean dynamics, and neuromorphic learning. The same record also shows that many published studies are application papers first and DeepHyper methods papers second: they establish that the framework scales to demanding workloads, but they do not always expose every low-level detail needed for exact reruns (Pasini et al., 22 May 2026, Pasini et al., 15 Apr 2026, Dorier et al., 2022).