Joint Fidelity Hyperparameter Optimization

• JF-HPO is a framework that jointly optimizes hyperparameters and fidelity dimensions (e.g., model size, training steps) to improve efficiency.
• The method integrates Bayesian optimization, fidelity-aware scheduling, early stopping, and checkpoint reuse to navigate the configuration–fidelity space.
• Empirical results demonstrate up to 14.9× computational efficiency gains with competitive or improved model accuracy across benchmarks.

Joint Fidelity Hyperparameter Optimization (JF-HPO) denotes a class of hyperparameter optimization procedures in which fidelity is not restricted to a single externally fixed budget schedule, but is coupled to hyperparameter search itself. In the named formulation introduced for LLM reinforcement learning, JF-HPO “simultaneously adapts both model size and training budget as fidelity,” using a smaller in-family proxy model at lower fidelities and the target model only at the highest fidelity (Chen et al., 2 Jun 2026). More broadly, the surrounding literature supports a wider interpretation in which fidelity may be a budget hyperparameter embedded in the query space, a scheduler-controlled resource dimension, or a multi-factor workload descriptor; this suggests that JF-HPO is best understood as an optimization perspective centered on joint reasoning over configuration quality and evaluation fidelity rather than as a single fixed algorithmic template (Pfisterer et al., 2021, Zela et al., 2018).

1. Definition and conceptual boundaries

In the 2026 LLM-RL formulation, the hyperparameter optimization problem is written as a budget-constrained search over configurations $\phi \in \Phi$: $$\phi^{*} = \arg\max_{\phi \in \Phi} f(\theta, \phi), \text{s.t.} \sum_{i=1}^{N} C(\theta,\phi_{i}, r_{i}) < \mathcal{B},$$ where $C(\theta,\phi_i,r_i)$ is the time cost of trial $i$ at fidelity $r_i$, and fidelity is realized jointly through training steps and model size. The target model is denoted by $\theta$, a smaller proxy model by $\theta'$, and the training-budget fidelity by $r \in \{r_1,r_2,\ldots,r_{\max}\}$ (Chen et al., 2 Jun 2026).

Across related work, however, “jointness” is not uniform. Some systems expose fidelity directly as a queryable input, for example as a “budget hyperparameter” in a surrogate benchmark over $\Lambda$, so that lower-fidelity queries are generated by varying the budget parameter itself (Pfisterer et al., 2021). Other methods couple configurations and fidelities procedurally through Hyperband or Successive Halving, where the search object is the configuration and the scheduler controls which fidelity is assigned next (Zela et al., 2018). This suggests two technically distinct senses of JF-HPO: a stronger one in which the optimizer reasons over a joint configuration–fidelity domain, and a weaker one in which fidelity is coupled to search through promotion, pruning, or bracket allocation.

2. Antecedents in joint architecture–hyperparameter search

An important precursor is the argument that neural architecture search is a special case of hyperparameter optimization and should therefore be carried out jointly with ordinary training hyperparameters rather than in a two-stage “NAS first, tuning later” pipeline. One early study states that “architectural choices and other hyperparameter settings interact in a way that can render this separation suboptimal,” and further shows that using very small budgets during search and much larger budgets during post-processing is inefficient because low- and high-budget rankings may diverge sharply. In its CIFAR-10 study, the Spearman rank correlations of validation error were $0.87$ between 400 s and 1200 s, $$\phi^{*} = \arg\max_{\phi \in \Phi} f(\theta, \phi), \text{s.t.} \sum_{i=1}^{N} C(\theta,\phi_{i}, r_{i}) < \mathcal{B},$$0 between 400 s and 1 h, and $$\phi^{*} = \arg\max_{\phi \in \Phi} f(\theta, \phi), \text{s.t.} \sum_{i=1}^{N} C(\theta,\phi_{i}, r_{i}) < \mathcal{B},$$1 between 400 s and 3 h, while adjacent fidelities were much more correlated. The same work represented architecture and training choices inside one mixed search space and used BOHB to search jointly over 10 architecture variables and 7 training hyperparameters under budgets of 400 s, 1200 s, 1 h, and 3 h (Zela et al., 2018).

Benchmarking work reinforced this joint-search perspective. A tabular benchmark for feed-forward networks on UCI regression tasks explicitly combined four architectural choices with five training-pipeline hyperparameters and additionally exposed full learning curves “as an additional fidelity” with epochs as the budget, making it possible to study rank correlation and heteroscedastic noise across budgets in a controlled setting (Klein et al., 2019). Later, a multi-objective baseline suite for joint NAS+HPO formalized search over $$\phi^{*} = \arg\max_{\phi \in \Phi} f(\theta, \phi), \text{s.t.} \sum_{i=1}^{N} C(\theta,\phi_{i}, r_{i}) < \mathcal{B},$$2 with objectives such as classification performance and number of parameters, and extended methods such as BOHB, EHVI-based Bayesian optimization, and evolutionary search to the joint architecture–hyperparameter setting, sometimes with explicit epoch budgets up to 25 epochs (Guerrero-Viu et al., 2021).

3. Fidelity as a multi-dimensional design space

The modern JF-HPO literature treats fidelity as richer than “number of epochs.” In LLM reinforcement learning, the defining 2026 formulation uses only two discrete model-size fidelities—proxy and target—together with multiple training-step budgets, so that the optimizer asks not only how long a trial should run but also on which scale of model it should be evaluated (Chen et al., 2 Jun 2026). In benchmark design, YAHPO Gym places fidelity inside the input space itself through a budget hyperparameter such as epoch or trainsize, and LLMSYS-HPOBench goes further by exposing $$\phi^{*} = \arg\max_{\phi \in \Phi} f(\theta, \phi), \text{s.t.} \sum_{i=1}^{N} C(\theta,\phi_{i}, r_{i}) < \mathcal{B},$$3–$$\phi^{*} = \arg\max_{\phi \in \Phi} f(\theta, \phi), \text{s.t.} \sum_{i=1}^{N} C(\theta,\phi_{i}, r_{i}) < \mathcal{B},$$4 fidelity dimensions per system, yielding $$\phi^{*} = \arg\max_{\phi \in \Phi} f(\theta, \phi), \text{s.t.} \sum_{i=1}^{N} C(\theta,\phi_{i}, r_{i}) < \mathcal{B},$$5 fidelity settings across real-world LLM systems (Pfisterer et al., 2021, Wu et al., 8 May 2026).

Source	Fidelity variables	Salient role
JF-HPO for LLM RL	model size, training steps	proxy/target dual fidelity
Frozen Layers	trainable layers, training tokens	compute and memory fidelity
YAHPO Gym	epoch or trainsize	budget hyperparameter in query space
LLMSYS-HPOBench	3–5 task/workload factors	multi-factor, non-monotonic fidelity

A particularly important extension is “Frozen Layers,” which proposes the number of trained versus frozen layers as a fidelity source. Given a model with $$\phi^{*} = \arg\max_{\phi \in \Phi} f(\theta, \phi), \text{s.t.} \sum_{i=1}^{N} C(\theta,\phi_{i}, r_{i}) < \mathcal{B},$$6 layers and desired trainable depth $$\phi^{*} = \arg\max_{\phi \in \Phi} f(\theta, \phi), \text{s.t.} \sum_{i=1}^{N} C(\theta,\phi_{i}, r_{i}) < \mathcal{B},$$7, the method freezes the first $$\phi^{*} = \arg\max_{\phi \in \Phi} f(\theta, \phi), \text{s.t.} \sum_{i=1}^{N} C(\theta,\phi_{i}, r_{i}) < \mathcal{B},$$8 layers and trains the remaining $$\phi^{*} = \arg\max_{\phi \in \Phi} f(\theta, \phi), \text{s.t.} \sum_{i=1}^{N} C(\theta,\phi_{i}, r_{i}) < \mathcal{B},$$9 layers. The paper emphasizes three resource effects: no backward pass through frozen layers, no need to store their activations for backward, and no optimizer state for frozen parameters. For Pythia 1.4B at the lowest fidelity, this yielded a memory reduction of at least $C(\theta,\phi_i,r_i)$0 and a runtime speedup of at least $C(\theta,\phi_i,r_i)$1, while training up to $C(\theta,\phi_i,r_i)$2 of layers was reported as adequate for rank correlation $C(\theta,\phi_i,r_i)$3 on the studied search spaces. The same work then analyzed a joint fidelity space over trainable layers and training tokens, arguing that “except at the lowest fidelity where additional layers provide stronger correlations, a joint fidelity approach achieves better correlation with reduced computational cost in both wall-clock time and FLOPs” (Carstensen et al., 14 Apr 2025).

4. Algorithmic patterns

The named JF-HPO method for LLM reinforcement learning combines Bayesian optimization, fidelity-aware scheduling, early stopping, and checkpoint reuse. It maintains an observation set

$C(\theta,\phi_i,r_i)$4

fits a Gaussian-process surrogate over the joint hyperparameter–fidelity space,

$C(\theta,\phi_i,r_i)$5

and selects the next configuration–fidelity pair with the cost-aware acquisition function

$C(\theta,\phi_i,r_i)$6

Operationally, trials are run on the proxy model unless the highest fidelity is required; the method applies early stopping when KL-divergence and reward dynamics indicate failure, and it uses a registry-based checkpointing mechanism so that a configuration promoted to a larger fidelity resumes from an existing lower-fidelity checkpoint instead of restarting (Chen et al., 2 Jun 2026).

Other methods illuminate alternative algorithmic choices. BOHB-style joint NAS+HPO uses a joint configuration space but leaves fidelity largely to Hyperband and Successive Halving; its Bayesian component models interactions between parameters with KDEs, while its budget ladder implements progressive promotion rather than a direct surrogate $C(\theta,\phi_i,r_i)$7 (Zela et al., 2018). HyperJump uses a Gaussian process over configuration and budget, together with Expected Accuracy Reduction and relative jump risk, to skip low-risk stages within a HyperBand bracket (Mendes et al., 2021). FlexHB instead builds separate surrogates for each fidelity, adds Fine-Grained Fidelity measurements, uses GloSH for global ranking across current and historical configurations, and adapts bracket allocation through FlexBand using Kendall-$C(\theta,\phi_i,r_i)$8 between adjacent fidelities (Zhang et al., 2024). A different line of work uses conformalized quantile regression as the surrogate and, in the multi-fidelity case, aggregates each configuration’s trajectory to its last observed value before handing resource scheduling to ASHA; this is multi-fidelity-aware, but it does not model $C(\theta,\phi_i,r_i)$9 directly as a joint surrogate (Salinas et al., 2023).

5. Empirical evidence and benchmark support

The principal empirical evidence for JF-HPO in LLM reinforcement learning is that model size and training budget can be exploited jointly to improve both efficiency and final downstream performance. On GSM8K with Qwen-2.5 7B, average wall-clock time per HPO trial fell from $i$0 h for Random Search and $i$1 h for BOHB to $i$2 h for JF-HPO, corresponding to an “up to $i$3” computational efficiency improvement per trial over Random Search. Under the same 48-hour time budget, JF-HPO outperformed or matched baselines in 22 of 24 experimental runs, and average accuracy reached $i$4 for LLaMA-3.1 8B, $i$5 for Qwen-2.5 7B, and $i$6 for Qwen-3 14B. Compared with the VeRL Recipe, the paper reports performance improvements ranging from $i$7 to $i$8. The proxy-model assumption was supported on GSM8K by rank correlations across five sampled hyperparameter configurations and 120 rankings, with Spearman $i$9 and Kendall $r_i$0; the ablation study further showed degradations from $r_i$1 to $r_i$2 without the proxy model, $r_i$3 without checkpointing, and $r_i$4 without early stopping (Chen et al., 2 Jun 2026).

Earlier empirical work had already established the central budget-dependence phenomenon. In the CIFAR-10 study of joint architecture and hyperparameter search, architectures with fewer residual blocks performed better on average at 400 s, but this preference largely disappeared with more training time, and fANOVA showed that the interaction between CutOut length and number of residual blocks changed with budget. The BOHB-found configuration achieved $r_i$5 test error under a 3-hour per-configuration cap, outperforming several manually constructed baselines trained under the same 3 h budget and pipeline, including ResNet-18 at $r_i$6 and several Shake-Shake variants (Zela et al., 2018).

Benchmark infrastructure has been developed specifically to study such phenomena. YAHPO Gym offers 14 scenarios and over 700 multi-fidelity HPO problems with fidelity encoded as a budget hyperparameter, enabling queries over the full mixed search space rather than a fixed tabular schedule (Pfisterer et al., 2021). LLMSYS-HPOBench extends this idea to real-world LLM systems, currently containing 364,450 hyperparameter configurations, dimensionalities of 12–23, fidelity spaces with 3–5 dimensions leading to 932 settings, 3–9 inference objective metrics, and 2–10 cost metrics. Its case study reports that advanced HPO algorithms do not always dominate simple baselines and attributes this partly to the fact that rich multi-factor fidelities can induce “non-monotonic and non-linear relationships between measurement cost and approximation of objective metrics” (Wu et al., 8 May 2026).

6. Limitations, misconceptions, and open directions

A common misconception is that any multi-fidelity optimizer is automatically a JF-HPO method in the strongest sense. The literature does not support that equivalence. Some frameworks couple search and fidelity through bracket schedules, rungs, or promotions, but do not make fidelity a first-class modeled variable. MO-DEHB, for example, is explicitly multi-fidelity and highly relevant to joint NAS+HPO, yet it treats fidelity as a scheduler-controlled resource inherited from Hyperband rather than as a freely optimized component of the candidate representation; FlexHB similarly couples BO and resource allocation but uses fidelity-specific surrogates rather than a unified surrogate over $r_i$7 (Awad et al., 2023, Zhang et al., 2024).

Transfer assumptions are another central limitation. The LLM-RL JF-HPO method works best when proxy and target models belong to the same family and share architecture, and the paper notes that transfer across major architectural shifts, such as dense proxies for sparse Mixture-of-Experts targets, remains unexplored. It also observes that highly sensitive hyperparameters such as learning rate may transfer less reliably than weaker ones such as the KL loss coefficient. Earlier joint NAS+HPO work is similarly limited by structured architecture spaces, scheduler-controlled fidelities rather than explicit $r_i$8 modeling, and experiments confined to CIFAR-10 with a 3-hour maximum budget (Chen et al., 2 Jun 2026, Zela et al., 2018).

The newer fidelity sources and benchmarks indicate several open directions. Frozen-layer fidelities are attractive because they jointly change compute, memory, and feasibility, but they are not straightforwardly resumable: the paper explicitly states that it does “not pursue” continuation over layers as fidelities because that would turn the problem into training with a layer-unfreezing schedule (Carstensen et al., 14 Apr 2025). LLMSYS-HPOBench shows that fidelity can become a high-dimensional, workload-dependent object with non-monotonic effects, which suggests that future JF-HPO methods may need explicit models over vector-valued fidelities, multiple cost outputs, and constrained feasible fidelity sets rather than a single monotone budget ladder (Wu et al., 8 May 2026). A plausible implication is that the next generation of JF-HPO systems will need to combine joint configuration–fidelity modeling, cost-aware acquisition, and scheduler logic that remains valid even when low-fidelity signals are informative only locally or along particular fidelity directions.