Efficient Hyperparameter Optimization for LLM Reinforcement Learning

Abstract: Reinforcement learning (RL) for LLMs is highly sensitive to hyperparameter configurations, making hyperparameter optimization (HPO) essential yet computationally expensive. Existing multi-fidelity HPO methods remain inefficient for LLM RL due to the massive model scale and resource-intensive training cycles. In this paper, we propose Joint Fidelity Hyperparameter Optimization (JF-HPO), which simultaneously adapts both model size and training budget as fidelity. JF-HPO is empowered by: (i) it leverages a small proxy model of the target LLM for efficient training and evaluation in each HPO trial; (ii) it integrates carefully designed early-stopping strategies based on training dynamics; (iii) it introduces an efficient checkpointing mechanism to eliminate redundant computations. Compared with existing HPO methods, JF-HPO significantly improves the computational efficiency of each trial (up to 14.9 times), while achieving better or competitive predictive accuracy under the same time budget. Notably, compared with utilizing hyperparameter configurations from the VeRL Recipe, JF-HPO delivers performance improvements ranging from 5.8% to 111.6%.

• The paper introduces JF-HPO, which addresses hyperparameter sensitivity in LLM RL by jointly adapting model size and training budget.
• It employs proxy models, dynamics-aware early stopping, and checkpointing to reduce trial times by up to 14.9× while boosting performance up to 111.6%.
• Empirical evaluations on tasks like GSM8K and MATH demonstrate that JF-HPO outperforms traditional methods in 22 out of 24 experiments.

Efficient Hyperparameter Optimization for LLM Reinforcement Learning

Motivation and Problem Statement

Hyperparameter sensitivity constitutes a major challenge in reinforcement learning (RL) for LLMs. Configurations such as learning rate, clipping ratio, and KL penalty elicit substantial variance in task performance and stability metrics. Conventional hyperparameter optimization (HPO) approaches including grid search, random search, and multi-fidelity methods like Successive Halving and BOHB, are computationally prohibitive in LLM RL due to enormous model scale and the expensive nature of generation plus backpropagation cycles. The paper introduces Joint Fidelity Hyperparameter Optimization (JF-HPO), which leverages correlated fidelity adaptation on both model size and training budget within Bayesian optimization, enhancing computational efficiency and configurational search breadth under fixed time constraints.

JF-HPO: Methodological Overview

JF-HPO integrates three principal modules: proxy model utilization, dynamics-aware early stopping, and efficient checkpointing.

• Proxy model: HPO trials are executed on a significantly smaller proxy model (same architecture but reduced parameters) to obtain efficient approximations of hyperparameter ranking. This proxy is only swapped for the target model in the final full-fidelity evaluation.
• Joint fidelity: The method adapts both model size and training budget in the acquisition function, selecting configuration-fidelity pairs to optimize a time-cost-normalized expected improvement in Bayesian optimization.
• Early stopping strategies: Trials are terminated early if excessive KL divergence increase, reward descent, or sustained zero reward are detected, based on tightly thresholded criteria. This mitigates wasted computation on doomed configurations.
• Checkpointing: Registry-based checkpointing allows configuration trials to resume from prior partial training, eliminating redundant retraining in successive halving.

Figure 1: Overview of the joint fidelity hyperparameter optimization (JF-HPO) procedure combining proxy model adaptation, early stopping, and checkpoint registry.

Empirical Evaluation

JF-HPO is benchmarked across diverse tasks (GSM8K, MATH, OpenBookQA, MMLU) and multiple LLMs (Qwen-2.5 7B, LLaMA-3.1 8B, Qwen-3 14B). The paper deploys RL algorithms with Group Relative Policy Optimization (GRPO), which omits value function estimation and leverages normalized reward-based advantage calculation. Comparative baselines comprise VeRL Recipe official hyperparameters, Random Search, and BOHB.

Strong numerical results are reported: JF-HPO delivers performance gains ranging from 5.8% up to 111.6% relative to VeRL Recipe, and matches or outperforms baselines in 22 out of 24 experiments at equal time budget. The computational efficiency per trial is increased by up to 14.9× versus baselines.

Figure 2: Performance improvements of JF-HPO across various LLM RL tasks compared to baseline HPO methods.

Figure 3: Performance improvements on Qwen-2.5 7B illustrating robust gains across tasks.

Hyperparameter Effect Analysis

The paper elucidates sensitivity profiles for individual hyperparameters, demonstrating, for instance, that learning rate has maximal impact: performance increases up to 1e-6 learning rate, then declines as rate increases further. The transferability of hyperparameters from proxy to target model correlates with sensitivity; hyperparameters with weak impact are more reliably transferable.

Figure 4: Effect of different hyperparameters on GSM8K final performance for Qwen-2.5 0.5B.

Furthermore, learning rate scheduling exerts significant influence: larger learning rates paired with cosine schedulers outperform constant schedules due to smoother optimization dynamics.

Computational Efficiency

JF-HPO achieves order-of-magnitude throughput increases in trial execution by exploiting its proxy model and checkpointing mechanisms. For Qwen-2.5 7B, throughput is 8772 tokens/s (overall) versus 521.6 tokens/s for Random Search. Average HPO trial time is reduced from 8.80 hours to 0.59 hours, facilitating a 14.9× speedup, which translates directly into higher-quality configuration exploration.

Training Dynamics and Stability

By dynamically stopping trials with excessive KL divergence rise or reward drop, JF-HPO attains more stable training, consistently yielding higher training rewards and smoother KL loss profiles than baselines.

Figure 5: Training reward and KL divergence loss comparison demonstrating improved training stability with JF-HPO.

Figure 6: Additional results of training reward and KL divergence loss comparison, confirming stability across tasks and models.

Ablation and Rank Correlation

Ablation studies validate the contribution of each module: removing proxy models, checkpointing, or early stopping causes substantial accuracy drops. Rank correlation analysis between proxy and target models yields Spearman of 0.90 and Kendall of 0.80, substantiating the efficacy of fidelity transfer.

Task Difficulty and Generalization

JF-HPO's performance advantage increases with sample difficulty, especially on challenging subsets (e.g., Level 5 of MATH). Out-of-distribution evaluation demonstrates robust generalization: JF-HPO improves OOD performance on AMC 2023 and AIME 2025 tasks by 60.88% and an unbounded gain respectively over VeRL Recipe.

Practical and Theoretical Implications

Practically, JF-HPO enables efficient and effective HPO for LLM RL at scale, making routine tuning feasible for challenging tasks previously out of reach. Theoretically, the demonstrated fidelity correlation between proxy and target models paves the way for further research in scalable HPO frameworks and refined proxy selection, especially in settings featuring architectural divergence (e.g., dense vs. MoE). The pronounced sensitivity of difficult reasoning tasks to RL hyperparameters implies that advanced HPO is indispensable for optimal performance and model robustness.

Future Directions

JF-HPO's framework generalizes to other RL algorithms, such as DAPO and REINFORCE++. Potential developments include formal investigation of rank correlation bounds and proxy fidelity calibration for ultra-large models. Extension to subjective tasks (e.g., open-ended generation) remains to be empirically assessed.

Conclusion

JF-HPO systematically addresses the computational bottlenecks in hyperparameter optimization for RL in LLMs by jointly adapting model size and training budget, employing proxy models, checkpointing, and early stop strategies. The approach attains superior efficiency and accuracy, especially for complex reasoning tasks, and sets the stage for further theoretical refinement and broad practical application in scalable LLM training.