LLMSYS-HPOBench: Hyperparameter Optimization Benchmark Suite for Real-World LLM Systems

Abstract: LLM systems have been the frontier of AI in many application domains, leading to new challenges and opportunities for hyperparameter optimization (HPO) for the AutoML community. However, this type of system exhibits an unprecedented compound space of hyperparameter configuration from both the AI and non-AI components; rich and nonlinear implications from the fidelity factors; and diverse costs of measuring hyperparameter configurations, none of which have been fully captured in existing benchmarks. This paper presents the first (live) benchmark suite and datasets for HPO of real-world LLM systems, dubbed LLMSYS-HPOBench, covering data related to the inference objective values of hyperparameter configurations profiled from running the LLM systems. Currently, LLMSYS-HPOBench contains 364,450 hyperparameter configurations with a dimensionality of 12-23, 3-5 dimensions of fidelity factor leading to 932 settings, 3-9 inference objective metrics, and 2-10 cost metrics, together with generated logs from measuring the LLM systems. What we seek to advocate is not only a revalidation of the existing HPO algorithms over the frontier LLM systems, but also to provide an evolving platform for the AutoML community to explore new directions of research in this regard. The benchmark suite has been made available at: https://github.com/ideas-labo/llmsys-hpobench

• The paper presents an open, extensible benchmark specifically designed for hyperparameter optimization in real-world LLM systems.
• It employs systematic sampling across AI and non-AI parameters with multi-fidelity settings, providing comprehensive metrics and hardware traces.
• The study shows that advanced HPO methods sometimes underperform compared to random search, highlighting the need for cost-aware, system-specific strategies.

Authoritative Overview of LLMSYS-HPOBench: A Hyperparameter Optimization Benchmark for Real-World LLM Systems

Motivation and Benchmark Necessity

LLM systems constitute the dominant trajectory in modern AI, encompassing intricate architectures that conjoin foundational LLMs with diverse system-level mechanisms such as retrieval, planning, memory, and specialized tool modules. This composite nature yields expansive, high-dimensional hyperparameter spaces that incorporate both AI-centric parameters (e.g., generation temperature, context length, retrieval strategies) and non-AI components (e.g., scheduling, storage, hardware-specific controls). Existing benchmarks in hyperparameter optimization (HPO)—focusing either on classical ML, neural architecture search (NAS), or traditional system software configuration—are fundamentally mismatched to the distinctive requirements and nonlinear cost/quality trade-offs intrinsic to LLM systems.

The "LLMSYS-HPOBench" benchmark (2605.08305) directly addresses these deficiencies by delivering, for the first time, an open, extensible, and measured suite for empirical HPO research tailored to real-world LLM systems. This platform operationalizes systematic evaluation and enables the development, comparison, and diagnostic analysis of HPO strategies on heterogeneous, multi-objective, and fidelity-rich LLM deployments.

Construction and Contents of LLMSYS-HPOBench

LLMSYS-HPOBench encompasses comprehensive measured data from seven diverse LLM systems, including agentic frameworks (AUTOGPT, OPENHANDS), retrieval-augmented generation (RAG) pipelines (LIGHTRAG, NAIVERAG, HTMLRAG), and inference engines (VLLM, SGLANG). The current instance aggregates 364,450 distinct hyperparameter configurations spanning 12–23 variables, each evaluated under 3–5 fidelity factors, producing 932 fidelity settings. The underlying configuration space consists of both AI and non-AI hyperparameters, instantiated via systematic sampling (Latin hypercube, uniform random, distance-based) to maximize coverage given prohibitive measurement costs—approximately 2.3M CPU core-hours and 95K GPU-hours.

For each (configuration, fidelity) tuple, LLMSYS-HPOBench records:

• Multi-objective inference metrics (2–9 per system, e.g., F1, throughput, success rates),
• Granular cost metrics (2–10 per evaluation, including tokens consumed, API calls, wall-clock, and hardware resource utilization),
• Full hardware traces (CPU, GPU, RAM, VRAM, temperature, power),
• Detailed execution logs, facilitating error analysis and traceability,
• Standardized structured outputs for reproducibility.

This infrastructure is exposed through a unified Python API, facilitating seamless integration with external optimizers and automation pipelines.

Benchmark Distinctiveness

LLMSYS-HPOBench presents a constellation of features not available in prior HPO/system benchmarks:

• Compound, heterogeneous configuration spaces: Joint tuning of AI and non-AI parameters.
• Rich, high-dimensional fidelity factors: Up to five per system, with non-monotonic, nonlinear effects on cost/target metrics.
• Many-objective and cost-aware evaluation: Multi-objective settings spanning quality, resource, and economic criteria.
• Explicit hardware and system traces: Enabling hardware/cost/debug-aware optimization research.
• Structured error/logging datasets: Supporting advanced explainability, reliability studies, and meta-learning approaches.
• Live, extensible design: Open contributions and continual incorporation of new system/data profiles.

Empirical Evaluation and Case Study

A case study within the benchmark contrasts classic and contemporary HPO methods—including Random Search (RS), Hyperband (HB), BOHB, SMAC, and HEBO—on three representative LLM systems (NAIVERAG, LIGHTRAG, VLLM). Contradicting typical assumptions from standard HPO domains, advanced methods (e.g., HEBO, HB) do not consistently outperform random baselines across all LLM systems under cost-constrained scenarios. The relative effectiveness of multi-fidelity optimization is system-specific: clear in NAIVERAG, limited in LIGHTRAG and VLLM, where rich fidelity factors induce non-monotonic cost-approximation mappings. This heterogeneity underscores the necessity for LLM-specific HPO algorithmic designs. Besides, algorithmic superiority is highly budget-sensitive, implying that cost-aware optimization strategies are essential for practical deployment.

Implications and Prospects for AutoML and LLM Research

The introduction of LLMSYS-HPOBench engenders new practical and theoretical directions for AutoML:

• High-dimensional, non-monotonic fidelity optimization: Existing methods based on monotonic or single-fidelity assumptions degrade in LLM scenarios. There is a critical need for algorithms capable of navigating non-linear, non-monotonic optimization landscapes induced by multi-factor fidelities.
• AI/non-AI hyperparameter co-design: Interactions between AI and non-AI parameters demand bespoke optimization schemes, e.g., sequential or interaction-aware HPO, with dynamic adaptation to RAG, caching, and agent architectures.
• Many-objective, cost/hardware-sensitive optimization: The high number of explicit objectives and resource/cost constraints introduces challenging Pareto front exploration in massive configuration spaces, demanding scalable, cost-efficient, and hardware-coherent techniques.
• Meta-optimization, explainability, and log integration: The availability of execution logs facilitates the development of explainable HPO, diagnosis tools, and even LLM-driven or multi-modal optimization strategies leveraging logs as additional learning modalities.

Longer-term, the benchmark sets the foundation for empirical fitness landscape analysis of LLM system configuration—a currently unexplored avenue—and for direct technology transfer into industry-scale, production LLM deployments, removing the translation gap existing in ML/NAS- or system-only benchmarks.

Limitations and Extensibility

The current benchmark, although extensive, covers a finite subset of active LLM systems and discretizes the fidelity and hyperparameter domains for tractability. As a live benchmark, LLMSYS-HPOBench is designed for continuous community-driven expansion and refinement, aiming to increase system diversity, configuration granularity, and measurement depth. Noise in time-based metrics is handled via curation, but some irreducible variance is inherent given real-system profiling.

Conclusion

LLMSYS-HPOBench (2605.08305) constitutes a milestone for empirical HPO research in the era of LLM systems by introducing a large-scale, multi-factored, and extensible benchmarking environment anchored in real-world deployments. Its unique characterization of configuration, fidelity, metric, and log data supports not only fair, systematic comparisons of HPO techniques but also the exploration of new algorithmic principles tailored to the LLM domain. The benchmark opens clear practical and theoretical avenues for advances in cost-aware, multi-objective, and explainable AutoML for LLM systems.