ASAP: Agent-System Co-Design for Wall-Clock-Centered Auto HPO Research for ML Experiments

Abstract: Hyperparameter Optimization (HPO) is essential for maximizing machine learning model performance, and its core challenge is sample efficiency: finding strong configurations within a limited budget. Because every HPO tool relies on a surrogate prior that imparts its own inductive bias, individual tools struggle once problems become sufficiently diverse and drift from these priors. Motivated by the reasoning and generalization capabilities of LLMs, recent work has explored using LLMs for HPO and reports improved per-iteration performance. Yet these methods share two limitations with a common origin: they use the LLM as a single-tool replacement evaluated by iteration count. (i) Deployed in place of prior tools, the LLM is itself constrained by its pretraining objective to one family of inductive-biased proposals; this single-source setup still fails to handle the full diversity of problems. (ii) Per-iteration evaluation ignores that, in real runs, LLM inference or tool execution is paid serially on top of model evaluation every round, so iteration-count gains do not translate into end-to-end wall-clock gains. We present ASAP, an agent-system co-design that addresses both limitations. On the agent side, ASAP uses the LLM to integrate a diverse pool of inductive-biased optimizers and to select among their proposals each round. On the system side, ASAP re-architects the loop to reduce end-to-end wall-clock while preserving regret quality: a prefix-stable prompt maximizes KV-cache reuse across rounds; speculation parallelism hides the remaining LLM and tool latency under model evaluation via a relative-error accept test; and a Self-Tuner adapts the speculation threshold from execution logs off the critical path. Extensive experiments on diverse modern HPO tasks show that ASAP consistently outperforms baselines, underscoring the value of tool integration and agent-system co-design.

• The paper presents a novel agent-system co-design framework, ASAP, which pools diverse HPO optimizers under an LLM judge to minimize wall-clock regret.
• It introduces KV-cache-aware prompts, cross-iteration speculation, and an off-critical-path self-tuner to significantly reduce inference and evaluation overhead.
• Empirical results show that ASAP achieves lower normalized regret (~0.026) and robust performance across diverse optimization landscapes.

Agent-System Co-Design for Wall-Clock-Centered Auto Hyperparameter Optimization: Technical Analysis of ASAP

Introduction and Motivation

Hyperparameter optimization (HPO) remains a pivotal challenge in maximizing the empirical performance of ML models. Historically, HPO research has focused on iteration-based sample efficiency and surrogate-based optimization methods, such as Bayesian Optimization (BO) with Gaussian Processes or Tree-structured Parzen Estimators (TPE). The introduction of LLMs has motivated their use as agents for HPO, leveraging strong generalization and reasoning abilities inherited from their pretraining corpus. However, prior LLM-based HPO systems typically operate by substituting a conventional optimizer with an LLM as a single model-based proposer and primarily report per-iteration regret. In practice, these systems do not address two critical real-world limitations: (1) each HPO tool or LLM is bound to its own inductive bias and lacks robustness when the problem distribution drifts; and (2) per-iteration evaluation ignores the wall-clock overhead of LLM inference or tool execution, which can become a dominant cost for realistic run durations and hardware profiles.

This work, "ASAP: Agent-System Co-Design for Wall-Clock-Centered Auto HPO Research for ML Experiments" (2606.25207), addresses these gaps by formally motivating agent-system integration for HPO that directly optimizes wall-clock regret. It presents ASAP, a co-design framework that couples agent-level optimizer pooling with system-level pipeline re-architecture, targeting end-to-end wall-clock efficiency and robustness across diverse HPO landscapes.

Problem Formulation and System Constraints

Modern HPO faces intrinsic task diversity and shifting landscape properties. Every HPO tool—whether a statistical surrogate (e.g., GPs, random forests, TPEs) or an LLM-based proposal mechanism—relies on a surrogate prior, instantiating a strong but localized inductive bias. As a consequence, the tool's performance becomes non-uniform across heterogeneous tasks: e.g., smooth-kernel GPs collapse on combinatorial or highly rugged surfaces, and TPE/SMAC excel in multi-modal but may underperform on anisotropic or deceptive landscapes.

Further, real-world deployments must account for all sources of wall-clock latency: parallel ML evaluation (potentially GPU-intensive) and serial LLM/tool pipeline overhead. Most LLM-HPO literature substitutes a single tool and evaluates performance in iteration counts irrespective of the runtime cost per iteration. However, in practical deployments, especially when ML evaluations are expensive, the wall-clock cost of surrogate optimization and agentic reasoning cannot be ignored. This renders iteration-centric metrics insufficient and makes minimizing regret per real time (rather than per iteration) the true objective.

ASAP: Agent-System Co-Design

The core contributions of ASAP lie in two axes: (1) agent-level integration that pools a diverse set of inductive-biased optimizers under an LLM Judge, and (2) system-level architectural innovations that minimize wall-clock overhead without sacrificing regret performance.

Agent-Level Integration

Instead of wagering on a single inductive prior (and hence suffering from its fragility), ASAP instantiates a heterogeneous pool of HPO optimizers. These include both traditional statistical approaches (GP, TPE, SMAC, Optuna, SKOPT, TuRBO, DNGO) and an LLM proposer (LLAMBO). Each round, all optimizers propose hyperparameter configurations in parallel; the LLM-as-Judge consumes the pooled proposals, leveraging in-context learning to rank and select configurations based on expected improvement conditioned on global task context and accumulated history.

The theoretical underpinning is a per-round, shared-history advantage: although random integration (i.e., uniform selection from the pool) only matches the pool-average regret (as proved using No-Free-Lunch-style results), an informed LLM-as-Judge can discriminate among candidates with minimal additional inductive bias, leveraging both prior knowledge and the growing task trajectory. Empirically, ASAP’s Judge is shown to pick the best candidate roughly 3× more frequently than random selection, thus systematically escaping the grand-mean baseline incurred by any single tool.

System-Level Acceleration and Optimization

ASAP's system design breaks the strict serial loop of propose-Judge-evaluate, which incurs significant latency when ML evaluations are slow. Three principal mechanisms underpin the system improvement:

1. KV-Cache-Aware Prefix-Stable Prompts: ASAP exploits the mostly-append-only structure of in-context learning by carefully designing the LLM-Judge prompts so that all tokens up to the latest observation can be prefetched and cached. Each per-round candidate only requires a short (suffix) prefill, reducing LLM inference times and thus the per-round wall-clock cost of the Judge.
2. Cross-Iteration Speculation: During the dominant ML evaluation phase, the system speculates the outcome of the current configuration using the Judge's own prediction and proceeds to run the next round's tool proposals and Judge selection in parallel, effectively hiding their latency under the in-flight evaluation. Speculatively committed rounds are accepted or rejected based on an adaptive relative-error threshold, with fallback to the non-speculative path ensuring regret is never worsened.
3. Off-Critical-Path Self-Tuner: ASAP introduces an LLM-based Self-Tuner that periodically adapts the speculation accept threshold in light of recent error distributions and downstream regret/wall-clock trade-offs. The tuning process is fully amortized and never enters the critical path, so it can afford more sophisticated log-based analysis and is robust to regime shifts during optimization.

Empirical Results

Extensive experiments—covering seven architectures (classical tabular models, modern CNNs, transformers for text and protein tasks), multiple datasets, and three orders of magnitude in evaluation cost (from seconds to multi-hour deep learning jobs)—validate ASAP's efficacy.

• ASAP attains final normalized regret floors of ~0.026 compared to 0.05–0.16 for all baselines under wall-clock evaluation.
• Speculation removes up to 25% propose/Judge walltime on mid-cost HPOBench regimes, with the gain disappearing only when ML evaluation cost dwarfs all other overheads.
• On all four major landscape axes (Rugged, Multi-Modal, Deceptive, Anisotropic), ASAP is uniformly robust, whereas every single-bias baseline collapses on at least one.
• Case studies demonstrate that ASAP alone reaches ground-truth optima even where individual proposers are systematically misaligned.
• The integrated pool's commits are meaningfully split between the LLM and classical BO tools, confirming genuine heterogeneity is being exploited.
• Speculation reorders compute but does not perturb the explored configuration distribution or optimization trajectory.

Theoretical and Practical Implications

ASAP's framework decisively demonstrates that agent composition and systemic pipeline optimization are critical to bridging the gap between algorithmic and real-world optimization performance. The proposed discriminative integration—where the Judge selects among pooled proposals but need not act as a global generator—provides a compelling escape from No-Free-Lunch brittleness. This mindset generalizes beyond HPO: any finite set of agents (with complementary biases) can, in principle, be coordinated to robustly span diverse, shifting problem regimes.

On the system side, ASAP illustrates the quantifiable benefits of aligning agentic and runtime parallelism, particularly in regimes with non-trivial per-evaluation costs. Speculative acceleration with task-adaptive acceptance delivers highly predictable wall-clock gains, made robust by fail-safe fallback and an off-path Self-Tuner. Notably, the wall-clock improvements are only apparent in the non-trivial-eval regime (where $T_{\text{ML}} \gtrsim 5$s); when model fits are essentially free, model-free BOs regain the wall-clock advantage due to sheer iteration throughput.

Limitations and Future Work

ASAP's wall-clock speedups are gated by the ML evaluation cost: if dominant latency is not present, speculation cannot hide the propose/Judge stages. The system's cache-aware prompt design presumes backends with proper KV-cache sharing, and certain deployment environments may not realize these savings.

A promising avenue for future research involves meta-learning the integration weights or proposing a variant where the LLM-Judge itself is adaptively finetuned with task feedback. It would be fruitful to investigate automated pool composition—adding/removing optimizers during optimization based on runtime statistics—and to generalize speculative agent pipeline architectures to other sequential decision-making tasks.

Conclusion

ASAP establishes wall-clock-centered agent-system co-design as a necessary framework for robust and efficient real-world hyperparameter optimization. The joint agent-level integration and system-level architectural advances empirically and theoretically lead to consistently improved regret per unit wall-clock and generalization across problem regimes. Yet, most notably, the work provides a technical blueprint for how to combine agentic generalization with system-aware acceleration to address the acute needs of practical ML optimization workloads.

---

Reference:

"ASAP: Agent-System Co-Design for Wall-Clock-Centered Auto HPO Research for ML Experiments" (2606.25207).