SLO-Guard: Crash-Aware, Budget-Consistent Autotuning for SLO-Constrained LLM Serving

Abstract: Serving LLMs under latency service-level objectives (SLOs) is a configuration-heavy systems problem with an unusually failure-prone search space: many plausible configurations crash outright or miss user-visible latency targets, and standard black-box optimizers treat these failures as wasted trials. We present SLO-Guard, a crash-aware autotuner for vLLM serving that treats crashes as first-class observations. SLO-Guard combines a feasible-first Thermal Budget Annealing (TBA) exploration phase with a warm-started Tree-structured Parzen Estimator (TPE) exploitation phase; the handoff replays all exploration history, including crashes encoded as extreme constraint violations. We additionally contribute a configuration-repair pass, a GPU-aware KV-cache memory guard, and a four-category crash taxonomy. We evaluate SLO-Guard on Qwen2-1.5B served with vLLM 0.19 on an NVIDIA A100 40GB. Across a pre-specified five-seed study, both SLO-Guard and uniform random search attain 75/75 feasibility with zero crashes under the corrected concurrent harness, and are statistically tied on best-achieved latency (Mann-Whitney two-sided p=0.84). SLO-Guard's advantage is in budget consistency: more trials in the fast-serving regime (10.20 vs. 7.40 out of 15; one-sided p=0.014) and higher post-handoff consistency (0.876 vs. 0.539; p=0.010). Under concurrent load, SLO-Guard's cross-seed standard deviation on best latency is 4.4x tighter than random search's (2.26 ms vs. 10.00 ms). A harness-replication analysis shows that the consistency findings survive an independent sequential-dispatch measurement condition. The central claim is not that SLO-Guard finds a better final configuration, but that it spends a fixed tuning budget more predictably once the fast regime has been found.

• The paper introduces a two-phase, crash-aware autotuning framework that prioritizes budget consistency and regime stability for SLO-constrained LLM serving.
• Its method leverages a feasible-first annealed search and a warm-started TPE-exploit phase to efficiently avoid crashes and optimize low-latency configurations.
• Empirical results demonstrate significant latency variance reduction and enhanced operational repeatability, supporting robust high-throughput LLM deployments.

SLO-Guard: Crash-Aware, Budget-Consistent Autotuning for SLO-Constrained LLM Serving

Introduction

SLO-Guard introduces a two-phase, crash-aware autotuning framework for serving LLMs under explicit Service-Level Objectives (SLOs) in failure-prone search spaces. The core challenge addressed is autotuning the heterogeneous, conditional configuration space of engines like vLLM, where improper settings frequently lead to crashes, SLO violations, or poor operational regimes. SLO-Guard directly encodes crashes as first-class observations and focuses on robust regime discovery and budget consistency rather than pure optimality, a significant departure from classic black-box hyperparameter optimization approaches.

Problem Formulation

Unlike traditional algorithm configuration, the configuration space for LLM serving in vLLM is characterized by:

• Failure-Proneness: Many configurations induce hard failures (e.g., CUDA out-of-memory, invalid flag combinations) at various execution points.
• Conditional Structure: Several configuration knobs are meaningful only in the presence of specific parent parameterizations.
• Stringent Budget Constraints: Each trial is operationally expensive, mandating methods that function under small evaluation budgets (e.g., 15 trials).

The formal objective is constrained maximization of "goodput" (throughput within SLO bounds), discarding any configuration that crashes or exceeds SLO thresholds. The feasible set $\mathcal{F}$ is thus defined by crash status, tail time-to-first-token (TTFT), inter-token latency (ITL), and GPU memory consumption.

SLO-Guard Methodology

SLO-Guard is composed of two tightly coupled search phases:

1. TBA-Explore Phase:
   • Uses a feasible-first, annealed search to probe the feasible regions while recording both feasible and infeasible (crash) configurations.
   • Unsuccessful proposals are not ignored; instead, they update a category-conditioned "bad-region" tracker which informs later search and mitigates repeated exploration of known failure-prone subspaces.
   • Embedded is an engineering-driven repair map with a GPU-aware KV-cache memory guard, detecting and correcting infeasible configurations before execution.
2. Warm-Started TPE-Exploit Phase:
   • After a defined exploration threshold, the history (including crash-labeled trials encoded as extreme constraint violations) is replayed into an Optuna TPE sampler.
   • TPE's density-based exploitation efficiently sharpens sampling in the fast regime, leveraging all prior crash data.
   • The handoff commonly occurs around trial 7 in practical 15-trial budgets.

SLO-Guard's optimization sequence is distinct in handling the conditional configuration space of vLLM and efficiently discriminating between fast and slow operational regimes.

Empirical Results

Budget Allocation and Consistency

SLO-Guard and uniform random search both achieve 100% feasibility and zero crashes on Qwen2-1.5B served with vLLM 0.19. However, SLO-Guard exhibits compelling advantages on budget consistency metrics. It allocates significantly more trials to the "fast cluster" (lower-latency regime) and maintains higher post-handoff consistency (over 0.87 vs. 0.54 with random search; $p=0.010$).

Figure 1: SLO-Guard consistently yields a higher number of trials in the fast cluster and tighter variance in post-hit consistency across seeds.

Convergence and Variance Reduction

Examining best-so-far latency trajectories reveals that SLO-Guard, after initial exploration, reliably remains in the fast regime and refines configurations within it, unlike random search, which can revert to suboptimal settings even after successful exploration. The cross-seed standard deviation of best-achieved latency is $4.4\times$ lower with SLO-Guard than with random search (2.26 ms vs. 10.00 ms).

Figure 2: SLO-Guard trajectories remain stably in the fast cluster post-handoff, while random search frequently reverts to suboptimal latency regimes.

Figure 3: Best latency is indistinguishable between methods, but SLO-Guard achieves markedly lower cross-seed variance.

Replication and Harness Effects

Results are consistent under both sequential and concurrent dispatch harnesses. However, the variance reduction effect for best-latency is most pronounced under concurrent conditions, congruent with the practical demands of high-throughput LLM serving.

Figure 4: Transition from sequential to concurrent harness reveals the variance advantage of SLO-Guard, while consistency statistics remain robust.

Search Space Structure

Analysis of the configuration space shows a bimodal latency landscape, dominated by the binary enforce_eager knob. The fast regime lies entirely in enforce_eager=false configurations, guiding the interpretation of the search process as regime discovery.

Figure 5: The search space is sharply divided by the enforce_eager knob, emphasizing the role of regime discovery in efficient autotuning.

Search Phase Transition

Individual-seed trajectories illustrate the TBA-to-TPE handoff: initial trials explore the slower regime, with a discrete transition into stable, low-latency configurations post-handoff.

Figure 6: The TBA-to-TPE handoff marks a genuine change in search behavior, enabling consistent exploration of the fast cluster.

Implications and Future Directions

Practical Takeaways

Deployments benefit from explicit GPU-aware guards and benchmarks conducted under genuine concurrency, as sequential harnesses can hide critical variance and consistency effects. Furthermore, the experimental design in SLO-Guard explicitly addresses repeatability and operational reliability, which are under-explored in prior work on LLM serving autotuning.

Theoretical Implications

Crash-awareness confers significant improvements only under specific conditions: ragged, multimodal feasible sets and small trial budgets. In smoother spaces or with large budgets, classic BO and random search converge, but for practical deployments, regime stability and deterministic budget allocation are preferable to sporadic optimal hits.

Limitations

The dominance of enforce_eager constrains generality claims; more entangled or higher-dimensional configuration spaces may modulate SLO-Guard's current empirical advantages. Future work should include ablations (TBA-only, cold-start TPE, within-cluster tuning), expansion to other models and hardware, and evaluation against more realistic, production-shaped traffic with burstiness and multi-turn prompts.

Conclusion

SLO-Guard demonstrates that explicit modeling of crashes and SLO-exceeding configurations, combined with a two-phase optimization approach, yields substantial advantages in trial budget consistency and regime stability in the autotuning of vLLM for SLO-constrained LLM serving. The main contribution, substantiated by replicated empirical analysis, is not lower absolute latencies, but more reliable, reproducible allocation of finite search budgets to optimal operation modes. This perspective provides a practical foundation and experimental methodology for scaling and generalizing crash-aware autotuning in LLM-serving workloads.