SmartLLMs Scheduler (SLS)

• SmartLLMs Scheduler (SLS) is a dynamic, data-driven framework that integrates adaptive prediction, cost modeling, and online decision-making for efficient LLM serving.
• It optimizes resource allocation to achieve SLA compliance, low latency, and cost-effective throughput in both single- and multi-LLM deployments.
• By employing modular components such as adaptive caching and semantic-aware prioritization, SLS significantly improves performance metrics, including up to 63% lower tail latency.

SmartLLMs Scheduler (SLS) is a class of dynamic, data-driven schedulers designed for cost-effective, low-latency, SLA-compliant utilization of LLMs in diverse serving environments. Unlike legacy static or purely heuristic approaches, SLS frameworks tightly integrate historical performance feedback, adaptive prediction, and online decision-making to optimize for user-centric objectives—often characterized by service-level agreements (SLAs), cost constraints, and throughput goals—across single- and multi-LLM deployments.

1. Motivation and Fundamental Problems

The deployment of LLMs such as GPT-4 and Llama presents three central challenges for real-world applications: high financial cost, substantial and variable response time, and strong dependence of task performance on the model-task match. Existing static schedulers require extensive training data for accurate LLM performance and cost prediction per query, leading to inflexible, expensive, and often suboptimal decision policies when handling large-scale, heterogeneous workloads.

The SLS paradigm addresses these constraints by introducing online, feedback-driven optimization and explicit modeling of the LLM serving pipeline. Common objectives include maximizing goodput (the rate of requests meeting all SLA criteria), minimizing cost per successful request, achieving fairness or priority-aware latency, and providing strong isolation or adaptivity across diverse user/task classes (Liu et al., 5 Aug 2025, Gong et al., 14 Jul 2025).

2. Algorithmic Foundations: Core Components and Modeling

SLS frameworks decompose the scheduling loop into prediction, modeling, and selection components, often implemented as modular, pluggable stages:

• Adaptive Prediction: Predicts runtime properties relevant to scheduling, such as output length, compute/memory requirement, or expected success, using empirical history, semantic-aware retrieval, or lightweight ML models. For output-length and resource estimation, approaches include histogram-based predictors, quantile regression, or semantic-nearest neighbor sampling over prompt embeddings (Gong et al., 14 Jul 2025, Gan et al., 9 Mar 2026).
• Resource and Cost Modeling: Models memory, compute, and economic costs as a function of predicted request properties. For memory, the total KV-cache and activation footprint is estimated dynamically over future steps by integrating predicted output lengths per request; for compute, token-wise FLOPs or runtime are aggregated. Unified cost paradigms often use quadratic forms, e.g., $\mathcal{C}(I, O) = \frac{O^2}{2} + IO$ for resource consumption, capturing both prefill and decode (Gan et al., 9 Mar 2026).
• Admission and Scheduling Policy: LLM requests are selected and ordered using policies that maximize performance-to-cost ratios, meet per-request SLOs, or optimize global metrics such as goodput or average weighted waiting time. Techniques range from simulated annealing over permutations and batch sizes in multi-SLO settings (Huang et al., 21 Apr 2025) to Gittins index-based policies for minimizing expected tail latency under distributional uncertainty (Gan et al., 9 Mar 2026), and priority-aware heap scheduling for semantics-labeled requests (Hua et al., 13 Jun 2025).

A canonical SLS scheduling loop for single-server, multi-LLM, or distributed clusters can be abstracted as:

$$M_{peak}(B) = \max_{j=1,..,k} \alpha \left[ \sum_{i=1}^j (l_p^i + l_t^i) + j r_j \right]$$3 (Liu et al., 5 Aug 2025)

3. SLA and SLO-Aware Scheduling: Metrics and Guarantees

The central objective in SLS-driven serving is maximizing the number of requests that meet their SLOs, while minimizing cost and tail latency. SLAs typically specify constraints on:

• TTFT (Time-To-First-Token): Maximum permissible latency from queue entry to first output token
• TPOT/MTPOT (Time-Per-Output-Token / Max-Time-Per-Output-Token): Upper bounds on stepwise decoding delay
• e2e-Latency: Overall response time

SLS frameworks employ explicit mathematical expressions for batch-level memory, execution time, and queuing delay. For example,

$$M_{peak}(B) = \max_{j=1,..,k} \alpha \left[ \sum_{i=1}^j (l_p^i + l_t^i) + j r_j \right]$$

allows precise admission control to prevent request evictions and enforce hard memory constraints for 99%+ of requests (Gong et al., 14 Jul 2025). Simulated annealing and other scheduling methods optimize the scheduling permutation and batch assignments for multi-objective metrics $G=\frac{n}{t}$ (requests meeting SLO/$\sum$latencies) (Huang et al., 21 Apr 2025).

In multi-task or multi-SLO environments, SLS enables differentiated SLAs per request, supporting fine-grained constraints on TTFT, TPOT, and e2e-latency for mixed workloads (e.g., code completion and chatbot scenarios) (Huang et al., 21 Apr 2025).

4. Architecture: Caching, Feedback, and Cross-Layer Orchestration

SLS implementations combine several architectural techniques for high efficiency and adaptability:

• Adaptive Cache Manager: Detects semantically or syntactically similar queries via prompt embedding clustering; serves repeated requests from cache to minimize recomputation and response time. Caching threshold $\tau$ is dynamically tuned based on cache hit/success statistics (Liu et al., 5 Aug 2025).
• Dynamic, Performance-Cost Optimized Scheduling: Joint feature models over queries and LLM candidates (e.g., $[embed(q); one-hot(m)]$) guide the selection of the optimal LLM, considering both expected accuracy/success and token-level economic cost (Liu et al., 5 Aug 2025).
• Ongoing Feedback and Model Update: Real-time monitoring of output success and cost enables periodic retraining and adaptation of predictors. If error rates or cache effectiveness drift beyond tolerances, model parameters or cache policies are updated online (Liu et al., 5 Aug 2025).
• Two-Layer Orchestration (Cluster and Engine Level): Distributed SLS instances—e.g., PRISM (cluster-layer) and LENS (engine-layer)—collaborate via online, structurally-informed performance models to enable proactive routing and adaptive batching, minimizing “decision lag” and improving both micro and macro-level SLO attainment (Zhang et al., 27 Sep 2025).

5. Semantic and Priority Scheduling

SLS extends classic scheduling by leveraging request semantics learned or inferred from prompts:

• Semantic-Aware Priority Assignment: Lightweight LMs (e.g., DistilBERT) classify requests by urgency, legal, or domain-specific intent. Weights $w_i$ mapped from priority levels $\ell_i$ are incorporated in scheduling keys, enabling strict or lexicographic priority preservation and minimizing average weighted waiting time (Hua et al., 13 Jun 2025).
• Fair and Preemptive Policies: Heap-based data structures enable $O(\log n)$ scheduling, preemption, and cache-eviction based on joint semantic and resource-based keys [(–$w_i$, $$M_{peak}(B) = \max_{j=1,..,k} \alpha \left[ \sum_{i=1}^j (l_p^i + l_t^i) + j r_j \right]$$0)]. This strictly enforces that high-urgency queries preempt lower ones unless the former is not yet in the system, satisfying fairness or criticality constraints (Hua et al., 13 Jun 2025).

Empirical evaluations in real-world, high-stakes domains (e.g., emergency medical services) demonstrate $$M_{peak}(B) = \max_{j=1,..,k} \alpha \left[ \sum_{i=1}^j (l_p^i + l_t^i) + j r_j \right]$$1 to $$M_{peak}(B) = \max_{j=1,..,k} \alpha \left[ \sum_{i=1}^j (l_p^i + l_t^i) + j r_j \right]$$2 lower critical-path latency relative to FIFO or vanilla SJF, robust to moderate classifier or predictor errors (Hua et al., 13 Jun 2025).

6. Empirical Results and Comparative Evaluation

SLS frameworks have been evaluated across diverse hardware, software, and workload settings:

Task/Setting	Performance ↑	Latency ↓	Cost ↓	Baselines	Reference
Log Parsing, Apache	+198.8%	–63.3%	–69.7%	OptLLM, static LLM	(Liu et al., 5 Aug 2025)
Code Generation, CoNaLa	+198.8%	–63.3%	–69.7%	OptLLM, static LLM	(Liu et al., 5 Aug 2025)
SLO-Aware (Python-Code-23k)	up to 5x	–31.6%	n/a	vLLM, LMDeploy	(Huang et al., 21 Apr 2025)
Goodput (LightLLM, Llama-2)	2–3×	n/a	n/a	TGI, FastGen, vLLM	(Gong et al., 14 Jul 2025)
SLO attainment (PRISM+LENS)	+43%	–20–50%	up to 3× TPS	State-of-the-art	(Zhang et al., 27 Sep 2025)

• Key Techniques: Cache-driven reuse, feedback-driven predictor adaptation, semantic batch prioritization, cross-layer online orchestration, future-horizon memory prediction, and uncertainty-aware cost modeling.
• Observed Benefits: Improved SLO attainment (up to 5×), drastic reduction in average or tail latency (up to 63%), massive GPU cost savings (up to 98%), and much higher system goodput at high concurrency (Gong et al., 14 Jul 2025, Liu et al., 5 Aug 2025, Zhang et al., 27 Sep 2025).

7. Extensions, Limitations, and Future Directions

Limitations of current SLS frameworks include dependence on accurate online predictors (for length, urgency, or success), need for periodic sampling or ground-truth labels to bootstrap or adapt, and overhead for predictor retraining or cache management. Some SLS variants assume stationary distributions for accurate memory/success estimation; non-stationary workloads may require adaptive thresholding or additional model complexity (Gong et al., 14 Jul 2025, Gan et al., 9 Mar 2026).

Areas identified for extension:

• Integration of Multi-Dimensional Semantics: Beyond urgency (e.g., legalities, fairness, “cost of delay”).
• Clustered or Hierarchical Scheduling: Multi-GPU, heterogeneous hardware pools, and dynamic orchestration across regions or cloud providers (Zhang et al., 27 Sep 2025).
• Compositional Policies: Pluggable layers for utility maximization, deadline-awareness, or explicit fairness constraints (Gong et al., 14 Jul 2025).
• Advanced Resource/Compute Models: Incorporation of advanced KV-cache compression, speculative decoding, and per-request quantization or offloading (Gan et al., 9 Mar 2026).
• Self-Supervised Feedback Loops: Reducing reliance on labeled ground-truth via confidence estimation or online pseudo-labeling (Liu et al., 5 Aug 2025).

A plausible implication is that the SLS paradigm will become foundational for SLA-driven, large-scale LLM serving, with model-agnostic, adaptive, and feedback-integrated design as key enablers for next-generation enterprise and safety-critical deployments.