---
title: Multi-SLO-Aware Scheduling Techniques
url: https://www.emergentmind.com/topics/multi-slo-aware-scheduling
type: topic
---

# Multi-SLO-Aware Scheduling Techniques

Multi-SLO-aware scheduling is the real-time orchestration of resources and task assignments in distributed or parallel systems to simultaneously satisfy diverse, formally specified service-level objectives (SLOs) for multiple users, tenants, or applications. SLOs are typically expressed as quantitative constraints on metrics such as tail-latency percentiles, time-to-first-token (TTFT), throughput per token (TPOT), energy usage, or domain-specific deadlines. Modern multi-tenant serverless platforms, large language model (LLM) serving systems, and high-performance computing environments require these techniques to guarantee differentiated quality of service and optimize aggregate resource efficiency under high load and variable workload mixes.

## 1. Formal Problem Models and SLO Specification

A multi-SLO scheduling problem is generally characterized by:

- **Request/task set** $\{r_1, \ldots, r_N\}$ where each $r_i$ may specify different types and levels of SLOs.
- **SLOs** can take forms such as:
  - $(p_\ell, D_\ell)$: at least fraction $p_\ell$ of function $f_\ell$ requests must finish within deadline $D_\ell$ ([2306.03622]).
  - TTFT $T^{(f)}_i$, TPOT $T^{(o)}_i$, end-to-end deadline $D_i$, and utility weight $U_i$ for LLMs and real-time/priority inference ([2504.08784], [2510.18544], [2507.17769]).
  - Composite or environmental constraints, such as joint minimization of SLO violations and carbon emissions ([2409.00550], [2410.11875]).

The canonical objective is to maximize aggregate utility or service gain, frequently through a utility function $U_{\text{total}} = \sum_{i=1}^N X_i U_i$, where $X_i=1$ iff $r_i$'s SLO(s) are satisfied, and $U_i$ encodes relative importance ([2510.18544]). Constraints span resource capacities, admission or queuing strategies, batch or bin assignment policy, and per-iteration or per-phase SLO feasibility.

## 2. Scheduling Architectures and Algorithmic Patterns

Multi-SLO-aware schedulers employ diverse system architectures and algorithmic primitives:

- **SLO-based prioritization:** Functions or requests are ordered using online metrics that quantify SLO urgency, such as the Required-Request-Count (RRC) in FaaSwap, which dynamically tracks how many future requests must complete within deadline to achieve a tail-percentile SLO: 
  $$
  \mathrm{RRC}_\ell = \frac{p_\ell n_\ell - m_\ell}{1-p_\ell}
  $$
  Requests are prioritized in queues and grouped for service admission using two- or multi-class stratification ([2306.03622]).

- **Bin-packing/resource-aware placement:** Multi-dimensional bin-packing heuristics assign each request to a worker or device so as to respect multi-SLO and multi-resource constraints (KV cache, GPU compute, etc.), with stages such as prefill and decode often co-modeled via latency predictors ([2405.06856], [2411.06364], [2504.08784]).

- **Search/metaheuristics:** Simulated annealing, dynamic programming, and greedy or hybrid algorithms are applied to derive batch orderings or token allocations that maximize the number of SLO-compliant requests or utility under feasibility constraints ([2504.14966], [2504.08784]).

- **Service-gain maximization:** Schedulers like Tempo and SLICE utilize penalty or degradation functions linking SLO adherence to actual observable performance and utility, then iteratively re-prioritize or adjust request rates to keep SLO miss probability low ([2504.20068], [2510.18544]).

- **Interference-awareness and pipelined resource control:** Schedulers integrate interference models (e.g., NVLink/PCIe concurrency or cache pipeline contention) and perform swap/placement or eviction policies to avoid cross-SLO interference ([2306.03622], [2411.06364]).

## 3. Multi-SLO Coordination and Stage-specific Enforcement

Distinct phases of inference or serving, such as prefill and decode in LLM systems, often require coordinated SLO enforcement:

- **Stage-level SLO mapping:** Systems like Drift and SLOs-Serve associate explicit per-stage SLOs (e.g., TTFT and TPOT) to batched operations, translating aggregate deadlines into token-, chunk-, or block-level partitioning ([2504.14489], [2504.08784]).
- **Hardware partitioning:** Drift leverages low-level GPU SM partitioning (PD-multiplexing) to run TTFT (prefill) and TBT (decode) workloads concurrently, dynamically optimizing per-phase resource splits to jointly satisfy SLO targets ([2504.14489]).
- **Multi-tier binning:** SLO tiering and binning create multi-priority or SLO classes (e.g., PolyServe's S SLO bins), allowing scalable separation and routing of requests to the most appropriate instance or resource pool and enabling greedy auto-scaling and instance sharing among compatible tiers ([2507.17769]).

## 4. Practical Implementations and System Integration

Modern systems integrate multi-SLO-aware scheduling into diverse environments:

- **Serverless and containerized inference:** FaaSwap implements late-binding model swapping and interference-aware decisions; HarmonyBatch minimizes serverless provisioning cost by grouping requests across SLOs and resources, supporting both CPU and cGPU, and enforcing SLO-based timeouts ([2306.03622], [2405.05633]).
- **Sustainability and dual-objective frameworks:** CASA and sustainability-aware FaaS augment SLO scheduling with operational carbon and water usage constraints, utilizing hybrid evolutionary search and dual-objective local search to find Pareto-efficient tradeoffs ([2409.00550], [2410.11875]).
- **Edge and resource-constrained scenarios:** SLICE embodies utility-maximizing SLO assignment with dynamic per-request generation rate control, supporting tight TTFT and TPOT requirements for LLM inference on edge devices ([2510.18544]).
- **HPC systems and batch schedulers:** Extensions to SLURM allow joint malleable job reconfiguration subject to both performance- and power-based SLOs, solving at each tick for trajectories minimizing makespan, response time, and dynamic power-corridor violation ([2009.08289]).
- **LLM and diffusion serving:** Systems like PATCHEDSERVE and EcoSERVE utilize fine-grained batching (patch or KV cache) and urgency-based slack scoring to admit requests or tasks in an SLO-feasible manner, improving satisfaction ratio under contention ([2501.09253], [2411.06364]).

## 5. Complexity, Scalability, and Evaluation

Most multi-SLO scheduling problems are NP-hard even in simplified forms (e.g., knapsack or bin-packing generalizations) ([2510.18544]). Key strategies for scalability include:

- **Greedy or priority heuristic orderings** to enable $O(\log N)$ or $O(N\log N)$ per-iteration placement ([2405.06856], [2507.17769]).
- **Divide-and-conquer (multi-bin, staged, or two-pass merging):** HarmonyBatch achieves near-optimal groupings and batchings with per-group search time $O(\log b + \log c)$ ([2405.05633]).
- **GPU-accelerated multi-criteria decision:** For large data centers, GPU-parallelized AHP/TOPSIS can deliver near-real-time scheduling decisions for thousands of containers and QoS-linked links ([1909.07673]).
- **Dynamic feedback and online adaptation:** Control parameters (e.g., high/low watermark for scale-up, slack for SLO miss prediction) are constantly updated in frameworks such as FaaSwap, PolyServe, and Tempo to respond to bursty and heterogeneous workloads ([2306.03622], [2507.17769], [2504.20068]).

Empirical evaluations across systems demonstrate reduction in SLO violation rates up to $5\times$ over naive or baseline schemes, $80{-}91\%$ SLO attainment at much higher load, 30--82% cost savings, and goodput figures within a few percent of theoretical or oracle-optimal policies ([2306.03622], [2405.06856], [2507.17769], [2504.20068], [2405.05633], [2504.14966]).

## 6. Limitations, Practical Challenges, and Extensions

Common challenges include accurate online prediction of per-request requirements (e.g., output length), efficient scaling across heterogeneous resources, and balancing complicated trade-offs (e.g., carbon vs. SLO vs. cost). Limitations of current approaches:

- Degraded performance with poor response length prediction ([2411.06364]).
- Conservative admission in the face of hardware heterogeneity or bursty, unpredictable arrivals ([2507.17769], [2405.06856]).
- Most frameworks assume static profiling tables and may underperform with hardware or model upgrades ([2507.17769]).
- Cluster-scope admission and scaling decisions are often batched over short epochs and may not respond instantly to microburst changes ([2409.00550], [2410.11875]).
- Some frameworks, e.g., SLOs-Serve, assume discrete SLO tiering; extension to continuous or arbitrary per-request SLO contracts is an open research challenge ([2504.08784]).

Opportunities for further work include extending multi-SLO-aware scheduling to cost/energy/fairness objectives, heterogeneous/disaggregated clusters, live pre-emption and speculative execution, and more advanced hybrid online-offline optimization.

---

**Table: Representative Multi-SLO Scheduling Techniques and Domains**

| Technique/Framework    | Application Domain      | Scheduling Principle                        |
|-----------------------|------------------------|---------------------------------------------|
| FaaSwap [2306.03622]  | Serverless Inference    | RRC-based multi-class prioritization        |
| Aladdin [2405.06856]  | LLM Cluster Serving     | MIP + bin-packing w/ latency SLOs          |
| CASA [2409.00550]     | Serverless Autoscaling  | Local search, carbon/SLO dual-optimization  |
| PATCHEDSERVE [2501.09253] | Diffusion Inference | Slack-based patch scheduling                |
| SLICE [2510.18544]    | Edge LLM Inference      | Utility-maximizing rate control             |
| PolyServe [2507.17769]| Multi-SLO LLM Serving   | Multi-bin queueing + SLO-aware routing      |
| Drift [2504.14489]    | GPU LLM Serving         | Resource partitioning (PD-multiplexing)     |
| EcoSERVE [2411.06364] | LLM Serving             | Decoupled batching + KVC pipelining         |
| SLOs-Serve [2504.08784]| LLM Multi-Stage        | Dynamic-programming token allocation        |
| Network-aware MILP [1909.07673] | Container DC | MILP with bandwidth+placement SLOs          |

---

Multi-SLO-aware scheduling is now a core challenge in the design of highly effective, scalable serving systems for modern ML/AI, serverless, and HPC workloads. State-of-the-art research demonstrates the value of combining mathematical models, SLO- and interference-aware prioritization, resource-aware bin-packing, and adaptive scaling with practical, empirically validated heuristics. These frameworks support differentiated user SLAs, near-optimal resource efficiency, and robust operation in heterogeneous and multi-tenant environments.

Source: https://www.emergentmind.com/topics/multi-slo-aware-scheduling