---
title: 'MQFQ-Sticky: Fair GPU Scheduling Framework'
url: https://www.emergentmind.com/topics/mqfq-sticky
type: topic
---

# MQFQ-Sticky: Fair GPU Scheduling Framework

MQFQ-Sticky is a scheduling and memory-management framework for providing GPU acceleration to Functions as a Service in a black-box manner, without modifying function code, in settings where containerized serverless workloads encounter limited GPU concurrency, high cold-start overheads, and heavy queueing of function invocations [2507.08954]. It stands for Multi-Queue Fair Queuing with Stickiness and adapts fair queueing and anticipatory scheduling principles from I/O scheduling to function scheduling on GPUs, with the stated objective of balancing locality, fairness, and latency in dynamic, heterogeneous FaaS workloads [2507.08954].

## 1. System model and problem setting

MQFQ-Sticky is motivated by the mismatch between GPUs and the FaaS programming model. Functions-as-a-Service is organized around short, stateless functions in isolated sandboxes, with elasticity and scale-to-zero as core features, whereas GPUs are designed to batch large kernels, offer limited concurrency, and expose relatively small on-device memories with tight coupling between compute and memory placement [2507.08954]. In containerized FaaS, every invocation may need to attach a device, initialize frameworks, and load model and operator libraries; GPU cold starts frequently add seconds of latency and can be 75× worse than CPU cold starts [2507.08954].

The paper situates MQFQ-Sticky in workloads that include machine learning inference and training, audio and video pipelines, and scientific computing, with inter-arrival times, execution times, and memory footprints varying by orders of magnitude [2507.08954]. In such environments, naive FCFS or simple least-loaded queueing quickly produces long queues and high tail latency, because invocations serialize behind cold-start-heavy functions and memory thrashing [2507.08954]. MQFQ-Sticky therefore treats scheduling and GPU memory management as a coupled control problem rather than as separable runtime services [2507.08954].

The framework is described as delivering black-box GPU acceleration for arbitrary functions running in standard containers [2507.08954]. This is significant because prior GPU-sharing systems discussed in the paper, including Gandiva, Themis, Salus, and Paella, target inference or training workloads with more application-specific profiling or kernel-level control, whereas MQFQ-Sticky is explicitly designed for diverse, isolated serverless functions [2507.08954].

## 2. Queueing model, fairness mechanism, and tunable controls

The scheduler uses one queue per function and tracks virtual service times so that backlogged functions receive equal shares of GPU time [2507.08954]. The formulation is adapted from weighted fair queueing, where the paper recalls the classic tag update
$$
F_i^k = \max(F_i^{k-1}, V(a_i^k)) + \frac{L_i^k}{w_i},
$$
with $V(\cdot)$ the system virtual time, $L_i^k$ the job’s service demand, and $w_i$ the class weight [2507.08954]. In MQFQ-Sticky, each function $k$ has its own queue, and upon dispatch the queue’s virtual time is incremented by its historical average GPU service time $\tau_k$; with weights $w_i$ this yields proportional fairness in GPU service [2507.08954].

Three tunable controls organize the fairness–locality trade-off.

| Parameter | Meaning | Stated effect |
|---|---|---|
| **$T$** | Queue over-run threshold | Larger $T$ increases locality and batches at the cost of looser fairness |
| **$D$** | Allowed device concurrency | Can be fixed or dynamically adjusted from utilization feedback |
| **TTL** | Per-queue keep-alive time | Empty queues remain active for $TTL=\alpha \times IAT$ |

The scheduler maintains a global virtual time anchor, denoted `Global`, defined as the minimum virtual time over all queues [2507.08954]. A queue can dispatch while its virtual time satisfies `VT < Global + T`; this allows controlled queue over-run to capitalize on temporal locality while retaining formal fairness bounds [2507.08954]. The paper presents the fairness guarantee for backlogged functions $i$ and $j$ as
$$
| S_i/w_i - S_j/w_j | \le (D - 1) \cdot (2T + \tau_i/w_i - \tau_j/w_j),
$$
where $S_i$ is total device service time and $\tau_i$ is average runtime in the interval [2507.08954]. The implementation defaults to $w_i=1$, though per-tenant or per-function priorities are stated to be conceptually supported [2507.08954].

Anticipatory scheduling is another core component. Inspired by anticipatory disk I/O scheduling, the framework briefly waits to keep recently active queues hot, anticipating imminent invocations and avoiding thrashing [2507.08954]. Specifically, an empty queue is not immediately marked inactive; instead it remains active for
$$
TTL = \alpha \times IAT,
$$
where $IAT$ is the function’s inter-arrival time and $\alpha$ is tunable [2507.08954]. The paper states that even $\alpha \approx 0.1$ yields noticeable gains, and that per-function TTL using observed IATs outperforms a global TTL by about 15% [2507.08954].

This suggests that MQFQ-Sticky does not enforce strict processor-sharing semantics. Rather, it admits bounded fairness drift in exchange for preserving warm state and batching opportunities, with $T$, $D$, and TTL acting as explicit operator controls [2507.08954].

## 3. GPU memory stickiness and runtime residency management

The “stickiness” in MQFQ-Sticky refers to GPU memory and container residency. The scheduler is co-designed with a lightweight CUDA interposition shim using `LD_PRELOAD` that intercepts `cuMemAlloc` and replaces it with `cuMemAllocManaged` [2507.08954]. This provides transparent oversubscription and allows the control plane to steer memory placement [2507.08954].

The framework maintains a pool of GPU-warm containers for active functions and attempts to keep per-function state on device where possible, including model weights, operator kernels, CUDA runtime state, and any pre-initialized execution context [2507.08954]. When a queue becomes active, the system prefetches Unified Virtual Memory regions to the device using `cuMemPrefetchAsync`, overlapping that work with control-plane marshaling to keep prefetch off the critical path [2507.08954]. When a queue is throttled or inactive, on-device pages are evicted using LRU and moved back to host memory, again via `cuMemPrefetchAsync`, producing what the paper terms a “GPU-cold but host-warm” restart [2507.08954].

The residency policy is summarized by the score
$$
score_i = \beta_1 \cdot queue\_len_i + \beta_2 \cdot (Global + T - VT_i)/T + \beta_3 \cdot reuse\_prob_i,
$$
with operator-controlled weights $\beta$ and `reuse_prob_i` derived from IAT estimates [2507.08954]. Active queues with higher score receive prefetch priority, whereas throttled or inactive queues are demoted and eventually evicted in LRU order to respect GPU memory capacity [2507.08954].

This co-design is central to the framework’s interpretation of locality. The paper argues that larger batches and longer active periods lower latency because they reduce cold starts and repeated swapping of large model and operator state, but that these same behaviors can drift from strict fairness if not bounded by the queue over-run mechanism [2507.08954].

## 4. Dispatcher logic, isolation model, and implementation structure

MQFQ-Sticky runs per worker and integrates with NVIDIA-Docker containers and, when available, with MPS and MIG [2507.08954]. It keeps one MQFQ queue per function on the worker across one or more physical or virtual GPUs [2507.08954]. The dispatcher performs the following sequence [2507.08954]:

1. Compute `Global = min VT` over all queues.  
2. Update queue states:  
   - if empty and no in-flight invocations, mark inactive if `now − last_exec ≥ TTL`; otherwise keep active;  
   - if `VT − Global ≥ T`, mark throttled; else active.  
3. Form the candidate set of active queues with backlog and `VT < Global + T`.  
4. Sort candidates by descending queue length; for `D > 1`, break ties by ascending `in_flight`.  
5. Obtain a concurrency token via `get_D_token(chosen)`.  
6. Dispatch `chosen.pop()`, increment its `VT` by `τ_k`, and update `Global` if needed.

The paper states that this heuristic preserves MQFQ’s fairness constraints while intensifying locality, because it prefers longer queues within the fairness window and avoids oversaturating a single function concurrently when `D > 1` [2507.08954]. It also states that the per-dispatch work is $O(n)$ to scan queues and $O(|cand| \log |cand|)$ to sort candidates, while noting that priority structures can reduce this to $O(\log n)$ in principle [2507.08954].

Isolation is preserved at the container boundary [2507.08954]. User code is unmodified, runs in standard containers, and communicates through the platform’s agent; the `LD_PRELOAD` shim only interposes CUDA memory APIs and does not alter application logic [2507.08954]. When present, MPS is launched once per GPU and all function containers attach to it, improving low-level kernel overlap while MQFQ controls high-level fairness and locality [2507.08954]. MIG slices are treated as distinct vGPUs, though the paper notes that some functions degrade on smaller slices if they assume full-device resources [2507.08954].

The implementation is integrated into Iluvatar’s FaaS control plane, with approximately 3,000 lines of Rust, NVML polling every 200 ms, and a dispatcher thread [2507.08954]. Hardware used in evaluation includes an NVIDIA V100 system and an NVIDIA A30 system [2507.08954].

## 5. Empirical behavior and measured performance

The evaluation covers 24 function types spanning ML inference, multimedia, and scientific or HPC workloads, using both a Zipfian arrival mix and scaled Azure trace samples [2507.08954]. Default parameters are reported as $D = 2$, $T = 10$, $\alpha = 2$, and a warm-pool size of 32 containers [2507.08954]. The principal metrics are weighted mean latency, tail or jitter per function, throughput and GPU utilization, and fairness in device service time [2507.08954].

The paper reports several categories of results.

First, on service-time fairness, FCFS lets popular functions dominate in a microbenchmark with four queues, whereas MQFQ-Sticky equalizes GPU service across all four once multiple queues are active [2507.08954]. In the 24-function Zipf workload, the maximum service-time gap over 30 s windows is reported as less than 50, well below the stated theoretical bound of at most 411 [2507.08954].

Second, on end-to-end latency, MQFQ-Sticky reduces weighted mean latency by more than 2× over FCFS at moderate-to-high loads in the 24-function setting, and still improves latency by approximately 15% at higher loads when only long-running functions are present [2507.08954]. On the Azure medium trace with 19 functions, the paper reports the following comparisons [2507.08954]:

| Configuration | Reported average latency |
|---|---|
| Naive FCFS with stock GPU containers | ~3,000 s |
| FCFS + warm pool / memory management at $D=1$ | 51.8 s |
| MQFQ-Sticky at $D=1$ | 11.8 s |
| MQFQ-Sticky at $D=2$ | ~8.9 s |

The paper further states that Paella-style SJF worsens latency by 8×–20× due to head-of-line blocking of long functions, while Batch improves locality but lacks fairness and falls between FCFS and MQFQ-Sticky [2507.08954].

Third, on variability, MQFQ-Sticky cuts inter-function latency variance to about one third of FCFS and reduces per-function invocation latency variance by 3×–4× [2507.08954]. Fourth, on memory management, Prefetch+Swap reduces execution time by more than 33% versus stock UVM under 50% oversubscription and matches non-UVM ideal execution times, whereas CUDA `madvise` alone slightly worsens performance [2507.08954].

Fifth, on hardware-sharing mechanisms, the paper reports that pure MPS without MQFQ-Sticky degrades latency by 3%–240%, but that MQFQ plus MPS reduces latency by up to 80%, indicating complementarity between high-level scheduling and low-level kernel overlap [2507.08954]. On an A30, MIG with two slices increases average latency because some functions slow down on smaller slices [2507.08954]. In multi-GPU settings, adding a second V100 yields approximately 2.3× lower latency at $D=1$ and about 4× lower latency at higher $D$ [2507.08954].

The paper’s headline summary is that MQFQ-Sticky reduces function latency by 2× to 20× compared to existing GPU and CPU queueing policies [2507.08954].

## 6. Interpretation, limitations, and relation to adjacent systems

MQFQ-Sticky is presented as an integrated scheduler rather than merely a queueing discipline. Its principal claim is that fair queueing, anticipatory keep-alive, and GPU memory residency must be co-optimized in serverless GPU environments because latency inflation is driven simultaneously by queueing, cold starts, and memory thrashing [2507.08954]. A plausible implication is that the framework’s contribution lies as much in the control surface it exposes—$T$, $D$, TTL, warm-pool sizing, and utilization caps—as in the formal fairness bound itself.

The paper is explicit about several limitations. Under very limited VRAM, the system falls back more frequently to GPU-cold but host-warm restarts, and LRU swapping may evict a container’s pages shortly before reuse [2507.08954]. Highly bursty arrivals can saturate $D$ and increase backlog even when TTL reduces thrashing [2507.08954]. Very long functions may still experience fairness drift when $T$ is large, though per-function weights can be used to prioritize SLAs [2507.08954]. Certain functions slow significantly on small MIG slices, and the framework cannot rewrite kernels to adapt [2507.08954]. The paper also notes that, as with any shared accelerator, standard CUDA sharing entails device- and driver-level side-channel risks, though MQFQ-Sticky is not claimed to introduce new ones beyond ordinary GPU sharing [2507.08954].

In relation to earlier systems, the paper positions MQFQ-Sticky against GPU sharing systems for ML such as Gandiva, Themis, Salus, and Paella, as well as against MPS- and MIG-based hardware sharing and disaggregated accelerators such as rCUDA and DGSF [2507.08954]. Its distinguishing features are stated to be black-box acceleration for arbitrary functions, an integrated UVM-based memory-management layer, anticipatory scheduling, and locality controls that retain fairness bounds [2507.08954].

The paper identifies several future directions: extending weights $w_i$ to dynamic SLAs, learning $\alpha$ per function from reuse-distance distributions, coordinating MQFQ-Sticky with cluster-level locality-aware load balancers and autoscaling, and adapting similar policies to other accelerators such as TPUs and FPGAs [2507.08954]. These directions are framed not as established results but as extensions of the same fairness-and-locality design principle [2507.08954].

Source: https://www.emergentmind.com/topics/mqfq-sticky