MQFQ-Sticky: Fair GPU Scheduling Framework

• The paper introduces MQFQ-Sticky, a framework that integrates fair queueing, anticipatory scheduling, and GPU memory stickiness to optimize FaaS performance.
• It leverages per-function queues with tunable controls (T, D, TTL) to balance locality, fairness, and latency by reducing cold starts and memory thrashing.
• Empirical evaluations show that MQFQ-Sticky cuts function latency by up to 20× compared to FCFS, ensuring equitable GPU service across diverse workloads.

MQFQ-Sticky is a scheduling and memory-management framework for providing GPU acceleration to Functions as a Service in a black-box manner, without modifying function code, in settings where containerized serverless workloads encounter limited GPU concurrency, high cold-start overheads, and heavy queueing of function invocations (Fuerst et al., 11 Jul 2025). It stands for Multi-Queue Fair Queuing with Stickiness and adapts fair queueing and anticipatory scheduling principles from I/O scheduling to function scheduling on GPUs, with the stated objective of balancing locality, fairness, and latency in dynamic, heterogeneous FaaS workloads (Fuerst et al., 11 Jul 2025).

1. System model and problem setting

MQFQ-Sticky is motivated by the mismatch between GPUs and the FaaS programming model. Functions-as-a-Service is organized around short, stateless functions in isolated sandboxes, with elasticity and scale-to-zero as core features, whereas GPUs are designed to batch large kernels, offer limited concurrency, and expose relatively small on-device memories with tight coupling between compute and memory placement (Fuerst et al., 11 Jul 2025). In containerized FaaS, every invocation may need to attach a device, initialize frameworks, and load model and operator libraries; GPU cold starts frequently add seconds of latency and can be 75× worse than CPU cold starts (Fuerst et al., 11 Jul 2025).

The paper situates MQFQ-Sticky in workloads that include machine learning inference and training, audio and video pipelines, and scientific computing, with inter-arrival times, execution times, and memory footprints varying by orders of magnitude (Fuerst et al., 11 Jul 2025). In such environments, naive FCFS or simple least-loaded queueing quickly produces long queues and high tail latency, because invocations serialize behind cold-start-heavy functions and memory thrashing (Fuerst et al., 11 Jul 2025). MQFQ-Sticky therefore treats scheduling and GPU memory management as a coupled control problem rather than as separable runtime services (Fuerst et al., 11 Jul 2025).

The framework is described as delivering black-box GPU acceleration for arbitrary functions running in standard containers (Fuerst et al., 11 Jul 2025). This is significant because prior GPU-sharing systems discussed in the paper, including Gandiva, Themis, Salus, and Paella, target inference or training workloads with more application-specific profiling or kernel-level control, whereas MQFQ-Sticky is explicitly designed for diverse, isolated serverless functions (Fuerst et al., 11 Jul 2025).

2. Queueing model, fairness mechanism, and tunable controls

The scheduler uses one queue per function and tracks virtual service times so that backlogged functions receive equal shares of GPU time (Fuerst et al., 11 Jul 2025). The formulation is adapted from weighted fair queueing, where the paper recalls the classic tag update

$$F_i^k = \max(F_i^{k-1}, V(a_i^k)) + \frac{L_i^k}{w_i},$$

with $V(\cdot)$ the system virtual time, $L_i^k$ the job’s service demand, and $w_i$ the class weight (Fuerst et al., 11 Jul 2025). In MQFQ-Sticky, each function $k$ has its own queue, and upon dispatch the queue’s virtual time is incremented by its historical average GPU service time $\tau_k$; with weights $w_i$ this yields proportional fairness in GPU service (Fuerst et al., 11 Jul 2025).

Three tunable controls organize the fairness–locality trade-off.

Parameter	Meaning	Stated effect
$T$	Queue over-run threshold	Larger $T$ increases locality and batches at the cost of looser fairness
$D$	Allowed device concurrency	Can be fixed or dynamically adjusted from utilization feedback
TTL	Per-queue keep-alive time	Empty queues remain active for $V(\cdot)$0

The scheduler maintains a global virtual time anchor, denoted Global, defined as the minimum virtual time over all queues (Fuerst et al., 11 Jul 2025). A queue can dispatch while its virtual time satisfies VT < Global + T; this allows controlled queue over-run to capitalize on temporal locality while retaining formal fairness bounds (Fuerst et al., 11 Jul 2025). The paper presents the fairness guarantee for backlogged functions $V(\cdot)$1 and $V(\cdot)$2 as

$V(\cdot)$3

where $V(\cdot)$4 is total device service time and $V(\cdot)$5 is average runtime in the interval (Fuerst et al., 11 Jul 2025). The implementation defaults to $V(\cdot)$6, though per-tenant or per-function priorities are stated to be conceptually supported (Fuerst et al., 11 Jul 2025).

Anticipatory scheduling is another core component. Inspired by anticipatory disk I/O scheduling, the framework briefly waits to keep recently active queues hot, anticipating imminent invocations and avoiding thrashing (Fuerst et al., 11 Jul 2025). Specifically, an empty queue is not immediately marked inactive; instead it remains active for

$V(\cdot)$7

where $V(\cdot)$8 is the function’s inter-arrival time and $V(\cdot)$9 is tunable (Fuerst et al., 11 Jul 2025). The paper states that even $L_i^k$0 yields noticeable gains, and that per-function TTL using observed IATs outperforms a global TTL by about 15% (Fuerst et al., 11 Jul 2025).

This suggests that MQFQ-Sticky does not enforce strict processor-sharing semantics. Rather, it admits bounded fairness drift in exchange for preserving warm state and batching opportunities, with $L_i^k$1, $L_i^k$2, and TTL acting as explicit operator controls (Fuerst et al., 11 Jul 2025).

3. GPU memory stickiness and runtime residency management

The “stickiness” in MQFQ-Sticky refers to GPU memory and container residency. The scheduler is co-designed with a lightweight CUDA interposition shim using LD_PRELOAD that intercepts cuMemAlloc and replaces it with cuMemAllocManaged (Fuerst et al., 11 Jul 2025). This provides transparent oversubscription and allows the control plane to steer memory placement (Fuerst et al., 11 Jul 2025).

The framework maintains a pool of GPU-warm containers for active functions and attempts to keep per-function state on device where possible, including model weights, operator kernels, CUDA runtime state, and any pre-initialized execution context (Fuerst et al., 11 Jul 2025). When a queue becomes active, the system prefetches Unified Virtual Memory regions to the device using cuMemPrefetchAsync, overlapping that work with control-plane marshaling to keep prefetch off the critical path (Fuerst et al., 11 Jul 2025). When a queue is throttled or inactive, on-device pages are evicted using LRU and moved back to host memory, again via cuMemPrefetchAsync, producing what the paper terms a “GPU-cold but host-warm” restart (Fuerst et al., 11 Jul 2025).

The residency policy is summarized by the score

$L_i^k$3

with operator-controlled weights $L_i^k$4 and reuse_prob_i derived from IAT estimates (Fuerst et al., 11 Jul 2025). Active queues with higher score receive prefetch priority, whereas throttled or inactive queues are demoted and eventually evicted in LRU order to respect GPU memory capacity (Fuerst et al., 11 Jul 2025).

This co-design is central to the framework’s interpretation of locality. The paper argues that larger batches and longer active periods lower latency because they reduce cold starts and repeated swapping of large model and operator state, but that these same behaviors can drift from strict fairness if not bounded by the queue over-run mechanism (Fuerst et al., 11 Jul 2025).

4. Dispatcher logic, isolation model, and implementation structure

MQFQ-Sticky runs per worker and integrates with NVIDIA-Docker containers and, when available, with MPS and MIG (Fuerst et al., 11 Jul 2025). It keeps one MQFQ queue per function on the worker across one or more physical or virtual GPUs (Fuerst et al., 11 Jul 2025). The dispatcher performs the following sequence (Fuerst et al., 11 Jul 2025):

1. Compute Global = min VT over all queues.
2. Update queue states:
   • if empty and no in-flight invocations, mark inactive if now − last_exec ≥ TTL; otherwise keep active;
   • if VT − Global ≥ T, mark throttled; else active.
3. Form the candidate set of active queues with backlog and VT < Global + T.
4. Sort candidates by descending queue length; for D > 1, break ties by ascending in_flight.
5. Obtain a concurrency token via get_D_token(chosen).
6. Dispatch chosen.pop(), increment its VT by τ_k, and update Global if needed.

The paper states that this heuristic preserves MQFQ’s fairness constraints while intensifying locality, because it prefers longer queues within the fairness window and avoids oversaturating a single function concurrently when D > 1 (Fuerst et al., 11 Jul 2025). It also states that the per-dispatch work is $L_i^k$5 to scan queues and $L_i^k$6 to sort candidates, while noting that priority structures can reduce this to $L_i^k$7 in principle (Fuerst et al., 11 Jul 2025).

Isolation is preserved at the container boundary (Fuerst et al., 11 Jul 2025). User code is unmodified, runs in standard containers, and communicates through the platform’s agent; the LD_PRELOAD shim only interposes CUDA memory APIs and does not alter application logic (Fuerst et al., 11 Jul 2025). When present, MPS is launched once per GPU and all function containers attach to it, improving low-level kernel overlap while MQFQ controls high-level fairness and locality (Fuerst et al., 11 Jul 2025). MIG slices are treated as distinct vGPUs, though the paper notes that some functions degrade on smaller slices if they assume full-device resources (Fuerst et al., 11 Jul 2025).

The implementation is integrated into Iluvatar’s FaaS control plane, with approximately 3,000 lines of Rust, NVML polling every 200 ms, and a dispatcher thread (Fuerst et al., 11 Jul 2025). Hardware used in evaluation includes an NVIDIA V100 system and an NVIDIA A30 system (Fuerst et al., 11 Jul 2025).

5. Empirical behavior and measured performance

The evaluation covers 24 function types spanning ML inference, multimedia, and scientific or HPC workloads, using both a Zipfian arrival mix and scaled Azure trace samples (Fuerst et al., 11 Jul 2025). Default parameters are reported as $L_i^k$8, $L_i^k$9, $w_i$0, and a warm-pool size of 32 containers (Fuerst et al., 11 Jul 2025). The principal metrics are weighted mean latency, tail or jitter per function, throughput and GPU utilization, and fairness in device service time (Fuerst et al., 11 Jul 2025).

The paper reports several categories of results.

First, on service-time fairness, FCFS lets popular functions dominate in a microbenchmark with four queues, whereas MQFQ-Sticky equalizes GPU service across all four once multiple queues are active (Fuerst et al., 11 Jul 2025). In the 24-function Zipf workload, the maximum service-time gap over 30 s windows is reported as less than 50, well below the stated theoretical bound of at most 411 (Fuerst et al., 11 Jul 2025).

Second, on end-to-end latency, MQFQ-Sticky reduces weighted mean latency by more than 2× over FCFS at moderate-to-high loads in the 24-function setting, and still improves latency by approximately 15% at higher loads when only long-running functions are present (Fuerst et al., 11 Jul 2025). On the Azure medium trace with 19 functions, the paper reports the following comparisons (Fuerst et al., 11 Jul 2025):

Configuration	Reported average latency
Naive FCFS with stock GPU containers	~3,000 s
FCFS + warm pool / memory management at $w_i$1	51.8 s
MQFQ-Sticky at $w_i$2	11.8 s
MQFQ-Sticky at $w_i$3	~8.9 s

The paper further states that Paella-style SJF worsens latency by 8×–20× due to head-of-line blocking of long functions, while Batch improves locality but lacks fairness and falls between FCFS and MQFQ-Sticky (Fuerst et al., 11 Jul 2025).

Third, on variability, MQFQ-Sticky cuts inter-function latency variance to about one third of FCFS and reduces per-function invocation latency variance by 3×–4× (Fuerst et al., 11 Jul 2025). Fourth, on memory management, Prefetch+Swap reduces execution time by more than 33% versus stock UVM under 50% oversubscription and matches non-UVM ideal execution times, whereas CUDA madvise alone slightly worsens performance (Fuerst et al., 11 Jul 2025).

Fifth, on hardware-sharing mechanisms, the paper reports that pure MPS without MQFQ-Sticky degrades latency by 3%–240%, but that MQFQ plus MPS reduces latency by up to 80%, indicating complementarity between high-level scheduling and low-level kernel overlap (Fuerst et al., 11 Jul 2025). On an A30, MIG with two slices increases average latency because some functions slow down on smaller slices (Fuerst et al., 11 Jul 2025). In multi-GPU settings, adding a second V100 yields approximately 2.3× lower latency at $w_i$4 and about 4× lower latency at higher $w_i$5 (Fuerst et al., 11 Jul 2025).

The paper’s headline summary is that MQFQ-Sticky reduces function latency by 2× to 20× compared to existing GPU and CPU queueing policies (Fuerst et al., 11 Jul 2025).

6. Interpretation, limitations, and relation to adjacent systems

MQFQ-Sticky is presented as an integrated scheduler rather than merely a queueing discipline. Its principal claim is that fair queueing, anticipatory keep-alive, and GPU memory residency must be co-optimized in serverless GPU environments because latency inflation is driven simultaneously by queueing, cold starts, and memory thrashing (Fuerst et al., 11 Jul 2025). A plausible implication is that the framework’s contribution lies as much in the control surface it exposes—$w_i$6, $w_i$7, TTL, warm-pool sizing, and utilization caps—as in the formal fairness bound itself.

The paper is explicit about several limitations. Under very limited VRAM, the system falls back more frequently to GPU-cold but host-warm restarts, and LRU swapping may evict a container’s pages shortly before reuse (Fuerst et al., 11 Jul 2025). Highly bursty arrivals can saturate $w_i$8 and increase backlog even when TTL reduces thrashing (Fuerst et al., 11 Jul 2025). Very long functions may still experience fairness drift when $w_i$9 is large, though per-function weights can be used to prioritize SLAs (Fuerst et al., 11 Jul 2025). Certain functions slow significantly on small MIG slices, and the framework cannot rewrite kernels to adapt (Fuerst et al., 11 Jul 2025). The paper also notes that, as with any shared accelerator, standard CUDA sharing entails device- and driver-level side-channel risks, though MQFQ-Sticky is not claimed to introduce new ones beyond ordinary GPU sharing (Fuerst et al., 11 Jul 2025).

In relation to earlier systems, the paper positions MQFQ-Sticky against GPU sharing systems for ML such as Gandiva, Themis, Salus, and Paella, as well as against MPS- and MIG-based hardware sharing and disaggregated accelerators such as rCUDA and DGSF (Fuerst et al., 11 Jul 2025). Its distinguishing features are stated to be black-box acceleration for arbitrary functions, an integrated UVM-based memory-management layer, anticipatory scheduling, and locality controls that retain fairness bounds (Fuerst et al., 11 Jul 2025).

The paper identifies several future directions: extending weights $k$0 to dynamic SLAs, learning $k$1 per function from reuse-distance distributions, coordinating MQFQ-Sticky with cluster-level locality-aware load balancers and autoscaling, and adapting similar policies to other accelerators such as TPUs and FPGAs (Fuerst et al., 11 Jul 2025). These directions are framed not as established results but as extensions of the same fairness-and-locality design principle (Fuerst et al., 11 Jul 2025).