---
title: Stall-Free Scheduling in High-Throughput Systems
url: https://www.emergentmind.com/topics/stall-free-scheduling
type: topic
---

# Stall-Free Scheduling in High-Throughput Systems

Stall-free scheduling encompasses algorithmic frameworks and system designs that aim to guarantee the continuous utilization of hardware resources (e.g., GPUs, storage servers) during high-throughput serving or training workloads, thereby eliminating idle intervals (“stalls”) that arise from resource contention, pipeline bottlenecks, or misaligned batch dispatch. Inference and training systems for large language models (LLMs) and distributed media streaming rely on stall-free scheduling to maximize throughput, satisfy service-level latency constraints, and minimize wasteful memory and I/O traffic. Modern approaches leverage prediction, resource modeling, and multi-stage scheduling to ensure the system never intentionally idles resources when work is pending.

## 1. Formal Metrics and Definitions

Stall-free scheduling is framed by several key latency and utilization metrics. In transformer-based LLM inference, the primary quantities are time-to-first-token (TTFT) and time-between-tokens (TBT):

- **TTFT** is the sum of prefill and first-token decode latency,
  $$
  TTFT = T_\mathrm{prefill} + T_\mathrm{decode}^{(1)},
  $$
  representing the elapsed time from request arrival to the first output token.

- **TBT** is the per-token latency for steady-state decoding, with stall-free schedules ensuring
  $$
  T_\mathrm{decode}^{(i)} \leq T_\mathrm{prefill}^{(i+1)}
  $$
  for all $i$, i.e., every decode finishes before the next prefill chunk completes.

- **Throughput** is total tokens per unit wall time:
  $$
  \Omega = \frac{\text{Total Tokens}}{\text{Wall-Clock Time}}.
  $$

Resource budgets—maximum compute capacity ($C_\mathrm{max}$), memory bandwidth ($B_\mathrm{max}$), and interconnect bandwidth ($I_\mathrm{max}$)—define the feasible region. Stall-free scheduling must maximize throughput $\Omega$ subject to strict TTFT and TBT constraints under these resource bounds. In multi-tenant or multi-request systems, analogous fairness and utilization metrics (e.g., GPU utilization, token fairness, latency fairness) further guide dispatcher behavior [2510.08055, 2508.16646].

## 2. Stall-Free Scheduling in LLM Serving

Contemporary LLM serving architectures achieve stall freedom via scheduling techniques that decouple prefill and decode operations, interleaving them to prevent idle GPU periods:

### Chunked Prefill

The chunked prefill scheduling axis splits the prompt along the token dimension into fixed-size chunks, interleaving the processing of new prompt tokens (prefill) with ongoing decode steps to amortize prefill overheads. At each iteration, for up to $N_\mathrm{chunks} = \lceil L / c \rceil$ token chunks:

- PrefillChunk: traverse all layers on the chunk.
- DecodeStep: process one token for each active request.

This approach enforces $T_\mathrm{decode}^{(i)} \leq T_\mathrm{prefill}^{(i+1)}$, stabilizing TBT. However, for large Mixture-of-Experts (MoE) models, each prefill chunk incurs expert weight reloading, increasing memory traffic and energy use—by up to 39% for long contexts—since the MoE layers are repeatedly traversed during chunked prefill [2510.08055].

### Layered Prefill

Layered prefill introduces a paradigm shift by partitioning the transformer model along the depth (layer axis) into $G$ contiguous layer groups. Each group, sized to match the workload of a typical chunk (e.g., $G(L) = \max(1, \lceil L / 512 \rceil)$), sequentially processes prefill and decode—interleaving prefill for one group per iteration while all groups participate in decode. After $G$ rounds, prefill is complete for all groups.

Layered prefill eliminates redundant MoE expert loads: each group is prefixed exactly once per request, so total expert-weight traffic is reduced from $N_\mathrm{chunks} \times W_\mathrm{reload}$ to $1 \times W_\mathrm{reload}$, yielding up to 39% bandwidth and 22% per-token energy savings without sacrificing stall freedom [2510.08055].

## 3. Holistic Stall-Free Batch Scheduling

In multi-tenant LLM serving (e.g., Equinox), the stall-free property is enforced by predictive resource-aware batching. Central challenges include unobservable resource cost before execution and the “scheduling paradox”:

1. **Predictive Admission**: Each enqueued request is annotated with predicted output tokens, latency, memory, and GPU utilization via model-specific predictors (MoPE). This prediction is used both for fairness tracking and feasibility testing.

2. **Work-Conserving Batching**: Requests are batched up to memory ($M$) and batch-size ($L_b$) limits, ensuring at least one request is always scheduled from any nonempty queue. A batch is dispatched as soon as no further additions are feasible.

3. **Stall-Free Guarantee**: Under perfect predictions, the scheduler never leaves the GPU idle when requests are pending, and never admits a batch that would overrun resources or induce mid-batch evictions. This is formalized as: whenever the global queue $Q$ is nonempty, a feasible nonempty batch $B$ is dispatched immediately [2508.16646].

Continuous updating of per-client “Holistic Fairness” scores maintains max–min fairness over latency, token usage, and resource efficiency, simultaneously optimizing stall freedom and tenant fairness.

## 4. Stall-Free Scheduling in Resource-Constrained Training

Offloaded LLM training often encounters severe GPU stalls, as seen in ZeRO-Offload, which synchronously updates all model parameters on the CPU, leaving the faster GPU idle for substantial portions of the iteration. ZenFlow achieves stall-free offloading by decoupling parameter updates and exploiting gradient importance:

- Important parameters (“top-k” input channels, selected by per-column gradient $l_2$ norm) are updated in-place on the GPU; others are accumulated on the CPU and updated asynchronously in double-buffered batches.
- Only the small set of “important” gradients is synchronized across shards, with empirical retention rates exceeding 90% for top-k channels across 100 steps, enabling lightweight selection and communication.
- Asynchronous double buffering (with accumulation interval $S$) ensures the CPU optimizer is fully overlapped with $(S\cdot T_\mathrm{GPU}) > T_\mathrm{CPU}$, rendering $T_\mathrm{stall, ZF}\approx0$, i.e., GPU stalls are effectively eliminated.

Empirically, ZenFlow yields up to 5× end-to-end speedup, 85% reduction in GPU stall time, and ~2× reduction in PCIe traffic compared to ZeRO-Offload, while maintaining statistical convergence and accuracy [2505.12242].

## 5. Stall-Free Scheduling in Cloud Video Streaming

Stall-free scheduling principles are also critical in cloud-based video streaming, where segments are erasure-coded and delivered over parallel streams from distributed storage servers. The goal is to minimize playback stalls by jointly optimizing:

- **Server selection** and **parallel-stream assignment** for each video segment, based on probabilistic schedules $q_{i, j}^{(\ell)}$ (server selection) and $p_{j, u_j}^{(\ell)}$ (stream selection).
- **Quality selection** probabilities $b_{i, \ell}$ and per-stream bandwidth allocation $w_{j,u_j}$.

A mathematical bounding approach using moment generating functions (MGFs) characterizes expected stall duration, with the optimization formulated as a joint minimization of expected stall and negative mean video quality. Using block-coordinate and inner-convex (NOVA) optimizations, the framework achieves near-zero mean stall durations at moderate loads while sustaining high video quality—tracing an explicit stall–quality trade-off frontier [1806.09466].

## 6. Implementation Guidance and System Integration

Practical considerations for deploying stall-free scheduling include:

- **Parameter Tuning**: Selecting appropriate chunk sizes or layer-group counts to balance scheduler overhead against per-iteration workload fits the desired TBT SLO. For extremely long contexts, hybrid chunked+layered prefill may further optimize MoE throughput [2510.08055].
- **Resource Matching**: Layered prefill is preferred for memory-bottlenecked GPUs; chunked prefill may be favorable in compute-bound settings.
- **Multi-Tenant Scenarios**: Predictive, fairness-aware dispatchers (e.g., Equinox) harmonize stall-freedom with inter-client fairness, leveraging continuous re-ranking and continuous batching [2508.16646].
- **Streaming and Training**: For streaming/video, increasing the number of parallel streams reduces stall; for offloaded training, the selection interval ($S$) and proportion top-k ($k$) mediate the speedup–convergence trade-off [2505.12242, 1806.09466].

Stall-free scheduling thus encompasses a family of resource-aware, algorithmically principled strategies that guarantee continuous, efficient system utilization—removing idle gaps due to batch misalignment, offload bottlenecks, or resource oversubscription. Recent advances in LLM serving, multi-tenant fairness, asynchronous offloaded training, and streaming jointly illustrate the broad applicability and significant system-level impact of stall-free methods.

Source: https://www.emergentmind.com/topics/stall-free-scheduling