Stall-Free Scheduling in High-Throughput Systems

• Stall-free scheduling is a set of techniques to prevent idle resource time in high-throughput systems, using methods like chunked and layered prefill.
• It employs predictive batching and multi-stage scheduling to satisfy latency constraints such as TTFT and TBT while optimizing system throughput.
• Applied in LLM serving, offloaded training, and video streaming, these strategies yield measurable gains in energy efficiency, bandwidth usage, and performance.

Stall-free scheduling encompasses algorithmic frameworks and system designs that aim to guarantee the continuous utilization of hardware resources (e.g., GPUs, storage servers) during high-throughput serving or training workloads, thereby eliminating idle intervals (“stalls”) that arise from resource contention, pipeline bottlenecks, or misaligned batch dispatch. Inference and training systems for LLMs and distributed media streaming rely on stall-free scheduling to maximize throughput, satisfy service-level latency constraints, and minimize wasteful memory and I/O traffic. Modern approaches leverage prediction, resource modeling, and multi-stage scheduling to ensure the system never intentionally idles resources when work is pending.

1. Formal Metrics and Definitions

Stall-free scheduling is framed by several key latency and utilization metrics. In transformer-based LLM inference, the primary quantities are time-to-first-token (TTFT) and time-between-tokens (TBT):

• TTFT is the sum of prefill and first-token decode latency,

$$TTFT = T_\mathrm{prefill} + T_\mathrm{decode}^{(1)},$$

representing the elapsed time from request arrival to the first output token.

• TBT is the per-token latency for steady-state decoding, with stall-free schedules ensuring

$$T_\mathrm{decode}^{(i)} \leq T_\mathrm{prefill}^{(i+1)}$$

for all $i$, i.e., every decode finishes before the next prefill chunk completes.

• Throughput is total tokens per unit wall time:

$$\Omega = \frac{\text{Total Tokens}}{\text{Wall-Clock Time}}.$$

Resource budgets—maximum compute capacity ($C_\mathrm{max}$), memory bandwidth ($B_\mathrm{max}$), and interconnect bandwidth ($I_\mathrm{max}$)—define the feasible region. Stall-free scheduling must maximize throughput $\Omega$ subject to strict TTFT and TBT constraints under these resource bounds. In multi-tenant or multi-request systems, analogous fairness and utilization metrics (e.g., GPU utilization, token fairness, latency fairness) further guide dispatcher behavior (Lee et al., 9 Oct 2025, Wei et al., 19 Aug 2025).

2. Stall-Free Scheduling in LLM Serving

Contemporary LLM serving architectures achieve stall freedom via scheduling techniques that decouple prefill and decode operations, interleaving them to prevent idle GPU periods:

Chunked Prefill

The chunked prefill scheduling axis splits the prompt along the token dimension into fixed-size chunks, interleaving the processing of new prompt tokens (prefill) with ongoing decode steps to amortize prefill overheads. At each iteration, for up to $N_\mathrm{chunks} = \lceil L / c \rceil$ token chunks:

• PrefillChunk: traverse all layers on the chunk.
• DecodeStep: process one token for each active request.

This approach enforces $$T_\mathrm{decode}^{(i)} \leq T_\mathrm{prefill}^{(i+1)}$$, stabilizing TBT. However, for large Mixture-of-Experts (MoE) models, each prefill chunk incurs expert weight reloading, increasing memory traffic and energy use—by up to 39% for long contexts—since the MoE layers are repeatedly traversed during chunked prefill (Lee et al., 9 Oct 2025).

Layered Prefill

Layered prefill introduces a paradigm shift by partitioning the transformer model along the depth (layer axis) into $$T_\mathrm{decode}^{(i)} \leq T_\mathrm{prefill}^{(i+1)}$$0 contiguous layer groups. Each group, sized to match the workload of a typical chunk (e.g., $$T_\mathrm{decode}^{(i)} \leq T_\mathrm{prefill}^{(i+1)}$$1), sequentially processes prefill and decode—interleaving prefill for one group per iteration while all groups participate in decode. After $$T_\mathrm{decode}^{(i)} \leq T_\mathrm{prefill}^{(i+1)}$$2 rounds, prefill is complete for all groups.

Layered prefill eliminates redundant MoE expert loads: each group is prefixed exactly once per request, so total expert-weight traffic is reduced from $$T_\mathrm{decode}^{(i)} \leq T_\mathrm{prefill}^{(i+1)}$$3 to $$T_\mathrm{decode}^{(i)} \leq T_\mathrm{prefill}^{(i+1)}$$4, yielding up to 39% bandwidth and 22% per-token energy savings without sacrificing stall freedom (Lee et al., 9 Oct 2025).

3. Holistic Stall-Free Batch Scheduling

In multi-tenant LLM serving (e.g., Equinox), the stall-free property is enforced by predictive resource-aware batching. Central challenges include unobservable resource cost before execution and the “scheduling paradox”:

1. Predictive Admission: Each enqueued request is annotated with predicted output tokens, latency, memory, and GPU utilization via model-specific predictors (MoPE). This prediction is used both for fairness tracking and feasibility testing.
2. Work-Conserving Batching: Requests are batched up to memory ($$T_\mathrm{decode}^{(i)} \leq T_\mathrm{prefill}^{(i+1)}$$5) and batch-size ($$T_\mathrm{decode}^{(i)} \leq T_\mathrm{prefill}^{(i+1)}$$6) limits, ensuring at least one request is always scheduled from any nonempty queue. A batch is dispatched as soon as no further additions are feasible.
3. Stall-Free Guarantee: Under perfect predictions, the scheduler never leaves the GPU idle when requests are pending, and never admits a batch that would overrun resources or induce mid-batch evictions. This is formalized as: whenever the global queue $$T_\mathrm{decode}^{(i)} \leq T_\mathrm{prefill}^{(i+1)}$$7 is nonempty, a feasible nonempty batch $$T_\mathrm{decode}^{(i)} \leq T_\mathrm{prefill}^{(i+1)}$$8 is dispatched immediately (Wei et al., 19 Aug 2025).

Continuous updating of per-client “Holistic Fairness” scores maintains max–min fairness over latency, token usage, and resource efficiency, simultaneously optimizing stall freedom and tenant fairness.

4. Stall-Free Scheduling in Resource-Constrained Training

Offloaded LLM training often encounters severe GPU stalls, as seen in ZeRO-Offload, which synchronously updates all model parameters on the CPU, leaving the faster GPU idle for substantial portions of the iteration. ZenFlow achieves stall-free offloading by decoupling parameter updates and exploiting gradient importance:

• Important parameters (“top-k” input channels, selected by per-column gradient $$T_\mathrm{decode}^{(i)} \leq T_\mathrm{prefill}^{(i+1)}$$9 norm) are updated in-place on the GPU; others are accumulated on the CPU and updated asynchronously in double-buffered batches.
• Only the small set of “important” gradients is synchronized across shards, with empirical retention rates exceeding 90% for top-k channels across 100 steps, enabling lightweight selection and communication.
• Asynchronous double buffering (with accumulation interval $i$0) ensures the CPU optimizer is fully overlapped with $i$1, rendering $i$2, i.e., GPU stalls are effectively eliminated.

Empirically, ZenFlow yields up to 5× end-to-end speedup, 85% reduction in GPU stall time, and ~2× reduction in PCIe traffic compared to ZeRO-Offload, while maintaining statistical convergence and accuracy (Lan et al., 18 May 2025).

5. Stall-Free Scheduling in Cloud Video Streaming

Stall-free scheduling principles are also critical in cloud-based video streaming, where segments are erasure-coded and delivered over parallel streams from distributed storage servers. The goal is to minimize playback stalls by jointly optimizing:

• Server selection and parallel-stream assignment for each video segment, based on probabilistic schedules $i$3 (server selection) and $i$4 (stream selection).
• Quality selection probabilities $i$5 and per-stream bandwidth allocation $i$6.

A mathematical bounding approach using moment generating functions (MGFs) characterizes expected stall duration, with the optimization formulated as a joint minimization of expected stall and negative mean video quality. Using block-coordinate and inner-convex (NOVA) optimizations, the framework achieves near-zero mean stall durations at moderate loads while sustaining high video quality—tracing an explicit stall–quality trade-off frontier (Alabbasi et al., 2018).

6. Implementation Guidance and System Integration

Practical considerations for deploying stall-free scheduling include:

• Parameter Tuning: Selecting appropriate chunk sizes or layer-group counts to balance scheduler overhead against per-iteration workload fits the desired TBT SLO. For extremely long contexts, hybrid chunked+layered prefill may further optimize MoE throughput (Lee et al., 9 Oct 2025).
• Resource Matching: Layered prefill is preferred for memory-bottlenecked GPUs; chunked prefill may be favorable in compute-bound settings.
• Multi-Tenant Scenarios: Predictive, fairness-aware dispatchers (e.g., Equinox) harmonize stall-freedom with inter-client fairness, leveraging continuous re-ranking and continuous batching (Wei et al., 19 Aug 2025).
• Streaming and Training: For streaming/video, increasing the number of parallel streams reduces stall; for offloaded training, the selection interval ($i$7) and proportion top-k ($i$8) mediate the speedup–convergence trade-off (Lan et al., 18 May 2025, Alabbasi et al., 2018).

Stall-free scheduling thus encompasses a family of resource-aware, algorithmically principled strategies that guarantee continuous, efficient system utilization—removing idle gaps due to batch misalignment, offload bottlenecks, or resource oversubscription. Recent advances in LLM serving, multi-tenant fairness, asynchronous offloaded training, and streaming jointly illustrate the broad applicability and significant system-level impact of stall-free methods.