Load-Balanced Stochastic Length (LBSL)

• Load-Balanced Stochastic Length (LBSL) is a framework that uses length-aware partitioning and stochastic modeling to balance heterogeneous workloads in systems like LLM inference and multi-machine scheduling.
• It employs dynamic programming for pipeline partitioning and adaptive boundary control to mitigate performance bottlenecks caused by variable sequence lengths and job sizes.
• Decentralized bid-ask load balancing and the L-function method enable efficient approximation algorithms that reduce latency and boost throughput in high-demand GPU clusters.

Load-Balanced Stochastic Length (LBSL) characterizes a class of load balancing and scheduling strategies for systems managing requests or jobs with substantial heterogeneity in resource demands—most notably, variable sequence lengths in LLM inference and stochastic job sizes in multi-machine scheduling. LBSL unifies techniques where balanced distribution, length-aware partitioning, and stochastic modeling are applied to minimize key performance metrics such as latency, tail-latency, and load norm. Applications span high-throughput GPU-based LLM serving systems and stochastic optimization in combinatorial allocation problems, leveraging mathematical constructs including dynamic programming for pipeline planning and the L-function method for moment-based objectives (Yuan et al., 22 Dec 2025, Molinaro, 2018).

1. Stochastic Length Heterogeneity in Modern Systems

Modern LLM inference workloads exhibit extreme variability in sequence lengths, especially as models expose context windows exceeding 128K tokens. In serving architectures employing Multi-Instance LLM Serving (MILS), GPU attention kernels are critically bottlenecked by sequence-length heterogeneity within a batch. Empirical findings demonstrate that batching requests of disparate lengths (range: 200–50K tokens) can inflate latency by 1.1–2.1× even under constant aggregate workload (Yuan et al., 22 Dec 2025). The principal challenge is that batch-based execution (e.g., attention with $O(n^2)$ and $O(n)$ regimes for prefill and decode phases, respectively) is memory-bound at long lengths and synchronizes across GPU streaming multiprocessors (SMs), resulting in worst-case per-batch stalls. The global arrival process for requests is highly skewed, rendering naive per-instance balancing ineffective.

In classic load balancing for $m$ machines and $n$ jobs, each job $j$ assigned to machine $i$ incurs random load $Y_{ij}$. Assignment policies are evaluated by the expected $\ell_p$-norm of loads, $\mathbb{E}[\|L\|_p]$, and must manage both variance in $Y_{ij}$ and systemic bottlenecks arising from the aggregate distribution of task lengths (Molinaro, 2018).

2. Dynamic Programming for Pipeline Partitioning

LBSL as embodied in the L4 runtime system addresses length heterogeneity cluster-wide by partitioning $O(n)$0 identical instances into $O(n)$1 "stages"; each stage handles a contiguous length range, and requests are migrated downstream as their lengths exceed defined boundaries (Yuan et al., 22 Dec 2025). Stage boundaries are determined by minimizing a learned quality-of-experience (QoE) objective for batched processing. For a batch $O(n)$2 of $O(n)$3 requests with input lengths $O(n)$4 and current lengths $O(n)$5,

$O(n)$6

with features $O(n)$7, $O(n)$8, $O(n)$9, $m$0, $m$1, and $m$2 are learned coefficients. The multistage partitioning problem is cast as computing

$m$3

with $m$4 the set of requests with lengths in $m$5 and $m$6 estimating migration cost. The search is made tractable via bucketing of sequence lengths into $m$7 bins and a two-phase heuristic combining E-stage solution construction (by assigning one instance per stage) and greedy adjacent-stage merging to optimize QoE.

3. Runtime Range Refinement and Adaptive Boundary Control

Offline partitioning is insufficient under nonstationary workloads. Periodic online range refinement is performed by each instance’s LoadTracker: local and downstream active sequence lengths are aggregated, and an optimal split is chosen to minimize the sum of batched QoEs,

$m$8

where $m$9 is the sorted array of observed lengths. Smoothing via exponential moving averages and thresholds on active request counts mitigate boundary thrashing. The initial boundary seed is inherited from the offline DP. This dynamic adaptation enables the system to respond to drift or sudden changes in the request-length distribution, maintaining stage homogeneity and mitigating the effect of global non-stationarity (Yuan et al., 22 Dec 2025).

4. Decentralized Bid-Ask Load Balancing

Imbalances may persist even under optimal partitioning due to stochastic arrivals, variable completion rates, and context evolution in LLMs. L4 implements a decentralized "bid-ask" protocol for both inter- and intra-stage load transfer, inspired by specialist markets. When an instance (sender) is overloaded, it issues an "ask" to candidate receivers indicating its buffered token count ($n$0). Receivers reply with a "bid" indicating their earliest service time ($n$1). The sender selects among candidates with the lowest $n$2, choosing the first to respond. The selected receiver attempts live migration of the request's KV cache (using cudaMemcpyPeerAsync or RDMA), with preemption and starvation detection mechanisms to ensure liveness. Migration is performed in asynchronous multi-rounds to overlap communication and computation. This fully decentralized protocol ensures robust, distributed balancing without centralized coordination (Yuan et al., 22 Dec 2025).

5. Length-Specialized Multi-Stage Pipeline

The integration of offline pipeline planning, runtime range refinement, and decentralized bid-ask balancing yields a length-specialized pipeline model. At initialization, profiling determines optimal $n$3 coefficients and stage boundaries. Incoming requests of length $n$4 are routed to the earliest stage with $n$5; after prefill and each decode, requests growing past the upper bound $n$6 are migrated downstream. Within-stage load is balanced among instances using the bid-ask protocol. This architecture achieves length-homogeneous batches per stage, ensuring high SM utilization and eliminating worst-case length-induced attention stalls (Yuan et al., 22 Dec 2025).

6. The L-Function Method in Stochastic Load Balancing

Stochastic $n$7 load balancing (SLBp) targets minimization of $n$8 in the face of random job sizes $n$9 (Molinaro, 2018). Molinaro introduces the L-function, $j$0, as a tool to decouple the $j$1-moment objective. Latała’s theorem asserts that if $j$2, then $j$3. This allows translation of moment constraints into tractable convex programs. The approach permits constant-factor approximation algorithms for stochastic load balancing, even under complex, instance-dependent random load distributions. The rounding process utilizes Generalized Assignment Problem (GAP) techniques and multi-scale decomposition to ensure $j$4-factor approximation of the optimal expected $j$5-norm.

7. Empirical Results, Limitations, and Insights

Key experimental observations on a 16-GPU (H20/H100) cluster with 8 LLMs (3B–32B parameters) and open-source Poisson arrival traces include: up to 67% reduction in end-to-end Time-to-First-Token versus vLLM, up to 69% reduction in tail TTFT, throughput improvements of up to 2.89× over state-of-the-art schedulers, and SLO attainment improved by 2–7× across targets. The DP and bid-ask algorithms are shown to scale efficiently (DP runtime: 0.06s for 16 instances at $j$6K), and range refinement achieves 21% lower latency than count-based splits. Ablation studies highlight the necessity of accurate QoE modeling and the superiority of decentralized bid-ask balancing (47% reduction in coefficient of variation of load imbalance).

Principal trade-offs include modest planning and migration overheads (amortized by system gains), performance sensitivity to available KV cache memory (since insufficient memory restricts attention batching), and the need for future work to extend the approach to fully heterogeneous instance configurations. The cluster-wide reshaping approach resolves per-instance bottlenecks and is orthogonal to advances in single-instance schedulers or attention kernels (Yuan et al., 22 Dec 2025, Molinaro, 2018).