---
title: Fluid-Guided Online Scheduling
url: https://www.emergentmind.com/topics/fluid-guided-online-scheduling
type: topic
---

# Fluid-Guided Online Scheduling

Fluid-guided online scheduling refers to a class of real-time scheduling and control policies that leverage tractable fluid (deterministic) approximations of underlying stochastic or high-dimensional queueing/network systems to inform, benchmark, or directly guide online decision-making. These methods are increasingly prominent in large-scale, resource-constrained systems such as distributed processing networks, cloud serving infrastructures, large language model (LLM) inference engines, and multiclass queueing environments. Fluid-guided algorithms exploit deterministic, usually continuous, “fluid” limits to translate high-complexity online scheduling challenges into optimization problems or stylized policy classes that achieve provable near-optimality or significant practical performance gains within system constraints.

## 1. Mathematical Formulation and the Role of Fluid Models

The foundation of fluid-guided scheduling lies in describing a dynamic processing or service system using a deterministic "fluid" model—a limiting behavior where job/service arrivals and departures become continuous flows rather than discrete events. For example, in a separated continuous linear program (SCLP) for fluid processing networks, the system is modeled by buffer (queue) levels $x(t)$ and control (service) rates $u(t)$ over a time horizon $[0,T]$, subject to resource and dynamical constraints:
\[
\max_{u(\cdot),\,x(\cdot)} \int_{0}^{T} \left[ (γ+(T-t)c)^{T}u(t) + d^{T}x(t) \right]dt
\]
subject to
\[
\int_{0}^{t}G\,u(s)ds + F\,x(t) \le \alpha + at,\quad H\,u(t)\le b,\quad u(t),x(t)\ge 0
\]
where $G$ is the control-to-buffer routing matrix, $F$ encodes state constraints (such as inventories), $H$ captures capacity constraints, and $a,\alpha,b,c,d$ are exogenous data and cost/reward vectors [2103.04405].

Fluid models replace stochastic queue evolution with ODEs or integral constraints, yielding tractable deterministic optimization problems. In LLM inference scheduling, the system is similarly abstracted: prompt arrivals of various types enter pipeline stages, resource (e.g., GPU memory) consumption is represented as fluid variables, and constraints are enforced on the instantaneous or time-averaged usage [2504.11320].

## 2. Fluid-Guided Online Algorithms and Threshold Policies

A central methodological advance in fluid-guided online scheduling is constructing online decision rules that are explicitly informed by the solution structure or equilibrium of the fluid model. This involves either direct computation of fluid-optimal values, or design of index- or threshold-based policies that closely track the optimal allocation predicted by the fluid system.

In queueing contexts with heterogeneous job classes and service priorities, an archetypal fluid-guided rule maintains a running estimate of system occupancy (e.g., number of queued tolerant jobs), consults a Pareto-efficient "lookup" derived from the fluid trade-off frontier, and implements a randomized or deterministic sub-policy at each state [1904.06480]. For instance, the Pareto-complete scheduling class $\varphi_{L,d}$ switches between admission control parameters or blocking probabilities based on threshold exceedance of queue levels, as dictated by fluid limit equations.

In LLM inference systems, the "WAIT" and "Nested WAIT" algorithms are constructed to meet fluid equilibrium thresholds for the number of prompts of each type in each stage, such that scheduling decisions for when to launch processing batches are matched to the deterministic fluid system’s capacity and allocation constraints. These algorithms maintain per-stage prompt counters and trigger batch processing when thresholds—explicitly computed from the fluid solution—are met [2504.11320].

## 3. Fluid Limit Analysis, Conservation Laws, and Trade-Off Frontiers

Analysing the fluid limit system often reveals pseudo-conservation laws or aggregated performance relationships that fundamentally constrain achievable trade-offs. For example, in two-class queues (eager and tolerant jobs):
- The tolerant class sees an effective service rate given by the leftover capacity after accounting for blocking and service of the eager class, which is a function only of the stand-alone blocking probability for the eager class.
- Mean sojourn time and blocking probability for the two classes are linked by a “fluid conservation law”: for a given blocking policy, one can precisely locate the achievable $(\text{blocking probability}, \text{mean delay})$ pair on a trade-off curve [1904.06480].

Similarly, for resource-constrained inference serving (LLMs under GPU memory limits), the limiting throughput and memory consumption are computed by balancing average arrival and completion rates through fluid equations; constraints on average memory lead directly to capacity constraints on achievable throughput and latency [2504.11320].

The Pareto frontier of achievable performance—such as pairs of mean latency and throughput—can be characterized by threshold or mixing policies parameterized to ride this fluid-optimal trade-off.

## 4. Online Implementation and Receding-Horizon Control

Practical deployment of fluid-guided online scheduling hinges on warm-startable algorithms and event-driven updating schemes. In SCLP-based systems, rolling-horizon control operates as follows:
- Upon arrival of new information (e.g., job burst, capacity change), the current state is truncated to the present time.
- Constraints and inflow data are updated, and the SCLP is re-solved for the remainder of the horizon, typically by continuing the previous solution path—resulting in minimal additional computational overhead [2103.04405].

In batch scheduling contexts (such as LLM serving), each scheduling cycle (or batch) is controlled by fluid-derived thresholds; the system only launches batches when expected buffer buildup matches the fluid equilibrium required to maintain optimal throughput and latency. The “Nested WAIT” design adapts to additional uncertainty (e.g., unknown output lengths) by partitioning into nested segments with type-specific thresholds, coupled via their evolution in the fluid model [2504.11320].

## 5. Computational and Theoretical Performance

Empirical studies consistently report that fluid-guided online scheduling yields computational advantages and strong optimality guarantees in large-scale or heavy-traffic regimes:
- In large-scale SCLP instances, the revised SCLP-simplex algorithm produces exact optimal values significantly faster than finely discretized LP solvers, with observed speed-ups of $10\times$ to $500\times$ as problem size grows [2103.04405].
- The WAIT/Nested WAIT algorithms in LLM inference sustain $10\%-30\%$ higher throughput over state-of-the-art online batching baselines (such as vLLM and Sarathi) without exceeding memory limits, with latency remaining comparable or slightly improved [2504.11320].
- Theoretical results establish that under heavy-traffic scaling, the gap (fluid throughput $-$ policy throughput) is $O((\zeta T)^{-1/2})$ or better, and latency/TTFT scales as $O(1)$ when fluid threshold inequalities are strictly satisfied.
- Standard policies such as FCFS may incur $\Omega(1)$ throughput deficits even when resource constraints are met, underlining the necessity of fluid-informed designs for optimality.

## 6. Broader Impact, Limitations, and Extensions

Fluid-guided scheduling bridges methodologies from operations research (control, queueing theory, large deviation analysis) and machine learning (resource-efficient model serving, adaptive inference). The approach:
- Enables tractable, provably near-optimal solutions in settings with high-dimensionality and dynamic constraints, avoiding the prohibitive overhead of full stochastic optimization or exhaustive simulation.
- Extends readily to model-predictive and robust/receding horizon control, as well as to hybrid systems combining discrete and continuous elements.

Notable limitations include:
- Reliance on accurate knowledge of key parameters (arrival rates, task sizes) to set thresholds; adaptive thresholding under nonstationary or adversarial workloads remains an open area.
- Extensions to multi-resource, multi-agent, and distributed settings may require new fluid approximations to account for inter-node/memory/pipeline bottlenecks.
- Some classes of time-varying or semi-infinite constraint systems remain beyond current fluid-guided scheduling theory [2103.04405].

A plausible implication is that further integration of fluid models with learning-based or data-driven parameter estimation may yield scalable and self-tuning schedulers for highly heterogeneous, unpredictable environments.

## 7. Representative Applications

Fluid-guided online scheduling has demonstrated efficacy across several domains:

| Domain                | Key Challenge              | Fluid-Guided Solution (Example)                   |
|-----------------------|---------------------------|---------------------------------------------------|
| Semiconductor fabs    | High-dimensional process flow | SCLP-simplex for transitional scheduling       |
| Multiclass queues     | Blocking vs. delay tradeoff  | Pareto-complete threshold policies              |
| LLM inference serving | Dynamic memory bottlenecks   | WAIT & Nested WAIT for batch scheduling         |

For LLM inference, prompt arrivals of mixed-length and variable output are scheduled to maximize throughput and minimize latency subject to strict memory caps by precomputing thresholds via a fluid model and applying them directly to scheduling decisions online [2504.11320]. In classical service networks, fluid-optimal policies inform admission control, server splitting, and resource allocation to balance loss and delay in non-convex trade-off regimes [1904.06480].

Fluid-guided online scheduling is thus a unifying paradigm for resource-constrained, high-load, and multi-class service environments, offering both rigorous theoretical guarantees and documented empirical impact.

Source: https://www.emergentmind.com/topics/fluid-guided-online-scheduling