CascadeServe Inference System

• CascadeServe is an end-to-end inference serving system that uses model cascades with increasing complexity to balance computational cost and predictive accuracy.
• It integrates automated offline planning, certainty-based routing, and hardware-aware placement to optimize resource allocation under variable query loads.
• CascadeServe achieves 2–3x GPU savings and reduced tail latency by dynamically adapting model cascades to meet strict latency and accuracy service-level objectives.

CascadeServe is a system architecture and methodology for end-to-end inference serving using model cascades to jointly optimize cost, latency, and accuracy in production machine learning deployments. CascadeServe is constructed around the premise that ensembles of models with increasing complexity provide a fine-grained trade-off between computational cost and predictive quality, enabling efficient resource allocation and robust online adaptation to variable workloads. Integrating automated offline planning, certainty-based routing, hardware-aware placement, and responsive online execution, CascadeServe delivers substantial reductions in GPU provisioning and tail latency compared to previous approaches (Kossmann et al., 2024).

1. Formalization and Operational Objectives

The CascadeServe paradigm entails a cluster of pretrained models $M = \{m_1, ..., m_K\}$ deployed on a hardware pool $H$ comprising $G$ GPUs with $V$ GB VRAM each. The user supplies service-level objectives (SLOs), which are typically a target $P_{95}$ latency ($L^*$) or minimum end-to-end accuracy ($A^*$), in conjunction with a maximum queries per second ($QPS_\text{max}$) capacity requirement (Kossmann et al., 2024).

CascadeServe must compute:

• A gear plan ($GP$): for each QPS interval $R_j \subset [0,QPS_{\max}]$, select gear $H$0, where $H$1 is a model cascade (ordered models and certainty thresholds), $H$2 a static replication map, and $H$3 a vector of per-replica batch sizes.
• Hardware placement $H$4: model/cascade replicas pinned to GPUs, subject to VRAM constraints.
• Batching configuration $H$5 to optimize throughput given per-replica queueing.

The joint objective is to minimize GPU count (cost) subject to:

• For all $H$6, $H$7 (latency SLO), or
• For all $H$8, $H$9 (accuracy SLO),
• $G$0.

2. Offline Planning: Gear Plan Generation

The offline phase precomputes a mapping from QPS ranges to optimal gears using an EM-inspired iterative procedure. Planning decomposes into four coupled subproblems (Kossmann et al., 2024):

1. Cascade Search (SP1): Enumerate all (practically feasible) sequences of models and certainty thresholds forming cascades $G$1, retaining only Pareto-optimal elements in the accuracy-throughput space.
2. Gear Assignment (SP2): Greedily assign a selected cascade $G$2 to each QPS subinterval $G$3, prioritizing the best trade-off in the non-SLO dimension, and fall back to adjacent points on the Pareto frontier if constraints are violated.
3. Placement and Load Balancing (SP3): Formulate and solve linear programs to map model replicas to GPUs, ensuring per-replica throughput, VRAM, and load constraints are respected.
4. Batch Size Tuning (SP4): Downstream simulation incrementally increases batch sizes $G$4 per gear and replica to maximize throughput under latency and memory constraints.

A discrete event simulator is utilized to estimate inference latency $G$5 and throughput $G$6 for candidate configurations to within 5% error. Error-driven backtracking handles infeasible plans by reverting to less expensive/faster cascades or adding additional replicas as necessary.

3. Cascade Structure, Certainty Routing, and Online Execution

A CascadeServe cascade $G$7 is an ordered sequence $G$8 with monotonic certainty thresholds $G$9. Inference proceeds as follows: input $V$0 is evaluated by model $V$1. If $V$2, output $V$3 is returned; otherwise, $V$4 is forwarded to $V$5. This procedure continues until either a threshold is satisfied or the final model is reached (Kossmann et al., 2024).

Online operation is organized as a producer–consumer pipeline:

• The Producer ingests user queries, computes instantaneous QPS over a 100ms window, uses table lookups to select the active gear $V$6 for the current QPS range, and enqueues requests to GPU queues for the first model in the active cascade.
• Inference Servers/Consumers maintain in-GPU queues, polling every 5 ms; when a queue reaches its batch threshold $V$7, a batched inference is launched. Outputs either satisfy the SLO (final output) or are forwarded to the next model as prescribed by the plan.

Online decision overhead is negligible ($V$8 µs per request), insensitive to the number of supported models or active QPS intervals.

Scale-down hysteresis guards against premature upgrade to more expensive gears when QPS drops: a gear switch is triggered only if both QPS $V$9 threshold and pending queue length $P_{95}$0 ($P_{95}$1), limiting gear thrashing.

4. Hardware Placement, Replica Management, and Simulation

Placement is handled via a sequence of linear programs to minimize maximum GPU utilization while honoring:

• VRAM limits: $P_{95}$2
• Kernel occupancy: $P_{95}$3

Replication is orchestrated such that every model $P_{95}$4 in a cascade handles a requisite fraction $P_{95}$5 of queries, maintaining uniform processing rates across cascade stages to prevent queuing. The optimal number of replicas $P_{95}$6 and, if necessary, partitions $P_{95}$7 (for oversize models) is computed per stage:

$P_{95}$8

This guarantees balanced per-stage latencies and aligns throughput, subject to the physical VRAM constraints and minimizing inter-GPU communication overhead.

All candidate plan evaluations are performed in a bespoke event-driven simulator using previously profiled batch runtimes. The simulator maintains model-specific certainty traces and simulates concurrent gear operation, buffering, and contention to yield accurate p95 latency and throughput metrics.

5. Latency, Accuracy, and Resource Trade-offs

CascadeServe exposes a continuous Pareto frontier spanning latency, accuracy, and GPU cost. For every feasible SLO (in terms of p95 latency or minimum accuracy), CascadeServe achieves 2–3x lower GPU cost or, equivalently, lower latency and higher accuracy at fixed GPU provisioning relative to earlier dynamic and static batching approaches. For instance, at a fixed latency target on HellaSwag with a Llama cascade, CascadeServe provides at least 1.2x higher accuracy across the spectrum than all tested baselines; at a fixed accuracy target, it reduces p95 latency by at least 30% (Kossmann et al., 2024).

CascadeServe’s adaptive QPS-dependent gear switching suppresses tail-latency spikes induced by demand bursts: at the peak of a synthetic trace, CascadeServe controls tail (p95) latency to $P_{95}$9400 ms on a single GPU (baselines may require 3–4 GPUs or see 4 s spikes).

The EM-style gear plan search converges rapidly (within 1 s to within 3% of optimal on small spaces; 2–3 minutes for full scale). This renders the offline computation negligible relative to typical system deployment cycles.

6. Limitations, Extensions, and Future Directions

CascadeServe currently assumes:

• Validation trace and (often Zipf-like) QPS distributions generalize to deployed workload; non-stationary or unexpected production usage distributions may necessitate re-planning.
• Static placement to avoid runtime load/unload penalties; dynamic model hot-swapping or elastic VRAM utilization remains an open direction.
• Cascades are “linear” (single-path); future work may generalize to trees, graphs, or dynamic per-sample cascade structures using RL or bandit techniques.
• Gear switching is controlled by table lookup and queue-based hysteresis; more sophisticated MPC or queuing-theoretic controllers may improve utilization further.

Practical deployment highlights the necessity of per-cascade certainty calibration and batch size tuning whenever models or workloads change, as well as the ability to execute planner updates on a per-shift or periodic basis.

7. Significance in Serving System Design

CascadeServe is the first system to integrate model cascades into end-to-end inference service for joint latency, accuracy, and cost optimization, achieving 2–3x GPU savings across various workloads while keeping online logic minimal (Kossmann et al., 2024). Its level of automation in offline gear plan computation and reliance on only lightweight table lookup for online adaptation contrasts with prior ad hoc or solely batch-size-tuning solutions.

Extensions of the core ideas have appeared in HybridServe (Xue et al., 18 May 2025), which incorporates energy metrics and skip connections for DNN serving with an explicit focus on energy saving, and more recent work such as Cascadia (Jiang et al., 4 Jun 2025), which introduces explicit bi-level optimization (MILP and Tchebycheff) for routing and resource allocation in LLM cascades. These lines of work reinforce the foundational insight of CascadeServe: structured, certainty-thresholded pipeline composition with hardware-aware assignment delivers strict SLO compliance, significant resource savings, and robust adaptability in online inference settings.