TurboServe: Streaming Video Serving System

• TurboServe is a serving system for streaming video generation that uniquely manages long-lived, stateful sessions with per-chunk low latency.
• It formulates serving as an online optimization problem, jointly coordinating session placement, GPU provisioning, and dynamic autoscaling.
• TurboServe employs closed-loop control, migration-aware scheduling, and runtime batching to reduce latency by up to 51.6% and GPU cost by up to 49%.

Searching arXiv for the named papers to ground the article in the current preprint record. arXiv search: TurboServe (Jiang et al., 17 Jun 2026), Vidu S1 (Zhang et al., 3 Jul 2026), RAPID-Serve (Masood et al., 16 Jan 2026). TurboServe is a serving system for streaming video generation, a workload in which users interact with long-lived sessions that generate video progressively, chunk by chunk, under a tight per-chunk latency target. In the research literature, it is defined not as a generative model but as serving infrastructure: the paper "TurboServe: Serving Streaming Video Generation Efficiently and Economically" presents it as the first serving system designed specifically for streaming video generation workloads, while the Vidu S1 system describes it as the cluster-level streaming serving component of a real-time interactive video-generation stack (Jiang et al., 17 Jun 2026, Zhang et al., 3 Jul 2026). Across both usages, TurboServe denotes the orchestration layer that makes continuous, low-latency, multi-user, multi-GPU video generation practical.

1. Workload model and problem setting

TurboServe is motivated by the claim that streaming video generation differs fundamentally from both offline video generation and standard LLM serving (Jiang et al., 17 Jun 2026). In offline video generation, a user sends one prompt and waits for the entire video; in streaming video generation, a user keeps a persistent session, the model generates video progressively, the session can alternate between active and idle, the system must preserve session state across those pauses, and each generated chunk must meet a tight latency target so that the video feels continuous.

This workload induces two heterogeneities. The first is session duration heterogeneity: some sessions are short clips, whereas others are interactive, long-running sessions lasting many minutes. If sessions remain fixed on their initial GPU placement, that placement becomes stale as sessions accumulate, depart, or become active again, creating bottlenecks that hurt worst-case chunk latency. The second is temporal user-demand heterogeneity: users switch between active and idle, new sessions arrive and others finish, and the number of active sessions rises and falls in bursts. Provisioning for peak demand wastes GPUs during quiet periods, while provisioning for average demand under-provisions during bursts and misses latency targets.

The central implication is that serving cannot be reduced to scheduling independent requests. TurboServe treats the workload as one of stateful, session-based streaming control, in which long-lived state must be preserved, active sessions must be repeatedly scheduled, and the system must react online to both load imbalance and burstiness (Jiang et al., 17 Jun 2026).

2. Scheduling formulation

TurboServe formulates serving as an online optimization problem that jointly coordinates session placement and GPU provisioning (Jiang et al., 17 Jun 2026). At each event $t$, the decision variables are the number of active GPUs $M(t)$, the set of active GPUs $\mathcal{G}(t)$, and the placement $\phi_i(t)$ of each session $s_i$, where a session is mapped either to a GPU $g_j$ or to $\emptyset$ for suspension.

The active session set is denoted $\mathcal{S}(t)$, with $N(t)=|\mathcal{S}(t)|$. Each session has an activity indicator $\alpha_i(t)\in\{0,1\}$, and $M(t)$0 means that the user is actively interacting and the session must run on a GPU. GPU operating cost is modeled as

$M(t)$1

which includes all provisioned GPUs, even those still in scale-out boot, loading, or warmup. The latency objective is the worst-case per-chunk latency across executing sessions:

$M(t)$2

where $M(t)$3 is session $M(t)$4's chunk latency.

The optimization objective is

$M(t)$5

subject to

$M(t)$6

and

$M(t)$7

Here, $M(t)$8 is the max concurrent sessions per GPU, and $M(t)$9 controls the cost-latency tradeoff. The paper notes that if $\mathcal{G}(t)$0 is too large, the system over-spends on GPUs; if $\mathcal{G}(t)$1 is too small, it packs too densely and violates latency targets (Jiang et al., 17 Jun 2026).

3. Closed-loop control architecture

TurboServe implements this formulation through a closed-loop scheduling system with four major components: a workload detector, a placement controller, an autoscaling controller, and a session manager (Jiang et al., 17 Jun 2026). The workload detector watches recent arrivals, departures, and active/idle transitions using a sliding window. The placement controller assigns sessions to GPUs and migrates sessions when needed to reduce bottleneck latency. The autoscaling controller changes the GPU budget over time based on runtime load feedback. The session manager tracks execution, suspension, termination, and migration.

Given a fixed GPU budget $\mathcal{G}(t)$2, the placement controller approximates the problem of minimizing worst-case per-chunk latency under current capacity. It begins from the previous placement $\mathcal{G}(t)$3, removes terminated sessions, and keeps existing assignments when possible to avoid unnecessary migration. Sessions needing assignment are

$\mathcal{G}(t)$4

For each such session, the controller evaluates feasible GPUs and chooses the one that minimizes resulting bottleneck latency.

After this incremental assignment phase, the controller performs migration-aware min-max rebalancing. It identifies the bottleneck GPU $\mathcal{G}(t)$5, considers migrating sessions away from that GPU, and evaluates the gain of each candidate move:

$\mathcal{G}(t)$6

where $\mathcal{G}(t)$7 is the bottleneck latency after the move, $\mathcal{G}(t)$8 is migration overhead modeled by the $\mathcal{G}(t)$9-$\phi_i(t)$0 communication model, and $\phi_i(t)$1 is a small weight on migration cost. The controller chooses

$\phi_i(t)$2

and applies the move if the gain is positive. This procedure is repeated until no positive-gain move remains. The explicit design target is to reduce the maximum per-GPU latency rather than only the average.

The autoscaling controller uses the maximum normalized GPU load after placement,

$\phi_i(t)$3

where $\phi_i(t)$4 is the number of sessions on GPU $\phi_i(t)$5. Rather than solving the cost-latency optimization exactly, it uses target tracking on utilization with hysteresis tolerance $\phi_i(t)$6: it scales out if $\phi_i(t)$7 and scales in if $\phi_i(t)$8. The target GPU budget is

$\phi_i(t)$9

where

$s_i$0

and the adjustment is

$s_i$1

TurboServe further adapts $s_i$2 using a workload volatility signal,

$s_i$3

where $s_i$4 is the number of newly activated sessions at event $s_i$5. High fluctuation maps to larger $s_i$6 and smaller $s_i$7; stable workloads map to smaller $s_i$8 and larger $s_i$9 (Jiang et al., 17 Jun 2026).

4. Runtime mechanisms

TurboServe’s control decisions are supported by runtime mechanisms intended for online, stateful, chunked execution (Jiang et al., 17 Jun 2026). On each GPU, coalesced chunk processing collects sessions whose next chunks are ready, groups ready sessions on the same GPU into a coalesced batch, runs the model once for the batch, and writes outputs and updated state back to each session. This is the principal batching mechanism for concurrent active sessions on one GPU.

For idle periods, TurboServe supports GPU-CPU offloading for suspension and resumption. Persistent session state is copied from GPU to host memory, the session is marked suspended, and the GPU slot is released; the state is later restored when the session becomes active again. The paper explicitly states that it does not rely on recomputation for rematerialization, unlike some LLM serving systems, because video generation is compute-heavy.

For rebalancing, TurboServe implements NCCL-based GPU-GPU migration. When the placement controller moves an active session, TurboServe migrates only the per-session state region, not the model replica. Migration occurs only at chunk boundaries. The consistency protocol is: the source GPU finishes the current chunk and freezes state, the target GPU fetches and installs state, and session ownership is updated after transfer completes. The runtime design also mentions one-sided RDMA/NIXL-style GPU memory access (Jiang et al., 17 Jun 2026).

These mechanisms align with how the Vidu S1 paper situates TurboServe in an end-to-end real-time stack. There, TurboServe is described as the serving-layer umbrella under which streaming requests, long-running interactive sessions, multi-GPU coordination, low-latency execution, and economic deployment are handled, while worker-side acceleration is attributed to TurboDiffusion (Zhang et al., 3 Jul 2026).

5. Role in Vidu S1 and interaction with model-side acceleration

In Vidu S1, TurboServe is the cluster-level streaming serving component of the real-time video-generation stack, whereas TurboDiffusion is the compute acceleration layer (Zhang et al., 3 Jul 2026). The division of labor is explicit: TurboDiffusion aligns with worker-side techniques such as attention acceleration, quantized linear layers, kernel fusion, CUDA Graph, and multi-GPU parallelism, while TurboServe is the serving and orchestration layer that makes the accelerated model deployable as a stream.

The Vidu S1 paper does not provide a full standalone algorithm or pseudocode specifically for TurboServe, but it does describe the main implementation ideas of the serving stack in which TurboServe is referenced. Inference uses sliding-window decoding so the system can generate arbitrarily long sequences under bounded memory and compute. The window includes a persistent reference context from the first frame and the first generated state, cached historical states, and the current state being denoised. This design keeps per-step latency constant regardless of output length. Historical key/value states are cached before RoPE is applied, and then RoPE is reapplied as the window moves so the model preserves positional consistency without recomputing everything. The system also maintains TwinCache, consisting of a noisy cache for intermediate denoising steps, a clean cache for final refinement, and a persistent reference context.

The paper’s hardware-software co-design includes SageAttention, SpargeAttention, SLA attention acceleration, custom W8A8 per-block quantized GEMM, custom Triton/CUDA kernel fusion, CUDA Graph replay, and Ulysses-style context parallelism across multiple GPUs. TurboServe is described as the serving-layer umbrella under which these techniques are deployed in a cluster (Zhang et al., 3 Jul 2026).

Within this stack, TurboServe’s significance is practical rather than generative. It is not the reason the model knows how to generate good video; that role is attributed to the training pipeline and diffusion model design. Its role is to support sustained streaming generation, coordinate computation over multiple GPUs, maintain low and stable latency, and enable practical throughput for consumer-grade deployment (Zhang et al., 3 Jul 2026).

6. Evaluation, operating characteristics, and related systems

TurboServe is evaluated on real-world production traces from Shengshu Technology across multiple model sizes and GPU clusters with up to 64 NVIDIA B300 GPUs (Jiang et al., 17 Jun 2026). The reported hardware consists of Cluster 1 with 16 NVIDIA H20 GPUs and Cluster 2 with 64 NVIDIA B300 GPUs; each server has 8 GPUs with NVLink interconnect at 956 GB/s and RDMA-enabled InfiniBand at 50 GB/s. The models are LongLive-style streaming video generation models, including LongLive-1.3B and larger variants up to 7B in some measurements. The main metrics are worst-case per-chunk latency and total GPU operating cost. Baselines include “TurboServe$g_j$0”, “TurboServe$g_j$1”, and “TurboServe$g_j$2”.

The main quantitative results are system-level. TurboServe reduces worst-case per-chunk latency by 37.5% on average, and up to 51.6%, versus baseline systems. It reduces GPU operating cost by 37.2% on average, and up to 49.0%, while meeting the same latency constraints. Ablation studies show that removing migration increases cost by 15.0% on average, up to 28.0%, while removing autoscaling increases cost by 42.9% on average, up to 80.4%. Placement scheduling completes within 15 ms for up to 64 GPUs and within 0.1 s for up to 256 GPUs; compared with an exhaustive-search oracle, TurboServe closes the placement gap to 3.6% on average, with maximum gap 6.5%, and is more than 10× faster. Measured migration overhead is 23–30 ms, about 2–3% of per-chunk latency (Jiang et al., 17 Jun 2026).

In Vidu S1, the combined stack of TurboDiffusion, TurboServe, and streaming inference design is reported to deliver 540p (960 × 540) output at up to 42 FPS on consumer GPUs, using a 3-step configuration and providing real-time capability beyond the 30 FPS threshold. The paper also states that the infrastructure saves per-step compute and memory while preserving generation quality, enabling real-time inference on multiple GPUs, and that measured efficiency includes 42 FPS average throughput on RTX 5090 GPUs (Zhang et al., 3 Jul 2026). A plausible implication is that TurboServe’s value lies in preserving throughput as sequence length grows, so that “infinite-length” generation remains operationally stable rather than only architecturally plausible.

A frequent point of confusion is the relationship between TurboServe and adjacent systems. TurboServe is not TurboDiffusion: the former is serving and orchestration, the latter is worker-side model acceleration (Zhang et al., 3 Jul 2026). It is also not RAPID-Serve. The paper "RAPID-Serve: Resource-efficient and Accelerated P/D Intra-GPU Disaggregation" addresses LLM inference serving through intra-GPU disaggregation of prefill and decode, concurrent execution on the same GPU(s), and Adaptive Resource Management, but it does not explicitly say TurboServe is RAPID-Serve or vice versa (Masood et al., 16 Jan 2026). The relationship is therefore comparative rather than identificatory. RAPID-Serve is best understood as a comparable system or alternative design in the broader serving-systems space, whereas TurboServe is tailored specifically to streaming video generation workloads (Masood et al., 16 Jan 2026).

The discussion around TurboServe also suggests several practical boundaries. The volatility-based adaptation of $g_j$3 depends on offline-profiled mappings; this suggests that new workloads may require retuning. The migration logic is lightweight enough for the reported cluster scales, but the paper implies that extremely large clusters may require partitioned scheduling. Broader validation across more diverse deployment environments is also presented as a natural next step (Jiang et al., 17 Jun 2026).