TridentServe: Stage-Level Diffusion Serving System

• TridentServe is a dynamic, stage-level serving system for diffusion pipelines that individually optimizes the encode, diffuse, and decode stages.
• It co-optimizes replica placement and request dispatch, achieving up to 2.5× mean and 3.6×–4.1× P95 latency reductions compared to static pipeline methods.
• The system leverages a communication-aware ILP, adaptive placement switching, and resource-aware dispatch to meet strict SLOs in dynamic GPU clusters.

TridentServe is a stage-level serving system for diffusion pipelines, introduced for text-to-image and text-to-video inference workloads that follow an encode–diffuse–decode architecture. It replaces static, pipeline-level deployment with a dynamic stage-level serving paradigm in which a placement plan for model replicas and a dispatch plan for request execution are co-optimized, allowing different stages of the same request to use different GPU sets and parallelism strategies. In experiments on a 128-GPU cluster, it consistently improved SLO attainment and reduced average and P95 latency by up to $2.5\times$ and $3.6\times/4.1\times$ over existing works (Xia et al., 3 Oct 2025).

1. Problem setting and motivation

TridentServe is motivated by a mismatch between conventional serving systems and the internal structure of diffusion inference. Modern diffusion pipelines, including systems such as Stable Diffusion 3, Flux, CogVideoX, and HunyuanVideo, can be abstracted as a fixed three-stage chain. The Encode stage transforms user guidance into a conditioning vector $c$, is typically implemented with a Transformer encoder such as T5-XXL or LLaMA-3, has processing length $l^E_{\text{proc}} \le 500$, and benefits mainly from batching rather than parallelism. The Diffuse stage is the core generative component, sampling latent Gaussian noise and iteratively denoising it through

$$\epsilon_t = \epsilon_\theta(\mathbf{x}_t, t, c), \qquad \mathbf{x}_{t-1} = \Phi(\mathbf{x}_t, t, \epsilon_t),$$

with $l^D_{\text{proc}}$ ranging from hundreds to $10^5$ tokens, pixels, or latent positions; it is compute-bound and typically consumes more than $70\%$ of end-to-end time. The Decode stage converts latents to pixels with a VAE-like decoder such as AutoEncoderKL; it is memory-bound and accounts for $15\%$–$30\%$ of runtime (Xia et al., 3 Oct 2025).

Existing serving systems are characterized as following a static, pipeline-level paradigm. They either co-locate all three stages on each GPU or manually disaggregate stages across GPUs and servers with fixed replica counts, then assign each request a fixed number of GPUs and the same degree of parallelism across all stages. The paper argues that this is inefficient because the three stages have sharply different scaling behavior, and because requests differ substantially in target resolution, video duration, diffusion step count, and SLO slack.

Two forms of heterogeneity are central. First, heterogeneity across stages of the same request: Encode is tiny, Diffuse scales well with sequence parallelism, and Decode gains much less from added GPUs. Second, heterogeneity across requests: low-resolution requests may be best on $3.6\times/4.1\times$0–$3.6\times/4.1\times$1 GPUs, while high-resolution ones may prefer $3.6\times/4.1\times$2–$3.6\times/4.1\times$3. Workload dynamics further compound the problem, since stage demand changes under diurnal or bursty patterns, implying that the throughput-optimal replica ratio among Encode, Diffuse, and Decode also changes over time. TridentServe is designed precisely around this stage-level asymmetry, request-level heterogeneity, and workload dynamism (Xia et al., 3 Oct 2025).

2. Stage-level serving abstractions

The core abstraction in TridentServe is the separation between model-side placement and request-side dispatch. A placement plan $3.6\times/4.1\times$4 specifies how stage replicas are laid out across GPUs: $3.6\times/4.1\times$5 Each GPU can host one of six basic placement types: $3.6\times/4.1\times$6 Placements containing Diffuse are Primary Replicas (PRs); Encode-only and Decode-only placements are Auxiliary Replicas (ARs) (Xia et al., 3 Oct 2025).

A request is executed on a Virtual Replica (VR), defined as a set of GPUs whose resident stages collectively cover $3.6\times/4.1\times$7. The system distinguishes four typical VR types.

VR type	Composition	Note
V0	$3.6\times/4.1\times$8	1 GPU, no inter-stage communication
V1	$3.6\times/4.1\times$9	Encode separated
V2	$c$0	Decode separated
V3	$c$1	Fully disaggregated

The corresponding request-side abstraction is the dispatch plan. For request $c$2 and stage $c$3,

$c$4

where $c$5 is the GPU set assigned to that stage and $c$6 is the chosen parallelism strategy, such as sequence parallelism of degree $c$7. The full request plan is

$c$8

The design principle is to minimize inter-stage communication by preferring VRs that co-locate stages whenever memory allows, while allocating PRs and ARs so that per-stage throughput remains balanced. This is the key conceptual difference from pipeline-level serving, where a request receives one fixed GPU allocation for all stages (Xia et al., 3 Oct 2025).

3. System architecture and execution workflow

TridentServe comprises five main components. The Profiler performs offline profiling to collect stage-level runtime and memory data for each model, resolution, video length, and parallelism degree. Its outputs include $c$9, the runtime of stage $l^E_{\text{proc}} \le 500$0 for request $l^E_{\text{proc}} \le 500$1 at degree $l^E_{\text{proc}} \le 500$2, and the corresponding peak memory. The Monitor periodically collects GPU status, remaining memory, executing dispatch plans, estimated finish times, and per-placement-type processing rates $l^E_{\text{proc}} \le 500$3 over sliding windows. The Dynamic Orchestrator uses profiler statistics and workload information to produce an initial placement and later compute re-placement plans when stage throughput imbalance is detected. The Resource-Aware Dispatcher periodically computes dispatch plans for pending requests. The Runtime Engine executes placement and dispatch plans, including GPU grouping, communication-group management, stage preparation, inter-stage transfer with compute/communication overlap, and merged execution of adjacent stages on the same GPU set (Xia et al., 3 Oct 2025).

The serving loop is explicitly staged. Profiling is performed first, and from it the system derives request statistics. At bootstrap, the Orchestrator computes an initial placement $l^E_{\text{proc}} \le 500$4, and the Runtime Engine materializes it by loading models accordingly. During online serving, new requests arrive continuously; the Monitor detects workload-pattern changes, and if imbalance is observed, the Orchestrator computes a new placement $l^E_{\text{proc}} \le 500$5. The Runtime Engine applies this with Adjust-on-Dispatch, a migration strategy that updates placement metadata immediately but delays actual stage migration until a dispatch requires the missing stage on a GPU. The Dispatcher then computes $l^E_{\text{proc}} \le 500$6 for pending requests under the current placement and GPU state, and the Engine executes them (Xia et al., 3 Oct 2025).

This organization implies that TridentServe is not only a scheduler but a full serving substrate with explicit mechanisms for stage-level mobility. The paper emphasizes that the system avoids downtime during placement switching by storing stage replicas in shared CPU memory per node and preferring intra-node GPU–GPU copy before falling back to pinned CPU loading.

4. Placement optimization and dispatch scheduling

Placement planning is driven by a communication-aware model of VR selection. Let $l^E_{\text{proc}} \le 500$7 denote communication volume from Encode to Diffuse and $l^E_{\text{proc}} \le 500$8 that from Diffuse to Decode. Since $l^E_{\text{proc}} \le 500$9, the paper assumes $$\epsilon_t = \epsilon_\theta(\mathbf{x}_t, t, c), \qquad \mathbf{x}_{t-1} = \Phi(\mathbf{x}_t, t, \epsilon_t),$$0. VR communication costs are therefore ordered as $$\epsilon_t = \epsilon_\theta(\mathbf{x}_t, t, c), \qquad \mathbf{x}_{t-1} = \Phi(\mathbf{x}_t, t, \epsilon_t),$$1, $$\epsilon_t = \epsilon_\theta(\mathbf{x}_t, t, c), \qquad \mathbf{x}_{t-1} = \Phi(\mathbf{x}_t, t, \epsilon_t),$$2, $$\epsilon_t = \epsilon_\theta(\mathbf{x}_t, t, c), \qquad \mathbf{x}_{t-1} = \Phi(\mathbf{x}_t, t, \epsilon_t),$$3, and $$\epsilon_t = \epsilon_\theta(\mathbf{x}_t, t, c), \qquad \mathbf{x}_{t-1} = \Phi(\mathbf{x}_t, t, \epsilon_t),$$4. For request $$\epsilon_t = \epsilon_\theta(\mathbf{x}_t, t, c), \qquad \mathbf{x}_{t-1} = \Phi(\mathbf{x}_t, t, \epsilon_t),$$5, the communication-minimizing feasible VR type is defined as

$$\epsilon_t = \epsilon_\theta(\mathbf{x}_t, t, c), \qquad \mathbf{x}_{t-1} = \Phi(\mathbf{x}_t, t, \epsilon_t),$$6

From a request set $$\epsilon_t = \epsilon_\theta(\mathbf{x}_t, t, c), \qquad \mathbf{x}_{t-1} = \Phi(\mathbf{x}_t, t, \epsilon_t),$$7, TridentServe computes the fraction of requests assigned to each VR type,

$$\epsilon_t = \epsilon_\theta(\mathbf{x}_t, t, c), \qquad \mathbf{x}_{t-1} = \Phi(\mathbf{x}_t, t, \epsilon_t),$$8

and then derives GPU counts $$\epsilon_t = \epsilon_\theta(\mathbf{x}_t, t, c), \qquad \mathbf{x}_{t-1} = \Phi(\mathbf{x}_t, t, \epsilon_t),$$9, followed by a split into primary and auxiliary counts according to measured throughput rates $l^D_{\text{proc}}$0 (Xia et al., 3 Oct 2025).

Dispatch planning is centered on the Diffuse stage because Diffuse dominates latency and is the stage most sensitive to parallelism. The exact stage-level scheduling problem is formulated as maximizing the number of requests meeting deadline $l^D_{\text{proc}}$1,

$l^D_{\text{proc}}$2

subject to GPU capacity, memory, and precedence constraints; the appendix shows that this exact formulation is NP-complete by reduction from job-shop scheduling. TridentServe therefore uses a two-step myopic strategy: first solve a small ILP for Diffuse, then derive Encode and Decode heuristically from the chosen Diffuse allocation (Xia et al., 3 Oct 2025).

The ILP uses decision variables $l^D_{\text{proc}}$3 indicating whether request $l^D_{\text{proc}}$4 is dispatched on primary placement type $l^D_{\text{proc}}$5 with degree $l^D_{\text{proc}}$6, and optimizes

$l^D_{\text{proc}}$7

where $l^D_{\text{proc}}$8 is an SLO reward weight with aging and $l^D_{\text{proc}}$9 is an inter-stage communication penalty. The constraints enforce option feasibility, at most one assignment per request, GPU-capacity limits

$10^5$0

and deadline feasibility through a big-$10^5$1 construction. The solution is then mapped to a concrete GPU set $10^5$2 and a profiled sequence-parallel strategy $10^5$3. Encode and Decode are subsequently placed either on the same primary type, on a subset of the Diffuse GPUs, or on auxiliary clusters, depending on whether the selected PR contains $10^5$4 or $10^5$5 (Xia et al., 3 Oct 2025).

The paper reports that, after feasibility filtering, this ILP solves in tens of milliseconds per tick, which is intended to make online use practical even at larger cluster scales.

5. Profiling, implementation, and empirical results

The implementation is reported as approximately $10^5$6K LOC of Python and Triton kernels, with the Runtime Engine accounting for about $10^5$7K LOC and the planners about $10^5$8K LOC. Execution is asynchronous, using Ray and Python coroutines; communication uses NCCL and NiXL, with GPUDirect RDMA for cross-node transfers. The integrated models include Stable Diffusion 3 Medium, Flux.1, and PixArt for text-to-image, and CogVideoX1.5-5B, HunyuanVideo, and HunyuanDiT for text-to-video. The main parallelization strategy is sequence parallelism; model parallelism is enabled only when a model does not fit on a single GPU, in which case the minimum MP degree needed to fit is treated as a logical device for the placement and dispatch algorithms (Xia et al., 3 Oct 2025).

The evaluation uses a cluster of $10^5$9 servers, each with $70\%$0 NVIDIA L20 GPUs of $70\%$1 GB, for a total of $70\%$2 GPUs. Workloads include steady light, medium, and heavy mixes; a dynamic workload that interleaves the steady mixes; and proprietary traces with strong time-of-day effects. SLO for each request is set to $70\%$3 the latency at its optimal parallelism. Baselines range from static pipeline-level allocation with FIFO scheduling to dynamic stage-level scheduling with manual disaggregation and SRTF, but none of the baselines includes automatic placement design (Xia et al., 3 Oct 2025).

The reported quantitative results are summarized below.

Comparison target	SLO attainment	Latency reduction
Static pipeline-level baseline (B1)	$70\%$4–$70\%$5 higher	Mean $70\%$6–$70\%$7, P95 $70\%$8–$70\%$9 lower
Other pipeline-level baselines (B2–B4)	$15\%$0–$15\%$1 higher	Mean $15\%$2–$15\%$3, P95 $15\%$4–$15\%$5 lower
Stage-level baselines (B5/B6)	$15\%$6–$15\%$7 and $15\%$8–$15\%$9 higher	Mean up to $30\%$0 and $30\%$1, P95 up to $30\%$2

A further claim is that for large models such as Flux and Hunyuan, pipeline-level co-location in baselines B1–B4 frequently causes OOM, whereas TridentServe never OOMs because it automatically disaggregates and migrates stages as needed. The largest gains appear under heavy, mixed, dynamic, and proprietary workloads, especially for large models and strongly time-varying arrival patterns (Xia et al., 3 Oct 2025).

The ablation study isolates three major mechanisms. Removing placement switching increases mean latency by up to $30\%$3 under Dynamic workloads for Flux and by roughly $30\%$4–$30\%$5 under Steady workloads. Removing stage-aware allocation and forcing all stages to use Diffuse’s GPU allocation raises average latency by approximately $30\%$6–$30\%$7 and reduces SLO attainment by about $30\%$8–$30\%$9. Replacing the ILP with greedy SRTF drops SLO attainment by up to roughly $3.6\times/4.1\times$00–$3.6\times/4.1\times$01. These results support the paper’s claim that the benefits are not attributable solely to disaggregation; the combination of automatic placement, per-stage GPU sizing, and SLO-aware Diffuse scheduling is central (Xia et al., 3 Oct 2025).

6. Position in the research landscape, limitations, and broader uses of the name

TridentServe is positioned against two main lines of prior work. The first is diffusion acceleration systems such as NIRVANA, DiffServe, PatchedServe, and FlexCache, which exploit step redundancy, patch caching, or model cascades and are described as orthogonal because they are lossy acceleration techniques rather than stage-aware serving substrates. The second is pipeline-level diffusion serving such as xDiT and VideoSys, which optimize kernels and parallelism but still allocate resources at the granularity of entire pipelines rather than individual stages. The paper’s stated novelty is the explicit exploitation of stage-level heterogeneity through dynamic placement plans and stage-level dispatch plans integrated with SLO-driven scheduling (Xia et al., 3 Oct 2025).

The limitations are also specific. The system assumes a standard three-stage encode–diffuse–decode structure rather than arbitrary DAGs, assumes a homogeneous GPU pool within a cluster, relies on accurate offline profiling, and fixes model parallelism at the minimum degree needed to fit rather than exploring MP/SP co-optimization. These are architectural constraints rather than empirical anomalies, and they delimit the scope of the current design.

The name TridentServe also appears in adjacent literature as a broader systems label attached to other TRIDENT-family methods. In LLM safety, it is used descriptively for a continuous service that red-teams, evaluates, and re-aligns models using tri-dimensional data synthesis around lexical diversity, malicious intent, and jailbreak tactics (Wu et al., 30 May 2025). In temporally constrained inference, it denotes a serving layer that wraps arbitrary neural models with DFA-guided decoding so that outputs satisfy LTL$3.6\times/4.1\times$02 specifications at inference time (Collura et al., 11 Jun 2025). In malware detection, the term is used as a natural deployment label for a system combining a static GBDT classifier, LLM-generated behavioral jq rules, and direct LLM analysis through majority vote (Saul et al., 30 Apr 2026). A related but earlier networking paper on TRIDENT packet switching also describes a “TridentServe”-style system as a blueprint derived from deterministic three-stage Clos switching (Sule et al., 2019).

This suggests a broader naming pattern in which “TridentServe” functions as a service-layer interpretation of a TRIDENT method. In the strict bibliographic sense, however, the term refers most directly to the 2025 diffusion-systems paper, where it denotes a dynamic stage-level serving system for diffusion pipelines rather than a safety framework, logic-constrained decoder, malware-analysis service, or packet-switch architecture (Xia et al., 3 Oct 2025).