TetriServe: Dynamic DiT Serving System

• TetriServe is a serving system for Diffusion Transformers that employs step-level sequence parallelism for elastic GPU allocation in mixed-resolution workloads.
• It integrates round-based scheduling, deadline-aware GPU allocation, and request packing to optimize service-level objectives under heterogeneous image-generation tasks.
• Evaluations demonstrate up to 32% higher SLO Attainment Ratio over fixed sequence parallelism baselines, all while preserving image quality.

Searching arXiv for the TetriServe paper and closely related serving work to ground the article. TetriServe is a serving system for Diffusion Transformer (DiT) inference under heterogeneous online image-generation workloads with per-request completion deadlines. It is defined by its use of step-level sequence parallelism, which allows the degree of sequence parallelism assigned to a request to vary across denoising steps rather than remaining fixed for the request’s entire lifetime. The system targets the mismatch between fixed-degree sequence parallelism and mixed-resolution, mixed-deadline workloads: low parallelism is efficient for small images but too slow for large ones, whereas high parallelism accelerates large images but wastes GPU resources on small ones. TetriServe addresses this by combining round-based scheduling, deadline-aware GPU allocation, and request packing, and is reported to achieve up to 32% higher SLO Attainment Ratio (SAR) than fixed-sequence-parallel baselines without degrading image quality (Lu et al., 2 Oct 2025).

1. Problem setting and motivation

TetriServe studies online serving of DiT-based text-to-image generation under explicit service-level objectives. In this setting, requests arrive online, share a finite GPU pool, and carry completion deadlines. The computational burden is substantial: the paper notes that on a single H100, generating a 2048×2048 image can take up to a minute, and 4096×4096 can exceed ten minutes (Lu et al., 2 Oct 2025). This makes deadline compliance nontrivial even before considering contention among concurrent requests.

The workload is heterogeneous primarily because resolution strongly affects token count and total computation. The paper profiles representative FLUX.1-dev resolutions as follows: 256×256 corresponds to 1024 latent tokens and 556.48 TFLOPs; 512×512 to 4096 latent tokens and 1388.24 TFLOPs; 1024×1024 to 16384 latent tokens and 5045.92 TFLOPs; and 2048×2048 to 65536 latent tokens and 24964.72 TFLOPs (Lu et al., 2 Oct 2025). These differences imply that a single static resource-allocation policy is structurally mismatched to the serving problem.

The paper’s critique of prior practice is directed at fixed sequence parallelism (SP). Existing systems assign one SP degree to an entire request. This creates a tradeoff that is favorable only for a subset of workloads: SP=1 or SP=2 works well for 256×256 but performs poorly on 2048×2048, while SP=4 or SP=8 helps large images but is inefficient for smaller ones (Lu et al., 2 Oct 2025). Under a uniform workload and tight SLOs, the paper reports that no fixed strategy exceeds SAR 0.6, which motivates a more elastic scheduling model (Lu et al., 2 Oct 2025).

A central empirical enabler is the predictability of DiT inference. Over 100 runs, the coefficient of variation of step execution time is reported as below 0.7% for all tested resolutions and SP degrees, supporting the use of offline profiling as a runtime decision substrate (Lu et al., 2 Oct 2025). This suggests that TetriServe’s control loop relies less on reactive correction than on accurate execution-time prediction.

2. Step-level sequence parallelism

TetriServe’s defining mechanism is step-level sequence parallelism. Instead of binding a request to a fixed SP degree for all denoising steps, the system allows the GPU allocation to change from one step, or small step block, to the next. A request can therefore run some steps with fewer GPUs and later steps with more GPUs, depending on urgency and expected benefit (Lu et al., 2 Oct 2025).

The rationale is twofold. First, the scaling efficiency of SP is resolution-dependent: small images often scale poorly, so high SP wastes GPU-hours, whereas large images benefit more from additional parallelism. Second, sublinear scaling means that extra GPUs may reduce latency while increasing total consumed GPU time. TetriServe therefore seeks the minimum GPU allocation that still keeps a request feasible with respect to its deadline (Lu et al., 2 Oct 2025). This is not merely a latency-minimization policy; it is a deadline-constrained resource-minimization policy embedded inside a larger packing objective.

The system differs from request-level SP in a way that has direct scheduling consequences. Under fixed SP, a running request effectively monopolizes that many GPUs until completion. Under step-level SP, the scheduler can reclaim GPUs between rounds, reassign them, and later scale a request up again. The paper’s interpretation is that this flexibility avoids the rigid head-of-line blocking induced by fixed allocations and allows the system to redistribute capacity toward urgent large jobs while preserving efficiency for small ones (Lu et al., 2 Oct 2025).

The paper also states that TetriServe improves serving without degrading image quality because it changes only the execution schedule, not the model weights, prompt, denoising algorithm, or total number of denoising steps (Lu et al., 2 Oct 2025). This suggests that step-level SP is treated as an output-preserving execution strategy, assuming numerical consistency across SP configurations.

3. System architecture and execution model

TetriServe comprises four main components: Request Tracker, Scheduler, Execution Engine, and Latent Manager (Lu et al., 2 Oct 2025). The Request Tracker maintains request metadata including resolution, deadline, remaining steps, and execution state. The Scheduler is the core decision-making module, responsible for deadline-aware GPU allocation and round-level packing. The Execution Engine runs distributed GPU workers that execute assigned diffusion steps. The Latent Manager preserves intermediate latent states across rounds and supports handoff when a request changes GPU group (Lu et al., 2 Oct 2025).

The request lifecycle is round-based. A request arrives and is registered by the Request Tracker. At the next scheduling round, the Scheduler determines whether the request should run, how many GPUs it should receive, how many steps it should execute in that round, and which GPUs it should occupy. The Execution Engine then performs the assigned steps, after which intermediate latents are retained for the next round. This repeats until all denoising steps complete and the final image is returned (Lu et al., 2 Oct 2025).

This design implies that TetriServe treats a DiT request as a stateful, elastically parallel job whose execution can be paused and resumed across rounds. The Latent Manager is therefore not ancillary; it is the mechanism that makes step-level SP operationally viable. The paper states that latent tensors are compact enough that transfer overhead is negligible, and deadline accounting therefore ignores transfer time (Lu et al., 2 Oct 2025). A plausible implication is that the scheduler is not required to internalize latent-migration cost in its core objective, though this assumption may be topology-dependent.

The system also includes practical mechanisms beyond the core scheduler. TetriServe attempts placement preservation to keep continuing requests on the same GPUs across rounds. It uses work-conserving elastic scale-up to assign idle GPUs to requests whose runtime would improve under a larger allocation. It supports selective continuous batching for identical small-resolution requests when batching does not hurt deadlines. The VAE decoder is run sequentially per request because it has large activation memory but small wall-clock cost relative to diffusion. The implementation also pre-creates NCCL process groups for many device combinations but warms only a compact commonly used subset to limit HBM usage (Lu et al., 2 Oct 2025).

4. Round-based scheduling and optimization

TetriServe makes the continuous-time online scheduling problem tractable by discretizing time into fixed-duration rounds of length $\tau$. If the current round starts at time $t_r$, the next round begins at

$t_{r+1} = t_r + \tau .$

Scheduling decisions are made only at round boundaries (Lu et al., 2 Oct 2025). This yields natural points for effective preemption and resource reconfiguration.

The paper first formulates a more general offline problem. There are $N$ total GPUs and $R$ outstanding requests. Each request $req_i$ has dependent steps $\{s_{i1}, s_{i2}, \ldots, s_{iS_i}\}$, and each step may use $k \in \{1,2,4,\ldots,N\}$ GPUs. If the runtime of step $s_{ij}$ under $k$ GPUs is $t_r$0, then the completion time of request $t_r$1 is

$t_r$2

where $t_r$3 is queueing delay before step $t_r$4 and $t_r$5 is the GPUs allocated to step $t_r$6 (Lu et al., 2 Oct 2025).

The objective is to maximize the number of requests completing by deadline:

$t_r$7

This is subject to step dependency constraints and the GPU capacity constraint

$t_r$8

The paper states that even a simplified single-step version is NP-hard by reduction to a zero-one integer linear program (Lu et al., 2 Oct 2025).

TetriServe therefore decomposes the problem. For each request, it first computes the minimal GPU allocation needed to remain deadline-feasible using offline-profiled runtimes. The optimization is

$t_r$9

This minimizes GPU-hour consumption subject to deadline feasibility (Lu et al., 2 Oct 2025). The paper’s phrasing makes clear that the scheduler is not seeking maximal acceleration per request, but the cheapest feasible execution mode.

After these candidate allocations are generated, TetriServe solves a round-level packing problem. For a request $t_{r+1} = t_r + \tau .$0 and allocation mode $t_{r+1} = t_r + \tau .$1, let $t_{r+1} = t_r + \tau .$2 denote the remaining steps and GPUs associated with that mode. In a round of duration $t_{r+1} = t_r + \tau .$3, the number of steps that can finish is

$t_{r+1} = t_r + \tau .$4

To determine whether failing to advance a request this round would make it irrecoverably late, the system uses the fastest possible per-step time

$t_{r+1} = t_r + \tau .$5

and computes a residual lower bound

$t_{r+1} = t_r + \tau .$6

A request survives if

$t_{r+1} = t_r + \tau .$7

The round scheduler then chooses at most one option per request, subject to total GPU capacity, to maximize the number of requests that survive to the next round (Lu et al., 2 Oct 2025).

The paper models this as a group knapsack problem and solves it with dynamic programming. With option set

$t_{r+1} = t_r + \tau .$8

and survival indicator

$t_{r+1} = t_r + \tau .$9

the DP transition is

$N$0

The paper states that the scheduler runs in $N$1 time and $N$2 space per round (Lu et al., 2 Oct 2025).

5. Implementation details and evaluation

The evaluation uses 8× NVIDIA H100-80GB with NVLink 4.0 and 4× NVIDIA A40-48GB, with FLUX.1-dev on H100 and Stable Diffusion 3 Medium (SD3) on A40 (Lu et al., 2 Oct 2025). The baselines are xDiT with fixed SP degrees SP=1, SP=2, SP=4, and SP=8. The main metric is SLO Attainment Ratio (SAR), defined as the fraction of requests finishing within deadline (Lu et al., 2 Oct 2025).

The default deadline configuration is resolution-specific: 1.5 s for 256×256, 2.0 s for 512×512, 3.0 s for 1024×1024, and 5.0 s for 2048×2048, scaled by an SLO Scale from 1.0× to 1.5× (Lu et al., 2 Oct 2025). Prompts are sampled from DiffusionDB; arrivals are Poisson at 12 req/min by default. Two workload mixes are studied: Uniform, with equal shares of 256, 512, 1024, and 2048, and Skewed, biased toward larger resolutions using

$N$3

with $N$4 and $N$5 (Lu et al., 2 Oct 2025).

The headline result is that TetriServe consistently outperforms all fixed-SP baselines. For FLUX on H100, it beats the best fixed strategy by 10% on average under the Uniform mix and by 15% on average under the Skewed mix (Lu et al., 2 Oct 2025). Under tighter deadlines, the gains are larger: 28% higher SAR for Uniform, SLO scale 1.1×, and 32% higher SAR for Skewed, SLO scale 1.2× (Lu et al., 2 Oct 2025). These are the paper’s principal quantitative claims.

The system also performs best across varying arrival rates from 6 to 18 req/min under the Uniform mix at 1.0× SLO, and it retains the highest SAR even under homogeneous workloads where all requests have the same resolution (Lu et al., 2 Oct 2025). This indicates that the benefits are not reducible to mixed-resolution adaptation alone; deadline-aware packing and elastic resource shaping remain useful even when resolution is fixed.

The paper’s most direct ablation studies step granularity. It reports that 1-step granularity is too expensive, 10-step granularity is too coarse, and 5-step granularity provides the best balance, especially as arrival rate increases (Lu et al., 2 Oct 2025). This result clarifies that step-level SP is not a claim that finer control is always better. Rather, scheduler responsiveness and reconfiguration overhead form a tradeoff, and moderate granularity is empirically preferred.

6. Relation to adjacent serving research and naming issues

The name TetriServe can be confused with several distinct systems and papers whose names contain “Tetris,” “Tetrys,” or similar variants. In the DiT serving context, however, the authoritative system name is TetriServe, introduced for heterogeneous image generation via step-level sequence parallelism (Lu et al., 2 Oct 2025). This should be distinguished from the datacenter call-packing system Tetris (Gandhi et al., 1 Aug 2025), the long-context LLM serving system Tetris based on Chunkwise Dynamic Sequence Parallelism (Li et al., 9 Nov 2025), the multimodal GPU DVFS system Tri-serve (Jia et al., 28 Jun 2026), and the transport-layer erasure coding scheme Tetrys (0904.4202).

The closest conceptual relative is the long-context LLM serving system called Tetris, which introduces Chunkwise Dynamic Sequence Parallelism (CDSP) for assigning SP sizes across intra-request token segments (Li et al., 9 Nov 2025). Both TetriServe and that system replace request-level static SP with finer-grained elastic SP, but they apply the idea to different computational structures: denoising steps for DiTs in TetriServe, and prompt chunks for long-context LLM prefill in Tetris (Li et al., 9 Nov 2025). This suggests a broader serving pattern in which static parallelism is increasingly treated as too coarse for heterogeneous inference.

A second relevant comparison point is TetriInfer, which disaggregates prefill and decode for mixed LLM workloads and uses chunking and two-level scheduling to reduce phase interference (Hu et al., 2024). While TetriInfer is not about DiTs, it is representative of a general trend toward phase-aware, workload-aware serving. TetriServe’s round-based control and elastic GPU allocation fit naturally into this trend, though its objective is SLO attainment rather than TTFT/JCT optimization (Hu et al., 2024).

Several limitations are explicit or implied in the TetriServe paper. The approach depends on accurate offline profiling and on the empirical fact that DiT step runtimes are highly stable. The formulation restricts GPU counts to powers of two, $N$6, which simplifies scheduling but may leave some cluster configurations unused. The per-round objective maximizes survival to the next round rather than solving the original offline objective globally. The evaluation is confined to DiT image generation, specifically FLUX.1-dev and SD3, and the paper does not establish generality to video diffusion or more complex multi-node GPU fabrics (Lu et al., 2 Oct 2025). These constraints do not invalidate the reported gains, but they define the scope of the system as presented.

In summary, TetriServe is a deadline-aware DiT serving system whose central technical idea is to make sequence parallelism dynamic at the granularity of denoising steps. By combining offline runtime profiling, round-based scheduling, deadline-feasible GPU-hour minimization, and dynamic-programming-based request packing, it addresses the inefficiency of fixed-SP serving under heterogeneous image-generation workloads. Its reported gains—up to 32% higher SAR than fixed-SP baselines—position it as a representative example of fine-grained elastic parallelism in contemporary inference-serving research (Lu et al., 2 Oct 2025).