---
title: 'TetriServe: Dynamic DiT Serving System'
url: https://www.emergentmind.com/topics/tetriserve
type: topic
---

# TetriServe: Dynamic DiT Serving System

Searching arXiv for the TetriServe paper and closely related serving work to ground the article.
TetriServe is a serving system for Diffusion Transformer (DiT) inference under heterogeneous online image-generation workloads with per-request completion deadlines. It is defined by its use of **step-level sequence parallelism**, which allows the degree of sequence parallelism assigned to a request to vary across denoising steps rather than remaining fixed for the request’s entire lifetime. The system targets the mismatch between fixed-degree sequence parallelism and mixed-resolution, mixed-deadline workloads: low parallelism is efficient for small images but too slow for large ones, whereas high parallelism accelerates large images but wastes GPU resources on small ones. TetriServe addresses this by combining round-based scheduling, deadline-aware GPU allocation, and request packing, and is reported to achieve up to **32% higher SLO Attainment Ratio (SAR)** than fixed-sequence-parallel baselines without degrading image quality [2510.01565].

## 1. Problem setting and motivation

TetriServe studies online serving of DiT-based text-to-image generation under explicit service-level objectives. In this setting, requests arrive online, share a finite GPU pool, and carry completion deadlines. The computational burden is substantial: the paper notes that on a single H100, generating a **2048×2048** image can take **up to a minute**, and **4096×4096** can exceed **ten minutes** [2510.01565]. This makes deadline compliance nontrivial even before considering contention among concurrent requests.

The workload is heterogeneous primarily because resolution strongly affects token count and total computation. The paper profiles representative FLUX.1-dev resolutions as follows: **256×256** corresponds to **1024 latent tokens** and **556.48 TFLOPs**; **512×512** to **4096 latent tokens** and **1388.24 TFLOPs**; **1024×1024** to **16384 latent tokens** and **5045.92 TFLOPs**; and **2048×2048** to **65536 latent tokens** and **24964.72 TFLOPs** [2510.01565]. These differences imply that a single static resource-allocation policy is structurally mismatched to the serving problem.

The paper’s critique of prior practice is directed at **fixed sequence parallelism (SP)**. Existing systems assign one SP degree to an entire request. This creates a tradeoff that is favorable only for a subset of workloads: **SP=1** or **SP=2** works well for **256×256** but performs poorly on **2048×2048**, while **SP=4** or **SP=8** helps large images but is inefficient for smaller ones [2510.01565]. Under a uniform workload and tight SLOs, the paper reports that **no fixed strategy exceeds SAR 0.6**, which motivates a more elastic scheduling model [2510.01565].

A central empirical enabler is the predictability of DiT inference. Over 100 runs, the coefficient of variation of step execution time is reported as **below 0.7%** for all tested resolutions and SP degrees, supporting the use of offline profiling as a runtime decision substrate [2510.01565]. This suggests that TetriServe’s control loop relies less on reactive correction than on accurate execution-time prediction.

## 2. Step-level sequence parallelism

TetriServe’s defining mechanism is **step-level sequence parallelism**. Instead of binding a request to a fixed SP degree for all denoising steps, the system allows the GPU allocation to change from one step, or small step block, to the next. A request can therefore run some steps with fewer GPUs and later steps with more GPUs, depending on urgency and expected benefit [2510.01565].

The rationale is twofold. First, the scaling efficiency of SP is resolution-dependent: small images often scale poorly, so high SP wastes GPU-hours, whereas large images benefit more from additional parallelism. Second, sublinear scaling means that extra GPUs may reduce latency while increasing total consumed GPU time. TetriServe therefore seeks the **minimum GPU allocation that still keeps a request feasible** with respect to its deadline [2510.01565]. This is not merely a latency-minimization policy; it is a deadline-constrained resource-minimization policy embedded inside a larger packing objective.

The system differs from request-level SP in a way that has direct scheduling consequences. Under fixed SP, a running request effectively monopolizes that many GPUs until completion. Under step-level SP, the scheduler can reclaim GPUs between rounds, reassign them, and later scale a request up again. The paper’s interpretation is that this flexibility avoids the rigid head-of-line blocking induced by fixed allocations and allows the system to redistribute capacity toward urgent large jobs while preserving efficiency for small ones [2510.01565].

The paper also states that TetriServe improves serving **without degrading image quality** because it changes only the execution schedule, not the model weights, prompt, denoising algorithm, or total number of denoising steps [2510.01565]. This suggests that step-level SP is treated as an output-preserving execution strategy, assuming numerical consistency across SP configurations.

## 3. System architecture and execution model

TetriServe comprises four main components: **Request Tracker**, **Scheduler**, **Execution Engine**, and **Latent Manager** [2510.01565]. The Request Tracker maintains request metadata including resolution, deadline, remaining steps, and execution state. The Scheduler is the core decision-making module, responsible for deadline-aware GPU allocation and round-level packing. The Execution Engine runs distributed GPU workers that execute assigned diffusion steps. The Latent Manager preserves intermediate latent states across rounds and supports handoff when a request changes GPU group [2510.01565].

The request lifecycle is round-based. A request arrives and is registered by the Request Tracker. At the next scheduling round, the Scheduler determines whether the request should run, how many GPUs it should receive, how many steps it should execute in that round, and which GPUs it should occupy. The Execution Engine then performs the assigned steps, after which intermediate latents are retained for the next round. This repeats until all denoising steps complete and the final image is returned [2510.01565].

This design implies that TetriServe treats a DiT request as a stateful, elastically parallel job whose execution can be paused and resumed across rounds. The Latent Manager is therefore not ancillary; it is the mechanism that makes step-level SP operationally viable. The paper states that latent tensors are compact enough that transfer overhead is negligible, and deadline accounting therefore ignores transfer time [2510.01565]. A plausible implication is that the scheduler is not required to internalize latent-migration cost in its core objective, though this assumption may be topology-dependent.

The system also includes practical mechanisms beyond the core scheduler. TetriServe attempts **placement preservation** to keep continuing requests on the same GPUs across rounds. It uses **work-conserving elastic scale-up** to assign idle GPUs to requests whose runtime would improve under a larger allocation. It supports **selective continuous batching** for identical small-resolution requests when batching does not hurt deadlines. The **VAE decoder** is run sequentially per request because it has large activation memory but small wall-clock cost relative to diffusion. The implementation also **pre-creates NCCL process groups** for many device combinations but warms only a compact commonly used subset to limit HBM usage [2510.01565].

## 4. Round-based scheduling and optimization

TetriServe makes the continuous-time online scheduling problem tractable by discretizing time into fixed-duration rounds of length $\tau$. If the current round starts at time $t_r$, the next round begins at
$$
t_{r+1} = t_r + \tau .
$$
Scheduling decisions are made only at round boundaries [2510.01565]. This yields natural points for effective preemption and resource reconfiguration.

The paper first formulates a more general offline problem. There are $N$ total GPUs and $R$ outstanding requests. Each request $req_i$ has dependent steps $\{s_{i1}, s_{i2}, \ldots, s_{iS_i}\}$, and each step may use $k \in \{1,2,4,\ldots,N\}$ GPUs. If the runtime of step $s_{ij}$ under $k$ GPUs is $T_{ij}(k)$, then the completion time of request $i$ is
$$
C_i = \sum_{j=1}^{S_i} \big[ Q_{ij} + T_{ij}(A_{ij}) \big],
$$
where $Q_{ij}$ is queueing delay before step $j$ and $A_{ij}$ is the GPUs allocated to step $j$ [2510.01565].

The objective is to maximize the number of requests completing by deadline:
$$
\text{Maximize} \quad \sum_{i=1}^{R} I_i, \quad \text{where } I_i =
\begin{cases}
1 & \text{if } C_i \leq D_i, \\
0 & \text{otherwise.}
\end{cases}
$$
This is subject to step dependency constraints and the GPU capacity constraint
$$
\sum_{i=1}^R \sum_{j=1}^{S_i} A_{ij}(t) \leq N, \quad \forall t .
$$
The paper states that even a simplified single-step version is NP-hard by reduction to a zero-one integer linear program [2510.01565].

TetriServe therefore decomposes the problem. For each request, it first computes the **minimal GPU allocation needed to remain deadline-feasible** using offline-profiled runtimes. The optimization is
$$
\min_{\{A_{ij}\}} \sum_{j=1}^{S_i} \big( A_{ij} \times T_{ij}(A_{ij}) \big)
\quad \text{s.t.} \quad
\sum_{j=1}^{S_i} \big( Q_{ij} + T_{ij}(A_{ij}) \big) \leq D_i .
$$
This minimizes GPU-hour consumption subject to deadline feasibility [2510.01565]. The paper’s phrasing makes clear that the scheduler is not seeking maximal acceleration per request, but the cheapest feasible execution mode.

After these candidate allocations are generated, TetriServe solves a round-level packing problem. For a request $i$ and allocation mode $m$, let $(s_i^m, A_i^m)$ denote the remaining steps and GPUs associated with that mode. In a round of duration $\tau$, the number of steps that can finish is
$$
q_i^m = \min\!\Big\{ s_i^m,\ \big\lfloor \tfrac{\tau}{T_i(A_i^m)} \big\rfloor \Big\}.
$$
To determine whether failing to advance a request this round would make it irrecoverably late, the system uses the fastest possible per-step time
$$
T_i^{\min}=\min_{k\in \{1, 2, 4, \ldots, N\}} T_i(k)
$$
and computes a residual lower bound
$$
\mathrm{LB}_i(o) = \Big(\sum_m \tilde{s}_i^m(o)\Big)\,T_i^{\min}.
$$
A request survives if
$$
t_{r+1}+\mathrm{LB}_i(o)\le D_i.
$$
The round scheduler then chooses at most one option per request, subject to total GPU capacity, to maximize the number of requests that survive to the next round [2510.01565].

The paper models this as a **group knapsack** problem and solves it with dynamic programming. With option set
$$
\mathcal{O}_i \leftarrow \{\textsf{none}\} \cup \{\,m \in \mathcal{M}_i \mid q_i^m>0 \wedge A_i^m \le N\,\},
$$
and survival indicator
$$
\mathrm{sv}_i(o) = \mathbb{I}[\,t_{r+1}+\mathrm{LB}_i(o) \le D_i\,],
$$
the DP transition is
$$
\mathrm{next}[c]\leftarrow\max\big\{\mathrm{next}[c],\ \mathrm{dp}[c-w_i(o)]+\mathrm{sv}_i(o)\big\}.
$$
The paper states that the scheduler runs in **$O(RN)$ time** and **$O(N)$ space** per round [2510.01565].

## 5. Implementation details and evaluation

The evaluation uses **8× NVIDIA H100-80GB** with NVLink 4.0 and **4× NVIDIA A40-48GB**, with **FLUX.1-dev** on H100 and **Stable Diffusion 3 Medium (SD3)** on A40 [2510.01565]. The baselines are xDiT with fixed SP degrees **SP=1**, **SP=2**, **SP=4**, and **SP=8**. The main metric is **SLO Attainment Ratio (SAR)**, defined as the fraction of requests finishing within deadline [2510.01565].

The default deadline configuration is resolution-specific: **1.5 s** for **256×256**, **2.0 s** for **512×512**, **3.0 s** for **1024×1024**, and **5.0 s** for **2048×2048**, scaled by an **SLO Scale** from **1.0× to 1.5×** [2510.01565]. Prompts are sampled from **DiffusionDB**; arrivals are **Poisson** at **12 req/min** by default. Two workload mixes are studied: **Uniform**, with equal shares of 256, 512, 1024, and 2048, and **Skewed**, biased toward larger resolutions using
$$
p_i \propto \exp(\alpha \cdot L_i/L_{\max})
$$
with $\alpha = 1.0$ and $L_i=(H_i \cdot W_i)/16^2$ [2510.01565].

The headline result is that TetriServe consistently outperforms all fixed-SP baselines. For FLUX on H100, it beats the **best fixed strategy by 10% on average** under the **Uniform** mix and by **15% on average** under the **Skewed** mix [2510.01565]. Under tighter deadlines, the gains are larger: **28% higher SAR** for **Uniform, SLO scale 1.1×**, and **32% higher SAR** for **Skewed, SLO scale 1.2×** [2510.01565]. These are the paper’s principal quantitative claims.

The system also performs best across varying arrival rates from **6 to 18 req/min** under the Uniform mix at **1.0× SLO**, and it retains the highest SAR even under homogeneous workloads where all requests have the same resolution [2510.01565]. This indicates that the benefits are not reducible to mixed-resolution adaptation alone; deadline-aware packing and elastic resource shaping remain useful even when resolution is fixed.

The paper’s most direct ablation studies step granularity. It reports that **1-step granularity** is too expensive, **10-step granularity** is too coarse, and **5-step granularity** provides the best balance, especially as arrival rate increases [2510.01565]. This result clarifies that step-level SP is not a claim that finer control is always better. Rather, scheduler responsiveness and reconfiguration overhead form a tradeoff, and moderate granularity is empirically preferred.

## 6. Relation to adjacent serving research and naming issues

The name **TetriServe** can be confused with several distinct systems and papers whose names contain “Tetris,” “Tetrys,” or similar variants. In the DiT serving context, however, the authoritative system name is **TetriServe**, introduced for heterogeneous image generation via step-level sequence parallelism [2510.01565]. This should be distinguished from the datacenter call-packing system **Tetris** [2508.00426], the long-context LLM serving system **Tetris** based on Chunkwise Dynamic Sequence Parallelism [2511.06247], the multimodal GPU DVFS system **Tri-serve** [2606.29629], and the transport-layer erasure coding scheme **Tetrys** [0904.4202].

The closest conceptual relative is the long-context LLM serving system called **Tetris**, which introduces **Chunkwise Dynamic Sequence Parallelism (CDSP)** for assigning SP sizes across intra-request token segments [2511.06247]. Both TetriServe and that system replace request-level static SP with finer-grained elastic SP, but they apply the idea to different computational structures: denoising steps for DiTs in TetriServe, and prompt chunks for long-context LLM prefill in Tetris [2511.06247]. This suggests a broader serving pattern in which static parallelism is increasingly treated as too coarse for heterogeneous inference.

A second relevant comparison point is **TetriInfer**, which disaggregates prefill and decode for mixed LLM workloads and uses chunking and two-level scheduling to reduce phase interference [2401.11181]. While TetriInfer is not about DiTs, it is representative of a general trend toward phase-aware, workload-aware serving. TetriServe’s round-based control and elastic GPU allocation fit naturally into this trend, though its objective is SLO attainment rather than TTFT/JCT optimization [2401.11181].

Several limitations are explicit or implied in the TetriServe paper. The approach depends on accurate offline profiling and on the empirical fact that DiT step runtimes are highly stable. The formulation restricts GPU counts to powers of two, $k \in \{1,2,4,\ldots,N\}$, which simplifies scheduling but may leave some cluster configurations unused. The per-round objective maximizes survival to the next round rather than solving the original offline objective globally. The evaluation is confined to DiT image generation, specifically FLUX.1-dev and SD3, and the paper does not establish generality to video diffusion or more complex multi-node GPU fabrics [2510.01565]. These constraints do not invalidate the reported gains, but they define the scope of the system as presented.

In summary, TetriServe is a deadline-aware DiT serving system whose central technical idea is to make sequence parallelism dynamic at the granularity of denoising steps. By combining offline runtime profiling, round-based scheduling, deadline-feasible GPU-hour minimization, and dynamic-programming-based request packing, it addresses the inefficiency of fixed-SP serving under heterogeneous image-generation workloads. Its reported gains—up to **32% higher SAR** than fixed-SP baselines—position it as a representative example of fine-grained elastic parallelism in contemporary inference-serving research [2510.01565].

Source: https://www.emergentmind.com/topics/tetriserve