---
title: 'Hetis: Dynamic LLM Serving for Heterogeneous GPUs'
url: https://www.emergentmind.com/topics/hetis
type: topic
---

# Hetis: Dynamic LLM Serving for Heterogeneous GPUs

Searching arXiv for the Hetis paper and closely related LLM serving systems to ground the article with current references.
Hetis is an LLM serving system for heterogeneous GPU clusters that combines fine-grained and dynamic parallelism to address the mismatch between memory capacity and computational power across mixed devices and the module-level performance imbalance within Transformer inference [2509.08309]. It is designed for production LLM serving under latency and throughput constraints in clusters containing both high-end and low-end GPUs, such as A100s together with 3090s or P100s, and departs from coarse-grained and static strategies by selectively parallelizing dense computation while dynamically distributing decode-phase Attention at head granularity [2509.08309]. The system is implemented by extending vLLM and is evaluated against Splitwise and Hexgen, with reported gains of up to \(2.25\times\) in throughput and up to \(1.49\times\) reduction in latency [2509.08309].

## 1. Problem formulation and motivation

Hetis targets the setting of production LLM serving in heterogeneous GPU clusters, where inference stresses compute, memory capacity, and communication simultaneously [2509.08309]. The paper emphasizes that this is particularly consequential for LLM inference because prefill and decode have different resource profiles, and because long-context decoding makes KV-cache capacity a first-order systems constraint; it gives the example that decoding a single sequence of length 10k on LLaMA2-13B requires over 8 GB of memory for KV cache [2509.08309].

The motivating systems observation is that GPU heterogeneity is not well described by a single performance ratio. For OPT-2.7B, using a batch of 3 requests in prefill and 25 in decode, Table 1 reports: A100 80GB with prefill \(0.06\) s and decode \(0.0097\) s; 3090 24GB with prefill \(0.147\) s and decode \(0.0143\) s; and P100 12GB with prefill \(1.47\) s and decode \(0.077\) s [2509.08309]. Relative to A100, the P100 is \(24.5\times\) slower in prefill and \(7.93\times\) slower in decode, but its memory is only \(6.67\times\) smaller [2509.08309]. This establishes the first bottleneck identified by Hetis: memory capacity and compute capability do not scale proportionally across heterogeneous GPUs.

The second bottleneck is module-level performance imbalance. The paper states that for Llama-70B decoding, the performance gap between A100 and P100 is much more severe for MLP than for Attention, and that processing MLP on a P100 can be up to \(40.4\times\) slower on average than on an A100 during decoding, while the gap in Attention is much smaller [2509.08309]. This implies that a uniform partitioning policy across all Transformer modules is structurally mismatched to heterogeneous hardware.

A plausible implication is that heterogeneous serving must be module-aware rather than merely device-aware. Hetis formalizes that position by distinguishing dense, compute-intensive operations from decode-phase Attention and optimizing them separately [2509.08309].

## 2. Limits of prior heterogeneous serving strategies

Hetis situates itself against two prior styles of heterogeneity-aware LLM serving: phase splitting or prefill-decode disaggregation, exemplified by Splitwise, and parameter splitting or asymmetric model partitioning, exemplified by Hexgen [2509.08309]. Its critique is that both remain too coarse-grained and too static for heterogeneous clusters.

In the phase-splitting approach, high-end GPUs handle prefill and low-end GPUs handle decode [2509.08309]. Hetis argues that this causes memory inefficiency because model weights are duplicated across separate prefill and decode workers. The paper gives the example that with an FP16 7B model on an A100 + 3090, only 10 GB remains on the decode worker, the 3090, for KV cache [2509.08309]. It also argues that Splitwise underuses high-end GPUs during decode and low-end GPUs during prefill [2509.08309].

In asymmetric parameter splitting, tensor, pipeline, or data parallelism is skewed according to heterogeneous compute capabilities [2509.08309]. Hetis argues that this can strand memory. For FP16 OPT-2.7B on A100 + 3090, compute-balanced splitting would place about 40% of parameters and caches on the 3090 and 60% on the A100; then the 3090 runs out of cache space first, leaving only 18.6 GB usable there, while the A100 still has 44.1 GB unused [2509.08309]. The throughput bottleneck becomes the weaker GPU’s memory. The paper further argues that static asymmetric model parallelism treats all modules alike, despite the fact that MLP and Attention respond differently to weak GPUs, and that adding more weak GPUs can backfire because communication grows, especially in prefill where intermediate tensors across layers are large [2509.08309].

The paper’s general conclusion is that coarse-grained and static parallelism is the wrong abstraction for heterogeneous LLM serving [2509.08309]. This suggests that an effective design must decide not only how much work each device receives, but also which kinds of work each device should be allowed to execute.

## 3. Core design: fine-grained and dynamic parallelism

Hetis’s core design philosophy is to control heterogeneous resources at a finer granularity than whole phases or whole layers and to do so dynamically rather than statically [2509.08309]. The system organizes LLM work into two broad classes.

The first class is compute-intensive dense work, including dense modules such as MLP and parameterized linear operations; the paper also includes prefill Attention in the optimized primary-worker execution because it is latency-sensitive and communication-heavy if spread too widely [2509.08309]. The second class is decode-phase Attention, which Hetis treats separately because it is parameter-free once \(q\), \(K\), and \(V\) are given, can be partitioned naturally by attention head, has lower and more uniform compute intensity across heterogeneous devices, and has a memory footprint dominated by KV cache, making low-end GPUs useful as cache holders [2509.08309].

From this distinction, Hetis derives three named ideas: **I1: optimized parallel configuration for dense operations**, **I2: dynamic head-wise Attention parallelization**, and **I3: global request scheduling via explicit latency quantification** [2509.08309]. The practical consequence is that Hetis does not force all GPUs to participate in all model computations, does not use one static partition for all modules, does not fix prefill and decode to disjoint sets of GPUs, and dynamically dispatches decode Attention across devices based on current load, memory usage, and network costs [2509.08309].

This module-aware decomposition is the defining conceptual novelty of the system. The paper explicitly positions Hetis not merely as heterogeneity-aware serving, but as module-aware heterogeneity-aware serving [2509.08309].

## 4. Architecture, worker roles, and serving workflow

Hetis consists of four named components: **Parallelizer**, **Profiler**, **Dispatcher**, and **Hauler** [2509.08309]. These define the runtime organization of the system.

Hetis assigns GPUs into two roles. **Primary workers** run all operations in both prefill and decode, including dense modules, and serve as the anchor for requests; **Attention workers** do not participate in dense computation and are dynamically pooled to execute decode-phase Attention and store KV cache [2509.08309]. In the paper’s Llama-70B evaluation, A100 and 3090 GPUs are used as Primary workers, while P100s are dedicated Attention workers [2509.08309].

The **Parallelizer** runs at initialization. It decides which GPUs become Primary versus Attention workers, searches for the best dense-compute parallelization over the heterogeneous cluster, and determines data parallelism, pipeline parallelism, and tensor parallelism among Primary workers [2509.08309]. The **Profiler** performs lightweight offline profiling to estimate Attention compute cost on each GPU type and communication cost between Primary and Attention workers; these measurements produce linear models used online [2509.08309]. The **Dispatcher** operates at runtime for each new request arrival, deciding how many attention heads of each request to place on each GPU, balancing compute load, memory occupancy, and communication overhead, and re-dispatching existing requests when long contexts create imbalance [2509.08309]. The **Hauler** implements KV-cache management and migration at head granularity, supporting fine-grained cache placement, partial cache migration when head assignments change, and low-priority migration to avoid interfering with live inference [2509.08309].

The serving workflow proceeds as follows: during initialization, the Parallelizer selects a dense-compute configuration and labels GPUs as Primary or Attention workers; the Profiler builds per-device compute and transfer models; requests are served by Primary workers; during decode Attention, some heads may remain local on the Primary worker while others are offloaded to Attention workers; the Dispatcher solves an optimization problem for newly arriving requests to choose head placement; and the Hauler manages head-wise KV-cache placement and migration as needed [2509.08309].

A plausible implication is that Hetis decomposes serving into a static control plane for dense execution and a dynamic control plane for decode Attention. The paper does not use that phrasing, but its architecture strongly reflects that separation.

## 5. Dense execution strategy and dynamic Attention distribution

### Dense-module parallelization

Hetis selectively parallelizes dense computations across a chosen subset of GPUs only, while decode-phase Attention can be distributed across all GPUs only when beneficial [2509.08309]. The rationale is that including weak GPUs in dense execution can cost more than it helps: dense operations have high arithmetic intensity and create major slowdowns on low-end GPUs, leading to synchronization delay, pipeline bubbles, communication overhead, and idle time on high-end GPUs if those weak devices are forced into TP or PP for MLP and projections [2509.08309].

The Parallelizer seeks an optimal dense execution configuration
\[
\sigma^* = \arg\min_{\sigma}\ C(\sigma, M, \mathcal{R}),
\]
where \(\sigma\) maps model parameters to devices and chooses DP/TP/PP structure, \(M\) is the model, \(\mathcal{R}\) is the inference request distribution, and \(C(\cdot)\) is total dense-module cost including communication and computation, following Hexgen-style modeling [2509.08309]. Because this optimization is NP-hard, Hetis uses hierarchical exploration: it groups devices into serving instances, evenly distributing GPU types across instances; filters infeasible configurations lacking enough KV cache space for decoding; optimizes pipeline partitioning by treating GPUs of the same type as one unified PP stage and minimizing the maximum stage computation cost \(C_p(\cdot)\), initially ignoring communication; heuristically excludes low-end GPUs from dense execution if
\[
\frac{C_p(\sigma - \kappa, M, \mathcal{R})}{C_p(\sigma, M, \mathcal{R})} \le 1 + \Delta,
\]
with \(\Delta = 0.05\) by default; and finally searches TP/PP combinations within the remaining stages using a Hexgen-like cost model [2509.08309]. The \(5\%\) threshold means that if removing a low-end GPU increases dense latency by at most \(5\%\), Hetis leaves it out of dense execution and reserves it for Attention work [2509.08309].

### Head-wise decode Attention

Hetis distributes decode Attention dynamically to lower-end GPUs at head granularity [2509.08309]. The paper gives three reasons why Attention is suitable for this treatment: it is parameter-free in decode, it can be split by head independently without requiring global softmax across devices, and performance disparity is smaller for Attention than for MLP [2509.08309]. It explicitly rejects partitioning by batch dimension as too coarse and prone to full-request migration and fragmentation under varying context lengths, and rejects partitioning by sequence-length dimension because it replicates \(q\) to all workers holding partial cache and increases communication substantially [2509.08309].

The paper’s communication microbenchmark reports that with one Attention worker and \(20\%\) load offloaded, head-wise splitting cuts communication overhead by about \(2.68\times\); with four workers, evenly distributed head-wise splitting yields up to \(3.55\times\) latency reduction over the alternative communication pattern [2509.08309]. Hetis expresses distributed head-wise Attention as
\[
\mbox{Attention}_j = \mathsf{Concat}\big(result_{1,j}, \cdots, result_{n,j}\big),
\]
\[
result_{i,j} = \mathsf{softmax}\Big(q_{h^i_j(t)}\cdot K^T_{h^i_j(t)} /\sqrt{d}\Big)\cdot V_{h^i_j(t)}.
\]
Here \(j\) is a request, \(i\) is a GPU, \(h_j^i(t)\) is the number of query heads of request \(j\) placed on GPU \(i\) at time \(t\), and \(d\) is head dimension [2509.08309]. Each GPU computes Attention for its assigned heads, and the outputs are concatenated [2509.08309].

This head-granularity design is the system’s finest unit of dynamic scheduling. It also enables weak GPUs to contribute chiefly through KV-cache storage and attention computation rather than through dense arithmetic, which is exactly the imbalance that motivated Hetis.

## 6. Online dispatching, re-dispatching, and implementation

### Online load dispatching

The Dispatcher uses a linear compute model for device \(i\),
\[
\tau_i(t) = a_i \cdot h_i(t) + b_i \cdot g_i(t) + c_i,
\]
where \(\tau_i(t)\) is Attention computation time on GPU \(i\) at time \(t\), \(h_i(t)\) is the total number of query heads assigned to GPU \(i\), \(g_i(t)\) is the total cache volume across those heads on GPU \(i\), and \(a_i,b_i,c_i\) are profiled coefficients [2509.08309]. Communication overhead between a Primary worker and Attention worker \(i\) is modeled as
\[
\rho_i(t) = \gamma_i \cdot d_i(t) + \beta_i,
\]
where \(\gamma_i,\beta_i\) are profiled transfer coefficients and \(d_i(t)\) is transferred data volume [2509.08309]. For decode Attention,
\[
d_i(t)=\left(2+\frac{2}{r}\right)\cdot h_i(t),
\]
where \(r\) is the ratio between the number of query heads and grouped key/value heads; the paper notes that this supports both GQA models such as Llama-70B and MHA models such as OPT-30B and Llama-13B [2509.08309].

For each pipeline stage \(k\) with \(N_k\) workers, the Dispatcher chooses
\[
\{x_1^j(t), x_2^j(t), \dots, x_{N_k}^j(t)\},
\]
where \(x_i^j(t)\) is the number of query heads of request \(j\) placed on GPU \(i\) at time \(t\) [2509.08309]. It enforces a head-integrity constraint,
\[
\sum_{i=1}^{N_k} x_i^j(t) \in \{0, H\}, \quad \forall j \in \mathbb{S}(t), \forall t,
\]
with \(x_i^j(t)/r \in \mathbb{N}\), and a memory-capacity constraint,
\[
\sum_{j\in\mathbb{S}(t)} x_i^j(t)\cdot l_j(t) \leq \frac{r\cdot M_i(t)}{2}, \quad \forall i, \forall t,
\]
where \(l_j(t)\) is context length of request \(j\) and \(M_i(t)\) is available cache capacity on GPU \(i\) [2509.08309].

For newly arrived requests \(J(t)\), Hetis minimizes the maximum per-device Attention time,
\[
\min \max_i f_i(\vec{x_i}(t)),
\]
subject to memory and head-allocation constraints [2509.08309]. For a Primary worker, the per-device cost is
\[
f_i\big(\vec{x}_i(t)\big) = a_i \cdot \left( h_i(t) + \sum_{j=1}^{J(t)} x_i^j(t)\right) + b_i \cdot \left(g_i(t)+ \frac{2}{r}\sum_{j=1}^{J(t)} l_j(t)x_i^j(t)\right) + c_i,
\]
while for an Attention worker,
\[
f_i\big(\vec{x}_i(t)\big)= \left(a_i + \left(2+\frac{2}{r}\right)\gamma_i\right) \cdot \left( h_i(t) + \sum_{j=1}^{J(t)} x_i^j(t)\right) + b_i \cdot \left(g_i(t)+ \frac{2}{r}\sum_{j=1}^{J(t)} l_j(t)x_i^j(t)\right) + c_i + \beta_i.
\]
After solving, cumulative load is updated by
\[
h_i(t+1)=h_i(t)+\sum_{j=1}^{J(t)}x_i^j(t),
\]
\[
g_i(t+1)=g_i(t)+\frac{2}{r}\sum_{j=1}^{J(t)}x_i^j(t)l_j(t).
\]
The paper notes that because the objective and constraints are linear, the problem can be reformulated as a linear program, giving a polynomial-time solution [2509.08309].

### Re-dispatching and KV-cache migration

Hetis supports re-dispatching because long-running requests can make earlier head placements suboptimal as context lengths grow unpredictably [2509.08309]. It computes an ideal Attention time \(f^*\) by solving an optimization over all ongoing requests; if the gap between current Attention time and ideal Attention time exceeds threshold \(\Theta\), it re-dispatches one request [2509.08309]. The default threshold is \(\Theta = 50\%\); the system picks the device with the longest Attention time, selects the request contributing most to that device’s load, and re-solves the dispatching problem for that request [2509.08309].

For memory balancing, if one GPU exhausts cache space, Hetis does not use traditional LIFO or LRU eviction directly, because not all requests consume memory on all devices [2509.08309]. Instead, before evicting, it checks whether cluster-wide free memory elsewhere can absorb the request through re-dispatching:
\[
\sum_i g_i(t) < \sum_i \frac{r\cdot M_i}{2}.
\]
The meaning, as stated in the paper, is that if the cluster still has aggregate free cache capacity, re-dispatch should be attempted instead of dropping or swapping requests [2509.08309]. Because old and new head assignments may overlap, Hetis migrates only the differing fraction of KV cache [2509.08309].

### Implementation details

Hetis is implemented by extending **vLLM** [2509.08309]. It reuses **PagedAttention** for partial Attention execution on assigned heads and implements new CUDA kernels for head-granularity KV-cache fetch and store [2509.08309]. KV cache is managed in fixed-size blocks, as in vLLM, but blocks are additionally split by head and indexed by sequence id, position in sequence, and head id; because head-wise indexing increases CPU-side bookkeeping, block indexing is accelerated with multi-core CPU parallelization during decode [2509.08309]. Communication uses **NCCL**, with additional communication groups enabling peer-to-peer transmission between Primary and Attention workers [2509.08309]. To avoid cache migration interfering with inference collectives, migrations run on low-priority CUDA streams [2509.08309].

The profiling overhead is modest: the system profiles eight values of \(h_i(t)\) and eight values of \(g_i(t)\) per device type, only needs to run the Attention module once per configuration because of layer identity in Transformers, and each run takes no more than 100 ms [2509.08309]. The primary-worker search takes 4 seconds on the authors’ cluster and 15 seconds in a simulated larger cluster with 5 GPU types and 32 GPUs each [2509.08309]. Supported models in evaluation are **Llama-13B**, **OPT-30B**, and **Llama-70B**, and the system supports both **MHA** and **GQA** [2509.08309]. The evaluation cluster contains one host with 4× A100-80GB, two hosts with 2× 3090 each, one host with 4× P100, a 100 Gbps LAN interconnect, and PCIe within hosts [2509.08309].

## 7. Evaluation, significance, and limitations

Hetis is evaluated on **ShareGPT** for chatbot, **HumanEval** for code completion, and **LongBench** for long-document summarization, against **Splitwise** and **Hexgen** [2509.08309]. Metrics include throughput or sustainable request rate, end-to-end latency, P95 TTFT, P95 TPOT, module-level Attention and MLP latency, KV-cache capacity, modeling accuracy, and overhead and sensitivity [2509.08309].

The main quantitative results are summarized below.

| Metric | Reported result | Comparison |
|---|---:|---|
| Throughput | up to \(2.25\times\) higher | vs Splitwise |
| Throughput | up to \(1.33\times\) higher | vs Hexgen |
| Latency reduction | up to \(1.49\times\) | compared to existing systems |
| P95 TTFT | up to \(1.22\times\) better | vs Hexgen on Llama-70B |
| P95 TTFT | up to \(1.47\times\) better | vs Splitwise on Llama-70B |
| TPOT | up to \(1.39\times\) better | vs baselines |
| KV-cache capacity | up to \(1.87\times\) improvement | over baselines |

The paper attributes these gains to four mechanisms. First, better dense execution reduces decode-phase MLP latency by up to \(1.29\times\), primarily by excluding low-end GPUs from dense critical paths when they would mostly add synchronization and communication overhead [2509.08309]. Second, dynamic Attention offloading reduces decode Attention latency by up to \(1.49\times\), reflecting efficient head-wise balancing of compute and communication [2509.08309]. Third, improved memory utilization uses weak GPUs as KV-cache and Attention resources instead of expending them on dense compute or extra model replicas, increasing cluster-wide admission capacity and throughput [2509.08309]. Fourth, re-dispatching under long contexts improves mean output latency by \(1.06\times\) and P95 output latency by \(1.14\times\) compared with a heterogeneous-aware LIFO baseline [2509.08309].

The modeling results are also reported as accurate: compute-model prediction accuracy reaches up to \(93.8\%\), transfer-overhead accuracy is \(92.4\%\)–\(96.1\%\), head-wise cache management increases storage overhead by \(13\%\), and reduces cache fetching time by \(26\%\) [2509.08309]. Sensitivity experiments indicate that the default \(\Theta = 50\%\) lies in an effective region, since too small a value causes excess migration while too large a value leaves load imbalance, and that even with profiling parameter errors up to \(\pm 20\%\), latency worsens by only up to \(6.9\%\) [2509.08309].

The paper states that Hetis performs especially well when the cluster includes significantly weaker GPUs, requests are dynamic and bursty, contexts are long or highly variable, decode dominates and KV-cache capacity matters, and uniform static partitioning would create module-level bottlenecks [2509.08309]. It helps less under very light load, where offloading is often unnecessary and Hetis deliberately keeps work local, and in settings where low-end GPUs contribute little memory or network overhead dominates [2509.08309]. The paper does not present a direct homogeneous-cluster comparison [2509.08309].

Several limitations are explicit or evident. Hetis relies on offline profiling coefficients \(a_i,b_i,c_i,\gamma_i,\beta_i\), so stable device and network behavior remains an assumption [2509.08309]. Head-wise cache management requires new kernels, new metadata structures, CPU-side indexing acceleration, and more complex migration logic, and incurs a \(13\%\) storage overhead [2509.08309]. The benefits depend on the heterogeneity structure and communication environment, and the paper does not describe dynamic re-selection of Primary versus Attention workers as workload mix evolves over long timescales [2509.08309]. It also does not discuss failure handling or multi-tenant interference [2509.08309].

In comparative terms, the paper’s novelty consists of selective parallelization of dense modules, dynamic head-wise Attention parallelism, online LP-based dispatching that explicitly models computation, communication, and cache usage, and head-wise KV-cache management and migration [2509.08309]. This suggests that the broader significance of Hetis lies less in any single optimization than in a systems principle: heterogeneous LLM serving becomes substantially more effective when Transformer execution is partitioned by module semantics rather than by a single cluster-wide rule [2509.08309].

Source: https://www.emergentmind.com/topics/hetis