Gyges: Cross-Instance Parallelism Transformation

• The paper introduces a method where live LLM instances dynamically shift between tensor-parallel configurations to optimize throughput for short requests and capacity for long contexts.
• Gyges employs innovations like page-friendly KV-cache layouts, dedicated weight padding, and a transformation-aware scheduler to minimize transformation overhead.
• Empirical results demonstrate up to 6.57× throughput improvement and significant reductions in transformation cost, proving the practical gains of dynamic instance reconfiguration.

Cross-Instance Parallelism Transformation, as developed in Gyges, is a runtime systems technique for LLM serving in which the parallelism configuration of already-running instances is changed dynamically to match incoming request dynamics, especially context length variance. Rather than fixing each deployment at startup as TP1, TP2, TP4, and so forth, Gyges begins from a throughput-oriented configuration and converts live instances across tensor-parallel regimes on demand, merging multiple TP1 instances into a higher-TP instance for long-context requests and later decomposing them when that pressure subsides. The defining claim is that parallelism is treated as a mutable runtime resource rather than a static provisioning choice, and that making this practical requires coordinated changes to KV-cache layout, model-weight layout, and request scheduling policy (Chen et al., 24 Sep 2025).

1. Deployment tension and the motivation for transformation

Gyges addresses a production constraint in which requests arrive with highly variable context lengths, while the best tensor-parallel setting depends strongly on that length distribution. The paper states a fundamental trade-off: low parallelism such as TP1 gives the best throughput for common short requests, but cannot fit or efficiently serve very long contexts because GPU memory is dominated by model weights; higher parallelism such as TP4 frees weight memory and expands usable KV-cache capacity, but introduces partitioning and communication overhead that reduces throughput (Chen et al., 24 Sep 2025).

A concrete example is given for Qwen2.5-32B on 4 H20 GPUs:

Configuration	Max supported sequence	Throughput
TP1	3.75K	single-instance 448 tps; total 1792 tps
TP2	41.25K	single-instance 670 tps; total 1340 tps
TP4	120.5K	single-instance 767 tps; total 767 tps

On this example, scaling from $4\times\text{TP1}$ to TP4 can cause over 57% throughput loss (Chen et al., 24 Sep 2025). The motivation for Cross-Instance Parallelism Transformation is therefore not merely to support long contexts, but to retain TP1-like throughput for the dominant short-request regime while still allowing temporary transitions to higher TP when long-context demand appears.

The paper’s operating assumption is that real workloads are dynamic and that long requests are rare but important. This implies that permanent static partitioning wastes resources in at least one regime: either high-throughput short requests are penalized by overprovisioned TP, or long requests are rejected or delayed by underprovisioned TP.

2. Transformation semantics and the Gyges model

The core transformation is explicitly cross-instance. In static parallelism, each instance is fixed at startup; in Gyges, the system reconfigures running instances at runtime. The representative transformation is

$4 \times (\text{TP1}) \;\to\; \text{TP4}.$

The paper emphasizes that this is not just a scheduling decision; it is a physical memory and computation-layout transformation across multiple already-running GPU instances (Chen et al., 24 Sep 2025). GPUs that were previously serving separate instances are re-partitioned into one larger cooperative instance. The immediate systems consequence is that the reduction in per-GPU weight footprint under higher TP can be converted into additional KV-cache capacity for long-context serving.

This model distinguishes Gyges from ordinary autoscaling. Traditional scaling would provision new workers and incur minute-scale delays, whereas Gyges responds using already-warm workers. It also differs from static heterogeneous deployment, because the same hardware pool is reused across parallelism regimes instead of being permanently carved into separate TP classes.

A recurrent misconception is to view Gyges as a policy for routing requests to preexisting TP1, TP2, and TP4 pools. The paper rules out that interpretation by making transformation itself a first-class operation. Live instances are merged when a request exceeds what TP1 can serve, and decomposed when long-request pressure disappears (Chen et al., 24 Sep 2025). This suggests that the central abstraction is not simply request placement, but runtime conversion between memory/computation organizations.

3. System organization and component responsibilities

Gyges comprises three major components, each targeting a distinct bottleneck in runtime TP transformation (Chen et al., 24 Sep 2025).

Page-friendly, header-centric layout for KV-cache transformations restructures KV-cache storage so that page-based allocation and later migration during TP conversion are efficient. The page-friendly aspect matches page allocation behavior and eliminates repeated shifting when new pages are added. The header-centric aspect changes the migration unit and reduces trimming cost during transformation.

Dedicated weight padding for model-weight transformations modifies the weight layout at load time so that intended TP configurations are page-aligned in advance. Because GPU memory management operates at a minimum granularity of 2 MB pages, many raw model partitions would otherwise fragment and force copying. Padding is introduced at known partition boundaries so that scale-up can release or reuse pages in place.

Transformation-aware scheduler reasons jointly about request characteristics, instance TP state, and transformation cost. The scheduler avoids the pathological behavior of naive load-balancing policies that repeatedly trigger unnecessary reconfigurations or oscillate between TP modes.

The three components are coupled rather than independent. Efficient KV migration without scheduler restraint would still allow thrashing; scheduler awareness without page-aligned weight and KV layouts would still incur large transition cost. Gyges therefore treats data layout, model layout, and control policy as a single transformation substrate.

4. KV-cache and model-weight transformation mechanisms

The KV cache is treated as the principal systems obstacle in long-context serving. Gyges first replaces a raw layout with a page-friendly layout, then introduces a header-centric hierarchy:

$$[\text{Block}, \text{Header}, \text{K/V}, \text{Token}].$$

This is contrasted with the raw layout $$[\text{K/V}, \text{Block}, \text{Token}, \text{Header}]$$ and the page-friendly layout $$[\text{Block}, \text{K/V}, \text{Token}, \text{Header}]$$ (Chen et al., 24 Sep 2025). The paper claims that the page-friendly layout eliminates memory shifting when adding pages,

$O(\#KV\ cache\ pages) \to 0,$

while the header-centric layout reduces trimming complexity during transformation,

$O(\#Local\ tokens) \to O(1).$

For scale-up, Gyges gives an explicit KV-cache splitting rule. If $H$ is the number of attention heads and the target tensor parallel size is $TP$, then worker $W_i$ retains heads in

$4 \times (\text{TP1}) \;\to\; \text{TP4}.$0

and sends the others to corresponding workers (Chen et al., 24 Sep 2025). During migration, Gyges uses phased all-to-all exchange: workers exchange KV data, exchange metadata about memory regions that will become free after that stage, and then reuse those freed regions in subsequent stages. The paper also notes that cuMemUnmap, cuMemMap, and cuMemSetAccess can overlap with GPU kernels, while all-to-all communication is launched on an independent communication stream.

Model-weight transformation is motivated by the observation that MLP weights constitute about 88% of total model weight (Chen et al., 24 Sep 2025). The raw FFN computation is written as

$4 \times (\text{TP1}) \;\to\; \text{TP4}.$1

where $4 \times (\text{TP1}) \;\to\; \text{TP4}.$2 is the input tensor, $4 \times (\text{TP1}) \;\to\; \text{TP4}.$3 is up_proj, $4 \times (\text{TP1}) \;\to\; \text{TP4}.$4 is down_proj, and $4 \times (\text{TP1}) \;\to\; \text{TP4}.$5 is the activation. Gyges pads these weights so that

• $4 \times (\text{TP1}) \;\to\; \text{TP4}.$6
• $4 \times (\text{TP1}) \;\to\; \text{TP4}.$7

and then shows

$4 \times (\text{TP1}) \;\to\; \text{TP4}.$8

The claim is therefore twofold: padding preserves correctness, and padding makes scale-up essentially an in-place transformation because workers can release or reuse aligned pages instead of copying misaligned fragments (Chen et al., 24 Sep 2025).

5. Scheduler logic and transformation control

The scheduler operationalizes the transformation model through two routines, schedule_request(request) and schedule_parallelism(instance_id) (Chen et al., 24 Sep 2025). The first scans instances, checks whether long-context-aware scheduling is needed, prefers instances that can already serve the request, and triggers execute_scale_up() only when necessary. The second monitors instances with TP $4 \times (\text{TP1}) \;\to\; \text{TP4}.$9 and triggers execute_scale_down() only when no long request is present and load is below a threshold.

The policy is designed around three competing objectives: throughput, latency or SLO compliance, and transformation cost. For each incoming request, the scheduler estimates whether existing instances can handle it given input length and current load; if an appropriate instance already exists, the request is routed there; otherwise a transformation candidate is selected. For scale-down, the system waits until long requests are gone, checks cache and load conditions, and only then decomposes the larger TP instance (Chen et al., 24 Sep 2025).

A notable empirical modeling choice is that the scheduler reasons primarily about prompt length rather than generated output length, because output contributes only 10.3% of total length in the measured workload (Chen et al., 24 Sep 2025). This narrows the dominant predictor of transformation need to the input side.

The scheduler is explicitly presented as an antidote to naive RR and LLF policies. Those policies may route a new long request to a TP1 instance even when a TP4 instance already exists, causing unnecessary new scale-ups, repeated oscillation, and throughput loss. Gyges instead is transformation-aware rather than merely load-aware. This suggests that, in the system’s formulation, scheduling quality depends on recognizing the future cost of changing the parallelism state, not just the present occupancy of an instance.

6. Empirical behavior, gains, and operating limits

The evaluation uses H20 nodes with 8 NVIDIA H20 GPUs, 96 GB each, NVLink, 2 TB DDR5, and a 128-core Intel Xeon Platinum 8469C, as well as A100 nodes with 8 NVIDIA A100 GPUs, 40 GB each, NVLink, 2 TB DDR5, and a 128-core Intel Xeon Platinum 8369B. Models include Llama2-7B, Llama3-8B, Qwen2.5-32B, and Qwen3-32B, and end-to-end results use production traces with dynamic request arrivals and context-length variation (Chen et al., 24 Sep 2025).

Across those settings, the main reported benefit is 1.75×–6.57× throughput improvement over state-of-the-art alternatives (Chen et al., 24 Sep 2025). Additional reported gains include TTFT reduced by up to 53%, TPOT reduced by up to 74%, and transformation cost reduced by up to 97% relative to Seesaw in the all-layer transformation case. Overlap further reduces TTFT by an additional 26.7%, and transformation overhead remains below 1% even when many layers are transformed.

On mechanism-level measurements, the paper reports basic KV transformation overhead of extra 3.15–4 ms; Gyges- without overlap reduces this by up to 61%; full Gyges with overlap reduces KV transformation time by 86%, reduces memory overhead by 91.6%, and keeps additional memory usage below 70 MB (Chen et al., 24 Sep 2025). For weight transformation, a Partial Swap baseline requires 611–696 ms per layer transformation; Gyges- with weight padding reduces this cost by 18.9%–42.2%; with overlap, cost decreases by up to 67.6% relative to the basic solution; extra compute overhead from padding is negligible at less than 0.1%; padding memory overhead ranges from 0% to 14%.

The paper also characterizes Gyges’ effective regime. It benefits most when workloads are dominated by short requests but contain occasional long-context outliers, when arrivals are bursty or irregular, and when static TP4 provisioning would waste throughput most of the time (Chen et al., 24 Sep 2025). Conversely, its scope is narrower when workloads are uniformly short or uniformly long, because the value of dynamic transformation declines as the workload ceases to alternate between throughput-favored and capacity-favored modes. The design is also primarily TP-oriented: PP, SP, and EP are discussed, but not the main path. Another assumption is that supported TP configurations are fixed in advance, enabling pre-padding of weights. The paper further states that 91.7% of multi-GPU deployments in their production data use TP (Chen et al., 24 Sep 2025).

7. Relation to adjacent transformation frameworks

Gyges belongs to a broader class of systems that realign computation boundaries to match hardware constraints, but it does so with a distinctive object of transformation: already-running LLM serving instances. In Matryoshka, the corresponding concept is Elastic Parallelism Transformation, defined over dimensions on which operations are commutative and associative, so that work can be split, reordered, regrouped, and scheduled to better match GPU execution. Matryoshka uses three primitives—Permutation, Deconstruction, and Combination—to make irregular quantum chemistry workloads appear “regular enough” for GPUs, and the paper explicitly notes that this is closely related in spirit to cross-instance parallelism transformation ideas such as Gyges (Wang et al., 2024).

The overlap is conceptual rather than domain-specific. Matryoshka realigns work across ERIs, basis-function combinations, recurrence paths, and workload tiles so that the hardware sees a dense, balanced, and more uniform stream of tasks; Gyges realigns work across tensor-parallel instances so that the serving system can alternate between throughput-optimized and memory-capacity-optimized layouts (Wang et al., 2024). A plausible implication is that both systems treat the original decomposition of work as negotiable, provided semantic equivalence is preserved.

A second related formulation appears in joint scheduling of multi-band radar sensing and DNN inference for cross-stage parallelism. There, each sensed band releases an inference branch as soon as sensing for that band completes, rather than waiting for all bands to finish. The key release-time abstraction is

$$[\text{Block}, \text{Header}, \text{K/V}, \text{Token}].$$0

with the decoupled baseline instead using a global barrier

$$[\text{Block}, \text{Header}, \text{K/V}, \text{Token}].$$1

The paper describes this as a cross-instance style idea in which each band/branch pair behaves like an instance whose compute can be released independently as soon as its own upstream sensing instance is done (Du et al., 20 Apr 2026). This suggests a common structural pattern across otherwise different domains: eliminating a global synchronization barrier and replacing it with per-instance release conditions.

Within that broader landscape, Gyges is specific in its systems claim. It does not merely observe that dynamic reordering can help; it makes runtime parallelism conversion itself the unit of optimization in LLM inference. Its distinctive contribution is therefore the combination of a cross-instance transformation semantics with concrete mechanisms—page-friendly, header-centric KV layout, dedicated weight padding, and transformation-aware scheduling—that reduce the cost of changing that semantics under live serving load (Chen et al., 24 Sep 2025).