Zorse: Heterogeneous GPU Training for LLMs

• Zorse is a system that optimizes dense transformer LLM training on heterogeneous GPU clusters by integrating pipeline and data parallelism.
• It addresses challenges in load balancing, memory fitting, and communication efficiency through adaptive scheduling and configuration planning.
• Evaluations on clusters up to 128 GPUs demonstrate up to 3× higher throughput and balanced HFU performance across varied GPU types.

A zorse is a hybrid offspring of a zebra and a horse. In large-scale machine learning systems research, Zorse denotes the system introduced in "Zorse: Optimizing LLM Training Efficiency on Heterogeneous GPU Clusters" (Guo et al., 14 Jul 2025), which targets efficient training of dense transformer LLMs on clusters composed of GPUs of different generations, memory capacities, and network interconnects. The system is designed around the claim that efficient heterogeneous training requires an integrated use of pipeline parallelism and data parallelism that is both communication- and memory-efficient, together with an adaptable configuration that supports asymmetric pipeline stages and heterogeneous GPUs within the same data-parallel group. Zorse combines these capabilities with a planner that automatically configures training strategies for a given workload, and is evaluated on heterogeneous clusters up to 128 GPUs and models up to 65B parameters (Guo et al., 14 Jul 2025).

1. Heterogeneous-cluster training problem

Zorse is motivated by the observation that heterogeneous GPU clusters are increasingly the norm because of rapid GPU release cycles, limited cloud availability, incremental procurement, and cost. The target environments combine GPUs of different generations such as T4, A10G, V100, A100, and H100, memory capacities from 16 GB to 94 GB, and interconnects spanning PCIe, NVLink, NVSwitch, Ethernet, InfiniBand, and cross-region WAN. Real deployments may span multiple datacenters or regions when a single region cannot supply enough GPUs (Guo et al., 14 Jul 2025).

The paper identifies three core challenges. Load balancing arises because compute throughput varies substantially across devices; the cited FP16 TFlops values are T4 65, A10G 125, V100 125, A100 312, and H100-NVL 989. Equal work assignment therefore causes faster GPUs to idle behind slower GPUs. Memory fitting is difficult because memory scales weakly relative to compute; mid- and high-end GPUs often have only $1\text{–}1.5\times$ more memory than much slower devices, so memory-heavy strategies can bottleneck on faster GPUs. Communication efficiency is constrained by strongly nonuniform bandwidth: examples include intra-node unidirectional bandwidths of approximately 3.0 GB/s on A10G/PCIe, 6.1 GB/s on T4/PCIe, 23.9 GB/s on V100/NVLink, and 222 GB/s on A100/NVSwitch, while inter-node cross-region links can be as low as 2.69–12.26 GB/s (Guo et al., 14 Jul 2025).

Within this setting, Zorse argues that several standard parallelization strategies are insufficient in isolation. Tensor parallelism is reported as inefficient on such clusters; empirically, HFU with tensor parallelism versus data parallelism on 8-GPU AWS VMs shows data parallelism superior except on 8$\times$A100 NVSwitch. Data parallelism alone is throttled by slow paths because global all-reduces and all-gathers are limited by the slowest links, especially WAN links. Pipeline parallelism alone does not solve memory heterogeneity and can incur large bubbles with many stages. The proposed remedy is a carefully engineered combination of PP and DP that balances compute while reducing communication and memory overheads (Guo et al., 14 Jul 2025).

2. Parallelization architecture and scheduling

Zorse partitions the cluster into $k$ GPU groups, with $k$ chosen by the planner. Pipeline parallelism runs across groups, while data parallelism runs within each group. This arrangement confines expensive DP collectives to faster intra-group links, such as intra-node NVLink or NVSwitch and intra-region networks, while using PP to cross slower inter-group links such as cross-node or cross-region connections. Within each group, Zorse employs ZeRO-2-style DP sharding: optimizer states and gradients are sharded, and parameters are gathered in a layer-by-layer fashion. Each PP stage is further decomposed into multiple ministages, yielding interleaved pipelining for finer-grained balancing and more memory-efficient materialization (Guo et al., 14 Jul 2025).

The system supports flexible asymmetric PP. Pipeline stages may contain different numbers of GPUs and different GPU types, and Zorse supports many-to-many reshuffle of microbatches across stage boundaries. Within a stage, DP can include mixed GPU types, with non-uniform microbatch assignment proportional to per-GPU speed and memory. This non-uniformity is central to the load-balancing strategy: stage-level balance is achieved by assigning layers proportional to the aggregate compute rates of groups, while intra-stage balance is achieved by assigning different microbatch counts to different GPUs (Guo et al., 14 Jul 2025).

Scheduling follows a GPipe-style policy at the level of ministages rather than 1F1B. For each ministage, all forward microbatches are executed sequentially, followed by all backward microbatches in reverse order. The paper states that this increases opportunities to offload or minimize resident tensors and to overlap communication with compute deterministically. Gradient accumulation occurs over the microbatches in the GPipe schedule, and optimizer steps are interleaved at the completion of each ministage’s backward pass so that gradients can be freed early while gradient synchronization overlaps with compute of subsequent ministages (Guo et al., 14 Jul 2025).

Many-to-many reshuffle between stages uses the “largest remaining time first” heuristic. When the $i$th microbatch finishes in the current stage, it is sent to the GPU in the next stage with the $i$th highest remaining workload. The stated purpose is to reduce stage makespan and internal stragglers. Communication libraries are also chosen asymmetrically: NCCL is used for intra-group DP collectives, while PP reshuffles use a hybrid policy in which NCCL is employed when safe and Gloo non-blocking P2P is used for cyclic cross-stage edges to avoid deadlocks (Guo et al., 14 Jul 2025).

3. Performance model, memory model, and planner

Zorse incorporates a two-phase planner that profiles the cluster and workload, partitions GPUs into groups for DP versus PP, and then configures microbatches, ministages, and layer boundaries using calibrated cost and memory models. The planner runs in under 3 minutes up to 128 GPUs (Guo et al., 14 Jul 2025).

Profiling is performed at two levels. For compute, Zorse profiles per-layer runtime on each GPU for small batch sizes and fits a linear model to predict FLOPsRate per GPU and per-layer latency across batch sizes. For networking, it profiles intra-node and inter-node bandwidths, restricted to unique VM-GPU configurations to reduce time. The cluster is represented as a fully connected graph $G=(V,E)$ in which vertices are GPUs and edge weights are measured bandwidths $w(u,v)$; cross-datacenter links appear as low-bandwidth edges (Guo et al., 14 Jul 2025).

Partitioning uses an approximate min-$k$ cut. The planner applies SPLIT greedy, described as a $2-2/k$ approximation, to create $\times$0 disjoint groups that maximize intra-group bandwidth and minimize inter-group bandwidth. The time complexity is $\times$1, and the partitioning is done once with $\times$2 to obtain all partitions. Once groups are chosen, Zorse computes the aggregate compute rate $\times$3 of each group as the sum of per-GPU layer rates, assigns layers to groups proportional to $\times$4, splits each group’s layers into evenly sized ministages, and orders groups by decreasing intra-group bandwidth in order to minimize pipeline startup stalls (Guo et al., 14 Jul 2025).

The main compute and communication model is stated explicitly. For stage $\times$5, with layer set $\times$6, the compute time per microbatch is

$\times$7

For backward with activation recomputation, $\times$8 is added similarly over recomputed layers. For ZeRO-2-style per-layer AllGather and ReduceScatter within DP group $\times$9 of size $k$0, ring-collective models are used:

$k$1

$k$2

For inter-stage PP communication,

$k$3

If multiple links are used concurrently, contention is modeled by reducing $k$4 proportionally to concurrent flows (Guo et al., 14 Jul 2025).

Pipeline efficiency is modeled as

$k$5

where $k$6 is the total number of ministages and $k$7 is the number of microbatches. The overlapped per-microbatch time of stage $k$8 is

$k$9

and the step makespan is approximated as

$k$0

Throughput for global batch size $k$1 is

$k$2

The system reports throughput in TFlops and HFU, with $k$3 (Guo et al., 14 Jul 2025).

The optimization problem is stated as minimization of $k$4 subject to memory constraints $k$5, feasible PP ordering across groups, DP group sizes and intra-group topology consistent with NCCL collectives, and network contention constraints on inter-group links. Phase 1 applies SPLIT to obtain $k$6; Phase 2 uses heuristic layer sizing proportional to aggregate compute rates, round-robin ministage ordering, enumeration over $k$7 and $k$8, and evaluation with latency and memory models to select the minimum feasible $k$9 (Guo et al., 14 Jul 2025).

4. Memory management, overlap, and efficiency mechanisms

Zorse’s memory strategy combines activation checkpointing, activation offload, parameter offload, optimizer-state sharding, and interleaved optimizer updates. Activations are checkpointed at transformer layer boundaries, with rematerialization during backward. Because GPipe without offload would retain approximately $i$0 layer-boundary activations, Zorse offloads boundary activations to CPU memory as soon as they are no longer needed in forward and prefetches them before backward. Separate CUDA streams are used so that host-device transfers do not block GPU compute (Guo et al., 14 Jul 2025).

Within DP groups, optimizer states and gradients are sharded in ZeRO-2 style, and parameters are gathered layer by layer. Due to interleaving, only the parameters of the current and next ministage are kept resident on the GPU, while the rest are offloaded to host memory. Interleaved optimizer updates further reduce peak GPU memory because gradients are freed immediately after each ministage, and ReduceScatter or Average operations are overlapped with future ministage compute (Guo et al., 14 Jul 2025).

The memory constraints are expressed per GPU $i$1 in group $i$2 as

$i$3

For parameters, if stage $i$4 has $i$5 layers and uniform ministage sizes $i$6,

$i$7

For gradients,

$i$8

For optimizer states under ZeRO-2 sharding across $i$9 GPUs,

$i$0

For activations, baseline GPipe without offload would require

$i$1

whereas Zorse reduces resident activations to

$i$2

with small $i$3, typically $i$4, independent of $i$5 and $i$6 (Guo et al., 14 Jul 2025).

The overlap condition is also made explicit: layer-by-layer prefetch can be hidden if $i$7, and similarly for activation offload and prefetch. The paper states that separate CUDA streams are used to meet this condition in practice. Empirically, the combined effect of these mechanisms is up to 40% reduction in memory utilization with $i$8 throughput drop as the number of ministages per GPU increases in homogeneous 16$i$9A100 and 16$G=(V,E)$0A10G experiments. For 16$G=(V,E)$1A10G on Llama 7B, increasing ministages per GPU from 1 to 8 reduces memory to 59% with 20% TFlops drop; for 16$G=(V,E)$2A100 on Llama 13B, memory drops to 62% with approximately 24% TFlops drop (Guo et al., 14 Jul 2025).

5. Experimental results and quantitative behavior

The evaluation uses Llama-family models from 7B to 65B, FP16 mixed precision, global batch size 1M tokens, and activation checkpointing. Three heterogeneous clusters are studied. Cluster A is an Azure 1-region system with 20 GPUs: 4$G=(V,E)$3H100-NVL (2$G=(V,E)$42) and 16$G=(V,E)$5A100-80GB (2$G=(V,E)$68), with NVSwitch intra-node and 50 Gbps inter-node. Cluster B is an AWS 1-region system with 64 GPUs: 8$G=(V,E)$7A100-40GB, 16$G=(V,E)$8A10G, 16$G=(V,E)$9V100, and 24$w(u,v)$0T4, with mixed NVLink and PCIe intra-node. Cluster C is an AWS 2-region system with 128 GPUs: region 1 has 16$w(u,v)$1A10G and 48$w(u,v)$2T4; region 2 has 16$w(u,v)$3V100 and 48$w(u,v)$4T4; communication includes cross-region WAN. Baselines are TorchTitan-Het, HexiScale, and Cephalo (Guo et al., 14 Jul 2025).

On Cluster A, Zorse achieves 4370.6 TFlops at 52.5% HFU for 7B, compared with TorchTitan-Het at 4223.8 TFlops and 50.7% HFU, HexiScale at 3193.5 and 38.3%, and Cephalo at 1714.5 and 20.6%. For 13B, Zorse reaches 4917.9 TFlops and 59.0% HFU, versus 3837.5 and 46.1%, 3270.3 and 39.3%, and 1656.3 and 19.9%. For 33B, Zorse reaches 5281.6 and 63.4%, while TorchTitan-Het reaches 944.5 and 11.3% and is reported OOM at larger configurations; HexiScale reaches 3064.2 and 36.8%, and Cephalo 1943.9 and 23.3%. For 65B, Zorse reaches 5239.1 and 62.9%; TorchTitan-Het is OOM, HexiScale reaches 2048.6 and 24.6%, and Cephalo 1937.6 and 23.3% (Guo et al., 14 Jul 2025).

On Cluster B, Zorse reports 3412.9 TFlops and 43.5% HFU for 7B, compared with 2033.5 and 25.9% for TorchTitan-Het, 1194.9 and 15.2% for HexiScale, and 2274.5 and 29.0% for Cephalo. For 13B, Zorse reports 2965.6 and 37.8%, compared with 1956.1 and 24.9%, 1152.7 and 14.7%, and 1992.2 and 25.4%. For 33B, Zorse reports 2658.3 and 33.9%; TorchTitan-Het is OOM, HexiScale reports 657.2 and 8.4%, and Cephalo 1373.3 and 17.5% (Guo et al., 14 Jul 2025).

On Cluster C, Zorse reports 3936.9 TFlops and 39.2% HFU for 7B, compared with 2441.7 and 24.3% for TorchTitan-Het, 2624.6 and 26.2% for HexiScale, and 1213.4 and 12.1% for Cephalo. For 13B, it reports 3358.0 and 33.5%, compared with 2061.6 and 20.6%, 1952.3 and 19.5%, and 1223.0 and 12.2%. For 33B, Zorse reports 1548.6 and 15.4%, while TorchTitan-Het and HexiScale are OOM and Cephalo reports 775.4 and 7.7% (Guo et al., 14 Jul 2025).

The paper summarizes these outcomes as up to $w(u,v)$5 higher training throughput than state-of-the-art baselines while maintaining HFU comparable to homogeneous subsets. It also reports that adding heterogeneous GPUs increases PFlops consistently while maintaining or improving HFU; an example is Cluster B, where HFU rises up to 43.5% as A100s are added. Per-GPU-type HFU on heterogeneous clusters is close to running only homogeneous subsets: for Llama 7B on Cluster B, T4 is 40% versus 30%, A10G 39% versus 33%, V100 56% versus 51%, and A100 59% versus 56% (Guo et al., 14 Jul 2025).

An ablation on Cluster A with Llama 65B attributes the gains to specific mechanisms. Baseline PP+ZeRO-2 in TorchTitan is OOM. Adding activation offload yields 3746 TFlops at 83% memory utilization. Adding interleaved pipelining and optimizer updates yields 4087 TFlops at 58% memory utilization. Adding heterogeneous PP by grouping by node into two stages yields 5239 TFlops at 73% memory utilization, with balanced compute and low PP bubble (Guo et al., 14 Jul 2025).

6. Deployment considerations, limitations, and reproducibility

The paper provides concrete deployment guidance. For Cluster A, the planner groups GPUs per node into two heterogeneous PP stages: Stage 1 is 4$w(u,v)$6H100 and Stage 2 is 16$w(u,v)$7A100, with DP within each stage, 4–6 ministages per GPU for large models, layer assignment approximately proportional to the 989:312 per-GPU rates, and H100 first in the group order to minimize startup stalls. For Cluster B, the planner creates four groups by GPU type, runs DP within each group and PP across groups, uses 4–6 ministages per GPU for 13B and 33B, assigns per-GPU microbatch sizes proportional to per-GPU FLOPsRate, and places PP over slow inter-group links. For Cluster C, DP groups are kept intra-region, PP is used across regions, V100s are split into smaller DP groups than A10Gs because of lower memory, T4s are split into two DP groups per region to avoid cross-region collectives, and more ministages per GPU are used to fit memory (Guo et al., 14 Jul 2025).

Several failure modes are explicitly identified. NCCL P2P can deadlock in asymmetric many-to-many reshuffles, so non-blocking Gloo is used for cyclic dependencies. Offload and prefetch streams must remain independent from the main compute stream; otherwise kernels can stall. Extremely small batches or ultra-slow WAN links can limit overlap, in which case the planner shifts toward fewer microbatches and larger intra-group DP buckets (Guo et al., 14 Jul 2025).

The main limitations are also explicit. Current Zorse does not support tensor parallelism. The paper states that TP can be beneficial for extremely large layers or tiny batches on homogeneous, ultra-fast interconnects, but is generally detrimental on heterogeneous, bandwidth-limited clusters. Configuration is static during a run; dynamic reconfiguration under transient heterogeneity or failures is future work, although the planner is fast enough to enable periodic refresh. The design assumes CPU memory is available for offloading and that PCIe or host-memory bandwidth is sufficient for overlap; extremely low host bandwidth or NUMA pathologies may reduce benefits. The target model class is dense transformer LLMs, and MoE-specific or operator-specific optimizations are out of scope (Guo et al., 14 Jul 2025).

Reproducibility is described at the systems level. Zorse is implemented on PyTorch FSDP, using the ZeRO-2/ZeRO-3 toolkit together with custom interleaved PP, per-layer sharding and resharding, and offload and prefetch streams; NCCL is used for DP collectives, and hybrid NCCL+Gloo for PP P2P. The paper states that complete cluster compositions, model sizes, precision settings, and baseline configurations are provided, and that planner profiling and search procedures, including algorithmic complexity, are described. Code or artifact availability is not specified. A plausible implication is that reproduction depends on reconstructing comparable cloud configurations and following the profiling and search procedures documented in the paper rather than reusing a released implementation (Guo et al., 14 Jul 2025).