HetRL: Heterogeneity-Aware RL Training

• HetRL is a distributed RL training system designed for RLHF and RLAIF workflows, efficiently scheduling tasks across GPUs with varying compute, memory, and network capabilities.
• It employs a multi-level search algorithm that combines nested successive halving and genetic algorithms to optimize task grouping and resource allocation.
• Empirical evaluations demonstrate that HetRL can achieve up to 9.17× throughput improvements by leveraging heterogeneity-aware scheduling and advanced load balancing techniques.

Searching arXiv for the specified HetRL paper and closely related RL scheduling systems. HetRL is a distributed system for efficient reinforcement learning training for LLMs in infrastructures with heterogeneous GPUs and networks. It targets RLHF and RLAIF post-training workflows in which multiple models and tasks must be coordinated across devices with different compute capability, memory capacity, HBM bandwidth, and interconnect properties. The system is organized around a heterogeneity-aware scheduling formulation, a multi-level search algorithm, profiling, load balancing, and an execution engine integrated with established LLM training and serving components. In extensive evaluation consuming 20,000 GPU-hours, it is reported to deliver up to $9.17\times$ the throughput of state-of-the-art systems, and $3.17\times$ on average, under various workloads and settings (He et al., 13 Dec 2025).

1. Scope, target workloads, and computational setting

HetRL is designed for RLHF and RLAIF pipelines using PPO and GRPO, in both synchronized and asynchronous variants. PPO is treated as the canonical case: the workflow involves four LLMs—actor, critic, reward, and reference—and six tasks: actor generation, reward inference, reference inference, critic inference, actor training, and critic training. GRPO removes the separate critic. The paper emphasizes that these tasks have heterogeneous compute, memory, and communication characteristics. Actor generation is described as memory-bound due to KV cache and HBM-bandwidth constrained, whereas actor and critic training are compute-bound due to activations, gradients, and optimizer states (He et al., 13 Dec 2025).

The workflow is represented as a DAG with a temporal split. Generation feeds three concurrent inference tasks—reference, reward, and critic—and those outputs then feed two concurrent training tasks, actor and critic. This structure makes scheduling nontrivial even in homogeneous clusters; in geo-distributed heterogeneous environments, the difficulty is compounded by divergent GPU capabilities and large variation in latency and bandwidth across machines and regions.

Task	Role in PPO workflow	Dominant characteristic
Actor generation	Actor model generates responses from input prompts	Memory-bound; HBM-bandwidth constrained
Reference inference	Computes reference token log-probabilities for KL penalty	Inference task
Reward inference	Produces per-sample scores via a frozen reward model	Inference task
Critic inference	Estimates per-sample values via the critic model	Inference task
Actor training	Updates actor parameters	Compute-bound; DP/TP/PP synchronizations
Critic training	Updates critic parameters	Compute-bound; DP/TP/PP synchronizations

A common misconception is to read HetRL as a new RL optimization algorithm. The system does not introduce a new substitute for PPO or GRPO; rather, it schedules PPO and GRPO workflows on heterogeneous hardware.

2. System architecture and execution stack

HetRL comprises four principal components: a profiler, a scheduler, a load balancer, and a distributed execution engine. The profiler collects computation power in TFLOPs, memory capacity in GBs, HBM bandwidth in GB/s, intra-machine bandwidth in GB/s, and inter-node latency in ms and bandwidth in Gbps across all available heterogeneous GPUs and links. The scheduler consumes the RL workflow description and the profiled hardware topology and searches for near-optimal execution plans. The load balancer applies both data-level and layer-level balancing strategies. The execution engine is built on verl with extended support for fine-grained resource assignment and load balancing; Megatron-LM is used as the training backend, vLLM is used for generation and inference, and communication uses NCCL (He et al., 13 Dec 2025).

Parallelization is expressed through standard data parallelism, pipeline parallelism, and tensor parallelism. HetRL treats the choice of DP, PP, and TP as part of the scheduling problem rather than as a fixed runtime configuration. This is central because different tasks stress different subsystems: generation is sensitive to HBM and decode batching, training to compute and synchronization, and pipeline efficiency to stage balance and communication topology.

The load balancer implements three concrete policies stated in the paper. During actor rollout, it adjusts per-GPU local batch sizes in DP groups. For inference tasks with known sequence lengths, it assigns longer sequences to more powerful GPUs. Across PP stages, it shifts layer boundaries to equalize stage times. These mechanisms are intended to reduce stragglers and improve overlap.

3. Formal scheduling problem and cost model

The paper defines a heterogeneity-aware RL training scheduling problem over a workflow computational graph $\mathbf{G}$ and a device topology graph $\mathbf{G_D}$. The optimization chooses a partitioning strategy $\rho$ and an assignment strategy $\sigma$ to minimize execution time subject to device-count, coverage, and memory constraints (He et al., 13 Dec 2025):

$$\min_{\rho, \sigma} \quad C(\rho, \sigma; \mathbf{G}, \mathbf{G_D})$$

subject to

$$|\{(i,j,k) : l^t_{i,j,k} \in V_L\}| \leq |V_D|, \forall t \in \{1,\ldots,T\}$$

$\bigcup_{d \in V_D} \sigma^{-1}(d) = V_L$

and

$$\max_{l \in \sigma^{-1}(d)} \text{working\_mem}(l) + \sum_{l \in \sigma^{-1}(d)} \text{model\_memory}(l) \leq \text{gpu\_mem}(d), \forall d \in V_D.$$

Here, $3.17\times$0 applies DP, PP, and TP and merges operators into tasklets, producing a tasklet graph $3.17\times$1, while $3.17\times$2 maps tasklets to devices. The problem is stated to be NP-hard; the appendix sketch reduces subcases to balanced graph partitioning, knapsack, or minimum makespan.

At the DAG level, HetRL composes task costs into an end-to-end iteration estimate. For synchronous PPO, the reported formulation is

$3.17\times$3

with

$3.17\times$4

The same framework is extended to asynchronous PPO, synchronous GRPO, and asynchronous GRPO through analogous formulas that add $3.17\times$5 or remove critic-related terms. Per-task costs are decomposed into compute time, HBM-bound decoding time, TP communication, PP communication, DP communication, and PP bubble overhead. The cost model explicitly uses device TFLOPs and HBM bandwidth and link-level latency $3.17\times$6 and bandwidth $3.17\times$7, so the planner is topology-aware rather than merely device-count-aware.

4. Multi-level search and scheduling algorithm

HetRL addresses the joint optimization by decomposing it into five levels. Level 1 partitions RL tasks into disjoint groups that execute on a common GPU set and whose associated models are co-located. Level 2 divides GPUs into disjoint groups and assigns task groups to GPU groups by size. Level 3 generates candidate assignments of task groups to specific devices. Level 4 enumerates feasible DP, PP, and TP strategies per task, given the assigned devices. Level 5 assigns tasklets to specific GPUs to complete execution plans (He et al., 13 Dec 2025).

The search procedure combines nested successive halving with a genetic algorithm. Successive halving is used at Levels 1 and 2 by treating task groupings and GPU groupings as arms in a non-stochastic best-arm identification problem under a fixed search budget $3.17\times$8. Candidate plans are evaluated by the cost model, more budget is allocated to promising arms, and worse halves are discarded at each round. For Levels 3 through 5, the system uses a genetic algorithm with mutation and refinement. The paper highlights two swap operators: GPU swaps across task groups at Level 3 and GPU swaps across tasklet groups at Level 5. Suboptimal candidates after swaps are removed from the population; stronger offspring replace weaker candidates.

This decomposition is significant because the design space jointly spans model placements, parallelization strategies, and fine-grained tasklet-to-device mappings. The reported convergence study shows that the nested SHA plus GA strategy finds lower-cost plans more consistently than verl’s scheduler and a “HetRL (simple)” ablation that disables SHA and restricts swaps to within a model.

5. Handling heterogeneity, communication, and imbalance

HetRL models heterogeneity along GPU compute performance, HBM bandwidth, memory capacity, and network latency, bandwidth, and locality. The profiler populates device and topology matrices, and the scheduler’s communication models explicitly account for TP all-reduce, PP point-to-point stage handoffs, and DP gradient all-reduce. Ring selection is performed so as to minimize the worst-edge time, reflecting the fact that in heterogeneous or geo-distributed environments a single poor link can dominate the effective communication schedule (He et al., 13 Dec 2025).

The system also models overlap and partial parallelism at the DAG level through $3.17\times$9, pipeline bubbles through $\mathbf{G}$0, and synchronous or asynchronous RL coordination through $\mathbf{G}$1. In asynchronous PPO and GRPO, generation can overlap with training, but the paper notes two caveats: asynchronous modes may incur accuracy concerns due to staleness and a higher memory footprint because two actor copies are maintained. The evaluation focuses on throughput rather than those accuracy questions.

The load-balancing subsystem is integral rather than auxiliary. Data-level balancing adapts local batch sizes during actor rollout to heterogeneous devices. When inference sequence lengths are known, longer sequences are mapped to faster GPUs. Layer-level balancing redistributes layers across PP stages so that stage times are better equalized under heterogeneous TFLOPs, HBM bandwidth, and communication conditions. This suggests that HetRL treats imbalance as a first-class systems problem rather than as a byproduct of suboptimal placement.

6. Empirical performance, deployment conditions, and limitations

The evaluation uses 64 GPUs in total: 24 $\mathbf{G}$2 A100, 24 $\mathbf{G}$3 L40S, and 16 $\mathbf{G}$4 L4. Four network scenarios are described: Single-Region, Multi-Region-Hybrid, Multi-Country, and Multi-Continent. The models are Qwen-4B, Qwen-8B, and Qwen-14B; the algorithms are PPO and GRPO, in synchronous and asynchronous modes; the dataset is GSM8k; the maximum prompt and response length is 1024; the global batch size is 1024; and 8 responses are generated per prompt. Baselines are verl (HybridFlow) and StreamRL, with the same backends—vLLM for inference and Megatron-LM for training (He et al., 13 Dec 2025).

The headline results are strongly scenario-dependent. In Single-Region, HetRL-Sync beats verl by $\mathbf{G}$5, and HetRL-Async beats StreamRL and verl by $\mathbf{G}$6. In Multi-Region-Hybrid, HetRL-Sync beats verl by $\mathbf{G}$7, and HetRL-Async beats StreamRL by $\mathbf{G}$8 and verl by $\mathbf{G}$9. In Multi-Country, HetRL-Sync beats verl by $\mathbf{G_D}$0, and HetRL-Async beats StreamRL by $\mathbf{G_D}$1 and verl by $\mathbf{G_D}$2. In Multi-Continent, HetRL-Sync beats verl by $\mathbf{G_D}$3, and HetRL-Async beats StreamRL by $\mathbf{G_D}$4 and verl by $\mathbf{G_D}$5. The paper also reports that load balancing improves synchronous PPO and GRPO throughput by up to $\mathbf{G_D}$6 in Single-Region and up to $\mathbf{G_D}$7 in Cross-Region scenarios, and that across GPU combinations HetRL beats verl by $\mathbf{G_D}$8.

The practical guidance is correspondingly specific. Accurate hardware and network profiling is necessary because cross-region latency and bandwidth materially affect planning. The search budget $\mathbf{G_D}$9 should be large enough for nested successive halving to concentrate resources on promising groupings. Data-level and layer-level balancing are recommended, and the paper explicitly advises tuning micro-batch counts $\rho$0 and decoding batch sizes $\rho$1 to balance PP bubbles and HBM-bound generation. Asynchronous modes usually yield higher throughput, but are presented as a throughput-oriented choice rather than a universally preferable one.

The limitations are also explicit. Hardware coverage is restricted to A100, L40S, and L4, with evaluation using AWS OFI NCCL and AWS EFA. The study focuses on throughput and does not analyze precision or convergence effects caused by heterogeneous GPUs or asynchronous staleness beyond known algorithmic trade-offs. Cost-efficiency is not compared. Current load-balancing strategies improve throughput but leave room for more advanced methods. Detailed optimization of $\rho$2 and $\rho$3 is deferred, as are fault tolerance and elasticity across geo-distributed clusters.

A possible source of ambiguity is nomenclature. This suggests that “HetRL” is not globally unique as a string in the literature: the term also appears in work on heterogeneous relational learning, such as the relational graph transformer paper (Dwivedi et al., 16 May 2025). In the context of (He et al., 13 Dec 2025), however, HetRL denotes the heterogeneous-environment RL training system described above.