---
title: 'AReaL-Hex: Heterogeneous Async RL'
url: https://www.emergentmind.com/topics/areal-hex
type: topic
---

# AReaL-Hex: Heterogeneous Async RL

Searching arXiv for the AReaL-Hex paper and closely related asynchronous RL / heterogeneous LLM systems to support the article.
Searching for AReaL and related asynchronous RL systems.
AReaL-Hex is a heterogeneity-aware asynchronous reinforcement learning training system for large language models that schedules rollout generation and policy model training over heterogeneous GPUs while enforcing data staleness bounds. It is built atop a fully asynchronous RL architecture and treats RL training as three coupled stages—rollout generation, reward computation, and policy/value updates—whose compute intensities, memory footprints, and communication patterns differ substantially from one another and from conventional homogeneous LLM pretraining workflows. Its central design claim is that HBM-I/O-bound generation and compute-bound optimization should be disaggregated onto different hardware pools and then jointly balanced by a scheduler that accounts for resource budgets, parallelization strategies, topology, and trajectory freshness [2511.00796].

## 1. System scope and problem setting

AReaL-Hex is positioned as a heterogeneity-aware extension of recent fully asynchronous RL systems, especially AReaL, for the specific setting of RL training for LLMs on mixed GPU clusters [2511.00796]. The motivating observation is that RL training is not a single homogeneous workload. Instead, it decomposes into rollout generation, reward computation, and policy/value model training, and these stages differ sharply in arithmetic intensity, memory-bandwidth pressure, and collective-communication demands. The paper argues that this difference creates an opportunity for heterogeneous deployment that is not available, or at least not as salient, in conventional large-scale pretraining [2511.00796].

The target optimization criteria are training throughput and cost-efficiency under bounded staleness. The system is therefore not merely a hardware placement heuristic. It is a joint execution framework that chooses how to split a mixed cluster between rollout inference and training, what per-stage parallelization strategy to use, and how to allocate GPUs and interconnect structure so that producer-consumer imbalance does not lead either to idleness or to stale rollout trajectories [2511.00796].

AReaL-Hex is evaluated on mathematical reasoning RL workloads with model scales of 1.5B, 7B, and 14B. Relative to homogeneous deployments of state-of-the-art asynchronous RL systems, it is reported to deliver up to \(1.50\times\) higher training throughput at the same total budget, and up to \(1.46\times\) lower training cost at the same throughput [2511.00796].

## 2. Architectural decomposition and asynchronous execution

The system adopts a fully asynchronous RL pipeline with three logical components: rollout workers, reward computation, and training workers [2511.00796]. Rollout workers continuously sample responses from the current policy; reward computation scores those trajectories; training workers consume them to update the policy/value model. Parameter flow proceeds in the reverse direction during synchronization: once a new model version is ready, training and rollout are temporarily paused and weights are broadcast from training workers to rollout workers via NCCL collective communication [2511.00796].

AReaL-Hex explicitly partitions the cluster into disjoint hardware pools for inference and training,
\[
\mathcal{D}_T \cup \mathcal{D}_I = \mathcal{D}, \qquad \mathcal{D}_T \cap \mathcal{D}_I = \emptyset,
\]
where \(\mathcal{D}_I\) is used for rollout generation and \(\mathcal{D}_T\) for policy/value updates [2511.00796]. This disaggregation is the enabling condition for heterogeneous specialization. The paper characterizes rollout generation as primarily HBM-I/O-bound, whereas policy/value optimization is compute-bound and communication-intensive. Reward computation is treated as part of the rollout-side cost and is profiled as relatively stable in the scheduling model [2511.00796].

The system’s staleness argument is architectural rather than algorithmically novel. It inherits a bounded-staleness asynchronous RL regime from AReaL, and then enforces balanced rollout/training rates so stale trajectories do not accumulate uncontrollably [2511.00796]. The scheduling logic is therefore inseparable from training stability: if training is too slow, rollout workers eventually stall; if rollout is too slow, trainers idle. AReaL-Hex treats these as dual failure modes of poor hardware allocation.

## 3. Scheduling formulation and the two-phase optimizer

The scheduler is given a heterogeneous GPU set \(\mathcal{D}\), a staleness parameter \(\eta\), and an expected rollout-length distribution \(P\), and seeks a training plan \(\sigma\), a rollout plan \(\tau\), and a partition \((\mathcal{D}_T,\mathcal{D}_I)\) minimizing the slower of the two asynchronous sides [2511.00796]. The optimization is formulated as
\[
\sigma^{*}, \tau^*, \mathcal{D}^*_T, \mathcal{D}^*_I
= \arg \min_{\sigma,\tau,\mathcal{D}_T,\mathcal{D}_I} \max\{C_T, C_I\},
\]
with
\[
C_T = C_{Train}(\sigma,\mathcal{D}_T,\delta(\eta)),
\]
and
\[
C_I
=
C_{Rollout}(\tau,\mathcal{D}_I,P,\delta(\eta))
+
C_{Reward}(\delta(\eta))
+
C_{Update}(\sigma,\mathcal{D}_T,\tau,\mathcal{D}_I,\delta(\eta)),
\]
subject to the disjointness and memory-capacity constraints on \(\mathcal{D}_T\) and \(\mathcal{D}_I\) [2511.00796]. The paper focuses on the regime \(C_I > C_T\), arguing that schedules with \(C_T > C_I\) are undesirable because training becomes the bottleneck, staleness grows, and rollout workers must pause [2511.00796].

The scheduler has two phases. In the search phase, it chooses per-stage execution strategies under a fixed resource split. For training, it searches over combinations of data parallelism, tensor parallelism, and pipeline parallelism, while imposing the practical restriction that TP and DP must use GPUs of the same type [2511.00796]. For rollout generation, it formulates a MILP over rollout-replica configurations \(\psi\), with replica counts \(y_\psi\), workload assignments \(x_\psi\), throughput estimates \(h_\psi\), and a makespan variable \(\Theta\). The rollout planner minimizes the slowest configuration’s completion time under GPU-type budgets and total rollout count \(B\) [2511.00796].

The finite scheduling window \(\delta(\eta)\) mediates the staleness-throughput tradeoff. The paper explicitly rejects both \(\delta=\infty\), which would be too expensive and require knowledge of the full training horizon, and \(\delta \approx 1\), which would be too myopic for asynchronous coupling. Instead, AReaL-Hex starts from a small \(\delta(\eta)\) and increases it until scheduling decisions stabilize [2511.00796].

In the repartition phase, the cluster is modeled as a graph \(G=(\mathcal{D},\mathcal{E})\) whose nodes carry compute power \(c_d\), HBM bandwidth \(m_d\), and memory capacity \(M_d\), while edges carry communication bandwidth \(\beta_{d_1,d_2}\) [2511.00796]. The partition objective maximizes training-side interconnect bandwidth and rollout-side HBM bandwidth while constraining the fraction of total compute assigned to training:
\[
\mathcal{D}^{*}_T, \mathcal{D}^{*}_I
=
\arg \max_{\mathcal{D}_T,\mathcal{D}_I}
\left(
\frac{\sum_{d_1,d_2\in\mathcal{D}_T}\beta_{d_1,d_2}}
{\sum_{d'_1,d'_2\in\mathcal{D}}\beta_{d'_1,d'_2}}
+
\frac{\sum_{d\in\mathcal{D}_I} m_d}
{\sum_{d'\in\mathcal{D}} m_{d'}}
\right),
\]
subject to
\[
\gamma_L
\le
\frac{\sum_{d\in\mathcal{D}_T} c_d}
{\sum_{d'\in\mathcal{D}} c_{d'}}
\le
\gamma_H.
\]
The parameters \(\gamma_L\) and \(\gamma_H\) are then adjusted by binary search until \(C_T \approx C_I\), and the overall EM-like alternation between search and repartition continues until \(\max\{C_T,C_I\}\) stabilizes [2511.00796].

## 4. Heterogeneous hardware model and stage-specific mapping

The experimental heterogeneous setting is centered on NVIDIA H20 and H800 GPUs. H20 is reported with 148 TFLOPS tensor-core compute, 4 TB/s memory bandwidth, and 450 GB/s single-direction NVLink intra-machine bandwidth, whereas H800 is reported with 756 TFLOPS tensor-core compute, 2 TB/s memory bandwidth, and 200 GB/s single-direction NVLink intra-machine bandwidth [2511.00796]. The pricing assumptions are \(\$1.85\)/hour for H20 and \(\$5.28\)/hour for H800 [2511.00796].

These asymmetries drive the system’s stage mapping. H20 is treated as more cost-efficient for rollout generation because rollout is HBM-I/O-bound, while H800 is treated as more cost-efficient for training because optimization is compute-bound and collective-communication-heavy [2511.00796]. The paper quantifies this asymmetry by reporting that H20 is about \(2.72\times\) more cost-efficient for rollout generation and H800 about \(3.12\times\) more cost-efficient for training [2511.00796].

The paper also reports per-token cost estimates by model scale. For inference, H20 is cheaper than H800 at 1.5B, 7B, and 14B; for training, the ordering reverses at all three scales [2511.00796]. This stage-dependent inversion is the empirical basis for AReaL-Hex’s central premise: a mixed cluster can outperform either homogeneous alternative if the scheduler keeps rollout and training sufficiently balanced.

Interconnect heterogeneity is modeled explicitly. The assumed inter-machine bandwidth is 5 GB/s between same-type H20 or H800 machines, and 1.5 GB/s between H20 and H800 machines [2511.00796]. This makes weight synchronization a nontrivial cost in the heterogeneous setting. The paper therefore treats update time as an explicit component of \(C_I\), not as a negligible overhead.

## 5. Empirical performance, ablations, and scheduler behavior

The evaluation uses mathematical reasoning RL tasks with DeepSeek Distilled Qwen-2.5 models at 1.5B, 7B, and 14B, trained with GRPO and compared primarily against homogeneous AReaL deployments on H800-only and H20-only clusters [2511.00796]. Throughput is defined from average tokens processed divided by average RL training step time, where step time includes weight synchronization latency and the maximum of rollout-side and training-side duration, averaged over 30 RL training steps [2511.00796].

The headline results are twofold. Relative to homogeneous H800 deployments, AReaL-Hex achieves up to \(1.50\times\) speedup on 14B, at least \(1.31\times\) speedup on 7B, and \(1.39\times\) speedup on average [2511.00796]. Relative to homogeneous H20 deployments, it achieves up to \(2.76\times\) speedup on 7B, at least \(2.29\times\) speedup on 14B, and \(2.62\times\) average speedup [2511.00796]. The paper attributes these gains to stage-hardware matching: versus H800-only, rollout latency falls; versus H20-only, training latency falls [2511.00796].

The breakdown is consistent with that interpretation. Compared to homogeneous H800, AReaL-Hex reduces rollout-generation latency by a maximum of \(1.61\times\), a minimum of \(1.35\times\), and an average of \(1.46\times\). Compared to homogeneous H20, it reduces model-training latency by a maximum of \(3.13\times\), a minimum of \(1.85\times\), and an average of \(2.46\times\) [2511.00796].

At equal throughput, the heterogeneous system is also cheaper. For representative settings, AReaL-Hex yields about \(1.42\times\) lower cost for 1.5B, \(1.31\times\) lower cost for 7B, and \(1.50\times\) lower cost for 14B relative to homogeneous H800 [2511.00796]. Across heterogeneous cluster sizes from 24 to 56 GPUs, the paper reports fairly stable per-dollar throughput of about 200 tokens/s/\$ for 1.5B, 62 tokens/s/\$ for 7B, and 24 tokens/s/\$ for 14B [2511.00796].

The principal cost of heterogeneity is synchronization. Update times increase markedly in the mixed setting: for 1.5B, 7B, and 14B, AReaL-Hex reports 10.06 s, 58.34 s, and 112.93 s, respectively, compared with lower homogeneous values on both H800 and H20 [2511.00796]. As a share of RL iteration time, weight synchronization averages 13.54% in AReaL-Hex, compared with about 1.9% in homogeneous H800 [2511.00796]. The system remains net-positive because the stage-matching gains outweigh this penalty.

The ablation evidence shows that heterogeneity alone is insufficient without optimized resource partitioning. Compared with a uniform-allocation baseline AReaL(u), optimized allocation improves throughput by \(1.57\times\) at 1.5B, \(1.68\times\) at 7B, and \(1.67\times\) at 14B, for an average gain of \(1.63\times\) [2511.00796]. On the scheduler side, replacing either phase with exhaustive search greatly increases convergence time. The full two-phase scheduler converges in 14.9 s at 24 GPUs, 23.1 s at 32 GPUs, 46.7 s at 40 GPUs, and 2.0 min at 56 GPUs, whereas ablated variants range from minutes to at least 40 minutes [2511.00796].

## 6. Constraints, limitations, and broader significance

AReaL-Hex’s formulation is deliberately restricted. It explicitly assumes the useful regime \(C_I > C_T\), treats reward cost as a profiled constant because it varies minimally in the reported setting, and relies on profiled throughput models, rollout-length distributions, and fixed cluster-topology information [2511.00796]. It also narrows the search space with practical restrictions: TP and DP are constrained to same-type GPUs, and rollout TP is restricted to remain within a single machine [2511.00796].

The paper does not provide a formal queueing-theoretic proof of bounded staleness, nor a new RL correction mechanism beyond the underlying AReaL asynchronous framework. It also leaves several extensions open: dynamic schedule adaptation as rollout distributions shift במהלך training, richer reward pipelines whose cost is not approximately constant, and more irregular heterogeneous clusters with more than two GPU types or more complex network structure. This suggests that AReaL-Hex is best understood as a systems-layer contribution rather than as a new RL objective or stability theorem [2511.00796].

Its broader significance lies in reframing asynchronous RL for LLMs as an intrinsically heterogeneous workload. The paper’s central contribution is not merely to run RL on mixed hardware, but to make heterogeneity schedulable by jointly optimizing per-stage parallelization, workload assignment, topology-aware partitioning, and bounded-staleness execution. In that sense, AReaL-Hex turns the architectural flexibility of asynchronous RL into a deployment strategy: heterogeneous clusters become advantageous precisely because rollout generation and policy optimization are different kinds of computation, and the system is designed to exploit that mismatch rather than suppress it [2511.00796].

Source: https://www.emergentmind.com/topics/areal-hex