---
title: Fused AR-A2A Algorithm for Hybrid Parallelism
url: https://www.emergentmind.com/topics/fused-ar-a2a-communication-algorithm
type: topic
---

# Fused AR-A2A Algorithm for Hybrid Parallelism

A fused AR-A2A communication algorithm refers to a distributed communication mechanism that overlaps or hybridizes All-Reduce (AR) operations, typically used in tensor parallelism (TP), with All-to-All (A2A) operations, commonly used in expert parallelism (EP), in order to optimize network utilization and reduce overall communication latency in large-scale distributed systems. This approach is particularly impactful for serving Mixture-of-Experts (MoE) models and other workloads heavily dependent on both intra-node and inter-node communication primitives [2601.08800].

## 1. Theoretical Motivation and Hybrid Parallelism

Classical model parallelism for LLMs and MoE architectures relies on either TP (AR) or EP (A2A). Tensor parallelism distributes each tensor along its hidden dimension across $d$ ranks, synchronizing partial results via AR (often as a Reduce-Scatter (RS) followed by All-Gather (AG)), which is communication-efficient on intra-node high-bandwidth interconnects but becomes the bottleneck over slower inter-node links (e.g., InfiniBand, RoCE). Expert parallelism, conversely, dispatches tokens across experts via A2A, which scales better between nodes but can be impaired by load imbalance and large inter-node message volumes [2601.08800]. 

The fused AR-A2A communication algorithm addresses these constraints by combining TP (intra-node AR) and EP (inter-node A2A) in a single hybrid scheme. Within each node, intermediate tensors are sharded using AR before A2A dispatch is performed between nodes, dramatically reducing inter-node communication volume by a factor of $1/n_\mathrm{proc}$, where $n_\mathrm{proc}$ is the number of intra-node devices. This design enables concurrent intra-node aggregation and inter-node routing through nonblocking primitives, effectively overlapping the fast intra-node AR with the slower inter-node A2A [2601.08800].

## 2. Algorithmic Structure and Overlap Mechanisms

Two principal fused routines are integrated into the MoE block’s forward pass:

### a) Fused AG–Dispatch

Input tensors ($X \in \mathbb{R}^{b/d_\mathrm{DP} \times s \times h}$) are split among $m=n_\mathrm{proc}$ intra-node GPUs along the hidden dimension. Each TP rank $r_\mathrm{TP}$ shards its portion of the tensor further into $n=n_\mathrm{node}$ slices for A2A dispatch. Nonblocking communication (isend/irecv) is used to send slices to matching ranks on remote nodes. Upon receipt of all slices, each rank recovers the full subset required for its experts by performing an intra-node AG. The key is that, except for the initial and final communication rounds, inter-node dispatch and intra-node AG are substantially overlapped in time [2601.08800].

### b) Fused RS–Combine

After local expert computation, each rank contains a partial result sharded along $h$. Each intra-node rank splits its output into $n_\mathrm{node}$ blocks and exchanges as before with peer ranks using A2A. Reduce-Scatter (RS) operations run in parallel within the node, enabling local aggregation weighted by router probabilities. The process completes with an All-Gather to reconstruct the full output tensor. In this overlap architecture, RS and A2A proceed asynchronously, with only the AG step forming a sequential dependency at the end [2601.08800].

## 3. Mathematical Formulation and Communication Costs

The fused AR-A2A strategy is formalized by the following communication cost models:

- AR/TP for a tensor of size $\text{size}$ across $d$ ranks:
  $$
  \text{AR}(\text{size}, d) = \text{RS}\left( \frac{\text{size}}{d}, d \right) + \text{AG}\left( \frac{\text{size}}{d}, d \right)
  $$
 
- A2A/EP for the same tensor:
  $$
  \text{A2A}(\text{size}, d) \propto (d-1) \times \frac{\text{size}}{d}
  $$

- Without overlap (pure EP combine):
  $$
  \lambda_{\mathrm{EP}} = \text{AR}(bsh, n_\mathrm{proc}) + 2 \times \text{A2A}(bshk, n_\mathrm{node})
  $$

- With fused algorithm (MixServe):
  $$
  \lambda_{\mathrm{mix}} = \text{AR}(bsh, n_\mathrm{proc}) + \text{AG}\left( \frac{bshk}{n_\mathrm{proc}}, n_\mathrm{proc} \right) + 2 \times \text{A2A}\left( \frac{bshk}{n_\mathrm{proc}}, n_\mathrm{node} \right)
  $$

The core speedup arises from overlapping intra-node RS and inter-node A2A:
$$
T_{\mathrm{overlap}} = \max\{ T_{\mathrm{RS}}, T_{\mathrm{A2A}} \} + T_{\mathrm{AG}}
$$
as opposed to the sum in sequential execution. In ideal conditions ($T_{\mathrm{RS}} = T_{\mathrm{A2A}}$), a $2\times$ speedup is achievable for the combine stage [2601.08800]. 

## 4. System Implementation and Hardware Considerations

All inter-node communications leverage non-blocking isend/irecv and completion tests to maximize asynchrony with local computation and intra-node collectives. Two communication streams per device—one for intra-node (AR/AG), one for inter-node (A2A)—prevent head-of-line blocking. Tensor shards use zero-copy views or pointer slicing to avoid memory transfers. Temporary buffers of up to $O(bsh)$ per rank are utilized for lower-latency aggregation. Thread and rank mapping is optimized so that identical TP ranks across nodes communicate along matching channels [2601.08800].

Hardware requirements include multiple GPUs per node (NVLink, HCCS, etc.) for high-bandwidth intra-node collectives and a lower-bandwidth inter-node link (InfiniBand, RoCE). The algorithm is compatible with contemporary accelerators, as evidenced by implementation on Nvidia H20 and Huawei Ascend 910B platforms [2601.08800].

## 5. Empirical Results and Quantitative Improvements

The fused AR-A2A communication design enables substantial performance gains:

- **Time-to-First-Token (TTFT):** $1.08\times$–$3.80\times$ acceleration (e.g., Qwen3 prefill drops from 600 ms to 157 ms on Ascend910B, a $3.8\times$ reduction).
- **Inter-Token Latency (ITL):** $1.03\times$–$1.66\times$ speedup (e.g., Qwen3 ITL drops from 134 ms to 81 ms).
- **Throughput:** Increase by $5.2\%$–$50.3\%$ (e.g., DeepSeek-R1 streaming throughput on H20 increases from 363 tokens/s to 545 tokens/s, $50\%$ improvement).

These improvements are attributable to three primary factors:
1. Shrinking inter-node message size by $1/n_\mathrm{proc}$ through hidden-dimension sharding,
2. Overlapping expensive A2A traffic with inexpensive intra-node collectives,
3. Avoidance of additional memory copies and host launches due to fused routines.

The algorithm was validated on state-of-the-art MoE models (DeepSeek-R1 and Qwen3) and outperformed vLLM and Tutel-based hybrid and expert-parallel implementations [2601.08800].

## 6. Related Approaches and Comparative Techniques

Prior works such as “Optimizing Distributed ML Communication with Fused Computation–Collective Operations” illustrate analogous principles of fusing collective communication, particularly in scale-up and scale-out settings for deep learning workloads [2305.06942]. These approaches indicate that persistent kernels, nonblocking GPU-initiated primitives (e.g., ROC_SHMEM/NVSHMEM), and zero-copy buffer management can yield 12–31% lower per-layer latency and a 21% training pass reduction at scale. These findings corroborate the performance of fused AR-A2A designs and reinforce the importance of concurrent arithmetic and network operations for large distributed AI systems.

## 7. Summary and Significance

The fused AR-A2A communication algorithm constitutes a scalable, high-efficiency method for hybrid parallelism in distributed serving systems, particularly for large MoE and LLM architectures. By concurrent exploitation of intra-node all-reduce and inter-node all-to-all, combined with nonblocking and zero-copy mechanisms, the algorithm achieves measurable reductions in communication overhead, end-to-end latency, and throughput constraints. This approach is corroborated by MixServe’s empirical results and is recognized as foundational for modern, resource-efficient distributed AI inference [2601.08800][2305.06942].

| Dimension                     | Classical (TP/EP)         | Fused AR–A2A                 |
|-------------------------------|---------------------------|------------------------------|
| Inter-node communication      | High (AR/EP)              | Sharply reduced ($1/n_\mathrm{proc}$) |
| Overlap of collectives        | None (sequential)         | Substantial (max concurrency) |
| Throughput/latency gain       | Modest                    | Up to $3.8\times$ TTFT, 50%+ throughput |

*This table situates the fused AR-A2A algorithm with respect to classical TP/EP designs using published metrics [2601.08800].*

Source: https://www.emergentmind.com/topics/fused-ar-a2a-communication-algorithm