---
title: RingX Sequence Parallelism
url: https://www.emergentmind.com/topics/ringx-sequence-parallelism
type: topic
---

# RingX Sequence Parallelism

“RingX sequence parallelism” is best understood here as an *Editor’s term* for the family of ring-based sequence-parallel methods in which the sequence dimension is sharded across devices, local query blocks remain on their home ranks, and remote key/value blocks or equivalent state are exchanged so that each local shard can compute attention against the full context without materializing all global activations on one device at once. The cited literature does not define a method literally named “RingX”; instead, it repeatedly identifies **Ring-Attention**, **zigzag ring attention**, and closely related context-parallel formulations as the nearest technical correspondence, while later systems reinterpret, hybridize, or replace the ring substrate according to topology, sparsity pattern, masking constraints, or attention algebra [2505.22296] [2509.26541].

## 1. Conceptual scope and historical lineage

A clear systems-level precursor appears in “Sequence Parallelism: Long Sequence Training from System Perspective,” which splits the input sequence into multiple chunks and feeds each chunk into its corresponding device, then integrates ring-style communication with self-attention through **Ring Self-Attention (RSA)** [2105.13120]. In that formulation, no single device is required to hold the whole sequence, and exact global attention is reconstructed by streaming nonlocal key/value information across the device group. The paper reports that, compared with tensor parallelism, the approach achieved **13.7$\times$** and **3.0$\times$** maximum batch size and sequence length respectively when scaling up to **64 NVIDIA P100 GPUs**, and with sparse attention it handled sequences of **over 114K tokens** [2105.13120].

Later work distinguishes two mature long-context sequence-parallel paradigms: **DeepSpeed-Ulysses**, which uses all-to-all communication to remap sequence shards into head shards, and **Ring-Attention**, which keeps the sequence partition and streams remote \(K,V\) blocks through point-to-point communication [2405.07719]. “USP: A Unified Sequence Parallelism Approach for Long Context Generative AI” formalizes this relationship by treating the overall sequence-parallel degree as
\[
N = N_{\text{ulysses}} \times N_{\text{ring}},
\]
so that pure Ulysses and pure Ring-Attention become special cases of a two-dimensional decomposition [2405.07719].

The terminology remains unstable in practitioner-facing reports. “360-LLaMA-Factory: Plug & Play Sequence Parallelism for Long Post-Training” states explicitly that it does **not** define a method named RingX and that the closest corresponding method is **Ring-Attention**, specifically the implementation built from `zigzag_ring_flash_attn_func` [2505.22296]. This suggests that “RingX sequence parallelism” is best treated as a family resemblance term rather than a canonical algorithm name.

## 2. The distributed attention contract

The defining contract of ring-style sequence parallelism is stable across dense-attention variants. The sequence is partitioned across \(n\) accelerators; each accelerator stores one local query chunk and the corresponding local \(K,V\) chunk; queries remain resident on their home accelerator; and \(K,V\) chunks are exchanged so that each local query chunk can eventually attend to every global \(K,V\) chunk [2509.26541]. Operationally, each accelerator computes attention for its local \(Q\) against a stream of local-or-received \(K,V\) blocks, typically using FlashAttention, and accumulates the final output and log-sum-exp statistics over iterations [2509.26541].

The ring schedule is attractive because it preserves two properties emphasized in “TASP: Topology-aware Sequence Parallelism.” First, **accessibility**: after the full rotation, every local query chunk has seen all \(K,V\) chunks. Second, **zero-copy**: there is no global duplication of \(K,V\); only one copy of each chunk is in flight along the ring [2509.26541]. The early RSA formulation expresses the same idea as a blockwise reconstruction of local output rows,
\[
O^n = \sum_{i=1}^N S_i^n V_i,
\]
where each device holds only its local query rows and incrementally accumulates contributions from remote value blocks [2105.13120].

A persistent misconception is that any long-context parallelism operating on attention is therefore “RingX.” “HelixPipe: Efficient Distributed Training of Long Sequence Transformers with Attention Parallel Pipeline Parallelism” is a counterexample: it is explicitly **not** a ring-based sequence-parallel method, but an inter-layer pipeline schedule that is orthogonal to sequence parallelism and can be integrated with Megatron SP, Ulysses, Ring Attention, Megatron context parallelism, or USP [2507.00394]. RingX-style methods concern intra-layer distributed attention over the sequence dimension; HelixPipe concerns inter-layer pipeline bubbles, stage memory balance, and communication/computation overlap across micro-batches.

## 3. Communication substrates and topology awareness

The principal systems critique of classical ring attention is not semantic but topological. “TASP” argues that Ring AllGather is badly mismatched to modern all-to-all accelerator fabrics: in an all-to-all topology with \(n\) accelerators, there are \(n(n-1)\) unidirectional links available in principle, but one ring transfer iteration uses only \(n\) directed edges, so the resulting utilization of full communication capacity is
\[
\frac{1}{n-1},
\]
which for \(n=8\) is approximately **14.3\%** [2509.26541]. The algorithmic contract still wants ring-style streaming for memory reasons, but the hardware exposes many more concurrent noninterfering links than a single ring uses.

TASP’s answer is to decompose both the physical topology and the Ring AllGather primitive. In a complete directed graph \(K_n\), Hamiltonian decomposition yields \(n-1\) edge-disjoint directed Hamiltonian cycles; TASP uses those cycles as orthogonal ring datapaths and decomposes Ring AllGather into the same number of concurrent ring-style transfers [2509.26541]. The logical attention semantics remain unchanged—sequence sharding, stationary \(Q\), streamed \(K,V\), and zero-copy access—but the transport schedule becomes topology-matched. On single-node and multi-node H100 systems and on MI300X, the paper reports up to **3.58** speedup over Ring Attention and Zigzag-Ring Attention [2509.26541].

A related but distinct hybridization appears in USP, which keeps Ring-Attention as one axis and Ulysses as another. The paper argues that ring-based and all-to-all–based sequence parallelism are complementary rather than mutually exclusive, and that the best split depends on architecture and network topology [2405.07719]. “LoongTrain” (described in the provided material as **EpicSeq**) pushes the same logic further with **2D-Attention** and **Double-Ring-Attention**, combining head parallelism and context parallelism to break head-count limits while reducing the number of P2P steps in the context dimension; it reports Model FLOPs Utilization improvements of up to **2.88x** over DeepSpeed-Ulysses and Megatron Context Parallelism [2406.18485].

For inference on multi-machine DiT systems, “StreamFusion” argues that the customary USP choice—Ulysses intra-node and Ring inter-node—is itself topology-suboptimal. It instead places **Ulysses for inter-machine communication** and **Ring for intra-machine communication**, adds **Torus Attention** to overlap inter-machine all-to-all operations with computation, and uses one-sided communication to reduce GPU sender-receiver synchronization; the reported gain is **1.35$\times$** on average and **up to 1.77$\times$** over the state of the art [2601.20273]. This suggests that RingX should be understood not as a fixed ring primitive, but as a mutable communication substrate whose best realization depends strongly on the bandwidth hierarchy.

## 4. Causal masks, load balance, and packed-sequence correctness

Causal masking introduces a second layer of complexity. In vanilla ring placement, contiguous sequence partitioning can create severe per-iteration imbalance because later query positions have more valid predecessors than earlier ones. “TASP” makes this explicit and notes that Megatron CP’s zig-zag placement fixes the imbalance for standard ring exchange, while its own **Zig-zag TASP** generalizes the idea to the multi-ring schedule and yields perfect load balance for causal attention [2509.26541]. USP likewise describes token reordering that pairs front and back chunks so each GPU receives a balanced mix of early and late positions under causal masking [2405.07719].

Packed sequences create a different problem: **cross-contamination** across unrelated packed segments. “HSAP: A Hierachical Sequence-aware Parallelism for Hybrid-Context Generative Models” states that existing approaches either ignore the scenario of hybrid-context sequences or sacrifice and limit parallelism degree to support it, and proposes **Sequence-Aware Parallelism** plus a hierarchical framework that uses JIT compilation to optimize the communication strategy of all device groups at NCCL level [2606.30460]. The central issue is not merely which remote \(K,V\) blocks arrive, but which *subregions* of those blocks are legally visible under both causality and packed-sequence boundaries.

Irregular sparsity creates an analogous scheduling problem. “db-SP: Accelerating Sparse Attention for Visual Generative Models with Dual-Balanced Sequence Parallelism” shows that once block-wise sparse attention is combined with Ulysses, Ring Attention, or USP, the dominant issue is often not the ring pass itself but workload imbalance across heads and across \((Q\text{-chunk},KV\text{-chunk})\) regions [2511.23113]. Its dual-balanced partitioning balances both levels and reports an average **attention speedup of 1.40$\times$** and **end-to-end speedup of 1.25$\times$** over state-of-the-art sequence-parallel methods [2511.23113]. A plausible implication is that RingX-style methods on sparse masks require a load-balancing layer in addition to a communication schedule.

## 5. Beyond dense softmax attention

Ring-style sequence parallelism is not restricted to dense softmax attention. “Linear Attention Sequence Parallelism” shows that for linear attention, the inter-shard dependency can be carried by a compact recurrent state rather than by full token-shaped activations [2404.02882]. LASP partitions the sequence into chunks, performs local intra-chunk computation, and passes a \(d \times d\)-style memory state around the ring in forward and a reverse-time gradient state in backward. The paper reports scaling sequence length up to **4096K** on **128 GPUs**, which is **8$\times$** longer than existing SP methods, and characterizes the communicated state as independent of sequence length [2404.02882].

“LASP-2: Rethinking Sequence Parallelism for Linear Attention and Its Hybrid” then argues that even this ring-style communication is suboptimal for linear attention because the right-product-first structure implies a smaller minimal communication requirement [2502.07563]. LASP-2 replaces the ring of small P2P exchanges with a single AllGather on intermediate memory states whose sizes are independent of sequence length, and reports training speed improvements of **15.2\%** over LASP and **36.6\%** over Ring Attention at **2048K** sequence length on **64 GPUs** [2502.07563]. The linear-attention case therefore marks an important boundary of the RingX concept: if the attention algebra admits a compact associative state, the ring may cease to be the preferred communication substrate.

A different non-ring alternative appears in “OSP-Next,” whose **Sparse Sequence Parallelism (SSP)** is specialized to the fixed sparse layout of Skiparse-2D Attention [2605.28691]. SSP partitions subsequences across ranks and switches sparse patterns through a single All-to-All communication; compared with Ulysses Sequence Parallelism, the paper says SSP reduces communication volume by **75\%** [2605.28691]. This reinforces the broader point that sequence parallelism is a family of distributed-attention execution strategies, of which ring schedules are only one important subset.

## 6. Systems integration, scheduling, and practical envelope

Recent work increasingly treats sequence parallelism as a scheduling and systems-integration problem rather than only an attention-kernel problem. “FlexSP” models heterogeneous sequence lengths in real training batches as an optimization problem and adaptively chooses different sequence-parallel group sizes for different sequences or micro-batches; it reports speedups of up to **1.98x** over state-of-the-art training frameworks [2412.01523]. “AutoSP” moves a Ulysses-style SP transformation into the PyTorch compiler stack and combines it with long-context-aware activation checkpointing, increasing trainable contexts by up to **2.7$\times$** on NVIDIA hardware and **2.5$\times$** on AMD hardware over a competitive hand-written baseline at negligible runtime cost [2604.27089]. Neither is itself RingX, but both indicate that sequence parallelism increasingly lives inside automated planners, graph rewrites, and runtime policies.

Serving systems expose a similar shift. “Tetris” treats ring-attention-style SP as an execution substrate and introduces **Chunkwise Dynamic Sequence Parallelism (CDSP)**, which assigns different SP sizes to different chunks of the same request [2511.06247]. Compared with state-of-the-art systems, it achieves up to **4.35$\times$** lower time-to-first-token under max sustainable loads, reduces median time-between-tokens by up to **40.1\%**, and increases max request capacity by up to **45\%** [2511.06247]. This suggests that even an efficient RingX kernel can leave substantial performance unrealized if SP is allocated only at request granularity.

A final adjacent development is “Folding Tensor and Sequence Parallelism for Memory-Efficient Transformer Training & Inference,” which introduces **TSP** by folding tensor parallelism and sequence parallelism onto a single device axis [2604.26294]. TSP attention reconstructs context through a sequence-wise key/value exchange, while gated MLP weight shards circulate in a ring and partial outputs accumulate locally [2604.26294]. This is not conventional RingX sequence parallelism, but it shows how ring schedules can migrate from attention into other sublayers when weight sharding and sequence sharding are coupled on the same ranks.

Taken together, these works portray RingX sequence parallelism not as a single algorithm, but as a durable design pattern: shard the sequence, keep local ownership of output positions, avoid global activation materialization, and use structured communication to reconstruct the needed context. The most active research questions now concern *which* structure—single ring, multi-ring, hybrid Ulysses-ring, topology-aware torus, sequence-aware packed scheduling, compact-state all-gather, or compiler-generated group layouts—best matches a given attention law, masking regime, and hardware fabric [2509.26541] [2601.20273]

Source: https://www.emergentmind.com/topics/ringx-sequence-parallelism