RingX Sequence Parallelism

• RingX sequence parallelism is a family of ring-based methods that shards the sequence across devices, keeping queries local and streaming remote K,V blocks.
• It enhances scalability by eliminating the need for any device to hold the full sequence, achieving memory efficiency and communication balance across accelerators.
• This approach adapts to dense, sparse, and linear attention formulations, combining topology-aware scheduling and hybrid strategies to boost performance.

“RingX sequence parallelism” is best understood here as an Editor’s term for the family of ring-based sequence-parallel methods in which the sequence dimension is sharded across devices, local query blocks remain on their home ranks, and remote key/value blocks or equivalent state are exchanged so that each local shard can compute attention against the full context without materializing all global activations on one device at once. The cited literature does not define a method literally named “RingX”; instead, it repeatedly identifies Ring-Attention, zigzag ring attention, and closely related context-parallel formulations as the nearest technical correspondence, while later systems reinterpret, hybridize, or replace the ring substrate according to topology, sparsity pattern, masking constraints, or attention algebra (Zou et al., 28 May 2025, Wang et al., 30 Sep 2025).

1. Conceptual scope and historical lineage

A clear systems-level precursor appears in “Sequence Parallelism: Long Sequence Training from System Perspective,” which splits the input sequence into multiple chunks and feeds each chunk into its corresponding device, then integrates ring-style communication with self-attention through Ring Self-Attention (RSA) (Li et al., 2021). In that formulation, no single device is required to hold the whole sequence, and exact global attention is reconstructed by streaming nonlocal key/value information across the device group. The paper reports that, compared with tensor parallelism, the approach achieved 13.7$\times$ and 3.0$\times$ maximum batch size and sequence length respectively when scaling up to 64 NVIDIA P100 GPUs, and with sparse attention it handled sequences of over 114K tokens (Li et al., 2021).

Later work distinguishes two mature long-context sequence-parallel paradigms: DeepSpeed-Ulysses, which uses all-to-all communication to remap sequence shards into head shards, and Ring-Attention, which keeps the sequence partition and streams remote $K,V$ blocks through point-to-point communication (Fang et al., 2024). “USP: A Unified Sequence Parallelism Approach for Long Context Generative AI” formalizes this relationship by treating the overall sequence-parallel degree as

$N = N_{\text{ulysses}} \times N_{\text{ring}},$

so that pure Ulysses and pure Ring-Attention become special cases of a two-dimensional decomposition (Fang et al., 2024).

The terminology remains unstable in practitioner-facing reports. “360-LLaMA-Factory: Plug & Play Sequence Parallelism for Long Post-Training” states explicitly that it does not define a method named RingX and that the closest corresponding method is Ring-Attention, specifically the implementation built from zigzag_ring_flash_attn_func (Zou et al., 28 May 2025). This suggests that “RingX sequence parallelism” is best treated as a family resemblance term rather than a canonical algorithm name.

2. The distributed attention contract

The defining contract of ring-style sequence parallelism is stable across dense-attention variants. The sequence is partitioned across $n$ accelerators; each accelerator stores one local query chunk and the corresponding local $K,V$ chunk; queries remain resident on their home accelerator; and $K,V$ chunks are exchanged so that each local query chunk can eventually attend to every global $K,V$ chunk (Wang et al., 30 Sep 2025). Operationally, each accelerator computes attention for its local $Q$ against a stream of local-or-received $K,V$ blocks, typically using FlashAttention, and accumulates the final output and log-sum-exp statistics over iterations (Wang et al., 30 Sep 2025).

The ring schedule is attractive because it preserves two properties emphasized in “TASP: Topology-aware Sequence Parallelism.” First, accessibility: after the full rotation, every local query chunk has seen all $\times$0 chunks. Second, zero-copy: there is no global duplication of $\times$1; only one copy of each chunk is in flight along the ring (Wang et al., 30 Sep 2025). The early RSA formulation expresses the same idea as a blockwise reconstruction of local output rows,

$\times$2

where each device holds only its local query rows and incrementally accumulates contributions from remote value blocks (Li et al., 2021).

A persistent misconception is that any long-context parallelism operating on attention is therefore “RingX.” “HelixPipe: Efficient Distributed Training of Long Sequence Transformers with Attention Parallel Pipeline Parallelism” is a counterexample: it is explicitly not a ring-based sequence-parallel method, but an inter-layer pipeline schedule that is orthogonal to sequence parallelism and can be integrated with Megatron SP, Ulysses, Ring Attention, Megatron context parallelism, or USP (Zhang et al., 1 Jul 2025). RingX-style methods concern intra-layer distributed attention over the sequence dimension; HelixPipe concerns inter-layer pipeline bubbles, stage memory balance, and communication/computation overlap across micro-batches.

3. Communication substrates and topology awareness

The principal systems critique of classical ring attention is not semantic but topological. “TASP” argues that Ring AllGather is badly mismatched to modern all-to-all accelerator fabrics: in an all-to-all topology with $\times$3 accelerators, there are $\times$4 unidirectional links available in principle, but one ring transfer iteration uses only $\times$5 directed edges, so the resulting utilization of full communication capacity is

$\times$6

which for $\times$7 is approximately 14.3\% (Wang et al., 30 Sep 2025). The algorithmic contract still wants ring-style streaming for memory reasons, but the hardware exposes many more concurrent noninterfering links than a single ring uses.

TASP’s answer is to decompose both the physical topology and the Ring AllGather primitive. In a complete directed graph $\times$8, Hamiltonian decomposition yields $\times$9 edge-disjoint directed Hamiltonian cycles; TASP uses those cycles as orthogonal ring datapaths and decomposes Ring AllGather into the same number of concurrent ring-style transfers (Wang et al., 30 Sep 2025). The logical attention semantics remain unchanged—sequence sharding, stationary $K,V$0, streamed $K,V$1, and zero-copy access—but the transport schedule becomes topology-matched. On single-node and multi-node H100 systems and on MI300X, the paper reports up to 3.58 speedup over Ring Attention and Zigzag-Ring Attention (Wang et al., 30 Sep 2025).

A related but distinct hybridization appears in USP, which keeps Ring-Attention as one axis and Ulysses as another. The paper argues that ring-based and all-to-all–based sequence parallelism are complementary rather than mutually exclusive, and that the best split depends on architecture and network topology (Fang et al., 2024). “LoongTrain” (described in the provided material as EpicSeq) pushes the same logic further with 2D-Attention and Double-Ring-Attention, combining head parallelism and context parallelism to break head-count limits while reducing the number of P2P steps in the context dimension; it reports Model FLOPs Utilization improvements of up to 2.88x over DeepSpeed-Ulysses and Megatron Context Parallelism (Gu et al., 2024).

For inference on multi-machine DiT systems, “StreamFusion” argues that the customary USP choice—Ulysses intra-node and Ring inter-node—is itself topology-suboptimal. It instead places Ulysses for inter-machine communication and Ring for intra-machine communication, adds Torus Attention to overlap inter-machine all-to-all operations with computation, and uses one-sided communication to reduce GPU sender-receiver synchronization; the reported gain is 1.35$K,V$2 on average and up to 1.77$K,V$3 over the state of the art (Yang et al., 28 Jan 2026). This suggests that RingX should be understood not as a fixed ring primitive, but as a mutable communication substrate whose best realization depends strongly on the bandwidth hierarchy.

4. Causal masks, load balance, and packed-sequence correctness

Causal masking introduces a second layer of complexity. In vanilla ring placement, contiguous sequence partitioning can create severe per-iteration imbalance because later query positions have more valid predecessors than earlier ones. “TASP” makes this explicit and notes that Megatron CP’s zig-zag placement fixes the imbalance for standard ring exchange, while its own Zig-zag TASP generalizes the idea to the multi-ring schedule and yields perfect load balance for causal attention (Wang et al., 30 Sep 2025). USP likewise describes token reordering that pairs front and back chunks so each GPU receives a balanced mix of early and late positions under causal masking (Fang et al., 2024).

Packed sequences create a different problem: cross-contamination across unrelated packed segments. “HSAP: A Hierachical Sequence-aware Parallelism for Hybrid-Context Generative Models” states that existing approaches either ignore the scenario of hybrid-context sequences or sacrifice and limit parallelism degree to support it, and proposes Sequence-Aware Parallelism plus a hierarchical framework that uses JIT compilation to optimize the communication strategy of all device groups at NCCL level (Zhang et al., 29 Jun 2026). The central issue is not merely which remote $K,V$4 blocks arrive, but which subregions of those blocks are legally visible under both causality and packed-sequence boundaries.

Irregular sparsity creates an analogous scheduling problem. “db-SP: Accelerating Sparse Attention for Visual Generative Models with Dual-Balanced Sequence Parallelism” shows that once block-wise sparse attention is combined with Ulysses, Ring Attention, or USP, the dominant issue is often not the ring pass itself but workload imbalance across heads and across $K,V$5 regions (Chen et al., 28 Nov 2025). Its dual-balanced partitioning balances both levels and reports an average attention speedup of 1.40$K,V$6 and end-to-end speedup of 1.25$K,V$7 over state-of-the-art sequence-parallel methods (Chen et al., 28 Nov 2025). A plausible implication is that RingX-style methods on sparse masks require a load-balancing layer in addition to a communication schedule.

5. Beyond dense softmax attention

Ring-style sequence parallelism is not restricted to dense softmax attention. “Linear Attention Sequence Parallelism” shows that for linear attention, the inter-shard dependency can be carried by a compact recurrent state rather than by full token-shaped activations (Sun et al., 2024). LASP partitions the sequence into chunks, performs local intra-chunk computation, and passes a $K,V$8-style memory state around the ring in forward and a reverse-time gradient state in backward. The paper reports scaling sequence length up to 4096K on 128 GPUs, which is 8$K,V$9 longer than existing SP methods, and characterizes the communicated state as independent of sequence length (Sun et al., 2024).

“LASP-2: Rethinking Sequence Parallelism for Linear Attention and Its Hybrid” then argues that even this ring-style communication is suboptimal for linear attention because the right-product-first structure implies a smaller minimal communication requirement (Sun et al., 11 Feb 2025). LASP-2 replaces the ring of small P2P exchanges with a single AllGather on intermediate memory states whose sizes are independent of sequence length, and reports training speed improvements of 15.2\% over LASP and 36.6\% over Ring Attention at 2048K sequence length on 64 GPUs (Sun et al., 11 Feb 2025). The linear-attention case therefore marks an important boundary of the RingX concept: if the attention algebra admits a compact associative state, the ring may cease to be the preferred communication substrate.

A different non-ring alternative appears in “OSP-Next,” whose Sparse Sequence Parallelism (SSP) is specialized to the fixed sparse layout of Skiparse-2D Attention (Ge et al., 27 May 2026). SSP partitions subsequences across ranks and switches sparse patterns through a single All-to-All communication; compared with Ulysses Sequence Parallelism, the paper says SSP reduces communication volume by 75\% (Ge et al., 27 May 2026). This reinforces the broader point that sequence parallelism is a family of distributed-attention execution strategies, of which ring schedules are only one important subset.

6. Systems integration, scheduling, and practical envelope

Recent work increasingly treats sequence parallelism as a scheduling and systems-integration problem rather than only an attention-kernel problem. “FlexSP” models heterogeneous sequence lengths in real training batches as an optimization problem and adaptively chooses different sequence-parallel group sizes for different sequences or micro-batches; it reports speedups of up to 1.98x over state-of-the-art training frameworks (Wang et al., 2024). “AutoSP” moves a Ulysses-style SP transformation into the PyTorch compiler stack and combines it with long-context-aware activation checkpointing, increasing trainable contexts by up to 2.7$N = N_{\text{ulysses}} \times N_{\text{ring}},$0 on NVIDIA hardware and 2.5$N = N_{\text{ulysses}} \times N_{\text{ring}},$1 on AMD hardware over a competitive hand-written baseline at negligible runtime cost (Gupta et al., 29 Apr 2026). Neither is itself RingX, but both indicate that sequence parallelism increasingly lives inside automated planners, graph rewrites, and runtime policies.

Serving systems expose a similar shift. “Tetris” treats ring-attention-style SP as an execution substrate and introduces Chunkwise Dynamic Sequence Parallelism (CDSP), which assigns different SP sizes to different chunks of the same request (Li et al., 9 Nov 2025). Compared with state-of-the-art systems, it achieves up to 4.35$N = N_{\text{ulysses}} \times N_{\text{ring}},$2 lower time-to-first-token under max sustainable loads, reduces median time-between-tokens by up to 40.1\%, and increases max request capacity by up to 45\% (Li et al., 9 Nov 2025). This suggests that even an efficient RingX kernel can leave substantial performance unrealized if SP is allocated only at request granularity.

A final adjacent development is “Folding Tensor and Sequence Parallelism for Memory-Efficient Transformer Training & Inference,” which introduces TSP by folding tensor parallelism and sequence parallelism onto a single device axis (Shyam et al., 29 Apr 2026). TSP attention reconstructs context through a sequence-wise key/value exchange, while gated MLP weight shards circulate in a ring and partial outputs accumulate locally (Shyam et al., 29 Apr 2026). This is not conventional RingX sequence parallelism, but it shows how ring schedules can migrate from attention into other sublayers when weight sharding and sequence sharding are coupled on the same ranks.

Taken together, these works portray RingX sequence parallelism not as a single algorithm, but as a durable design pattern: shard the sequence, keep local ownership of output positions, avoid global activation materialization, and use structured communication to reconstruct the needed context. The most active research questions now concern which structure—single ring, multi-ring, hybrid Ulysses-ring, topology-aware torus, sequence-aware packed scheduling, compact-state all-gather, or compiler-generated group layouts—best matches a given attention law, masking regime, and hardware fabric (Wang et al., 30 Sep 2025, Yang et al., 28 Jan 2026)