---
title: Tensor & Sequence Parallelism
url: https://www.emergentmind.com/topics/tensor-and-sequence-parallelism-tsp
type: topic
---

# Tensor & Sequence Parallelism

Tensor and Sequence Parallelism (TSP) refers to system and algorithmic strategies that jointly shard both model parameters (weights) and input activations (tokens, or sequence positions) across parallel computing resources. By integrating tensor parallelism (TP) and sequence parallelism (SP), TSP reduces both parameter-memory and activation-memory per device, thus alleviating hardware bottlenecks for training and inference on large-scale models, especially with long input contexts. Contemporary TSP approaches include “folding” TP and SP onto a single mesh axis, as well as 2D block decompositions (head-context parallelism) and hybrid topologies combining AllToAll and ring-based communication schemes. TSP enables hardware-aware scaling, supports integration with data, pipeline, and expert parallelism, and underpins recent advances in memory-efficient large-model training [2604.26294][2406.18485][2405.07719][2105.13120][2003.01039].

## 1. Principal Techniques and Architectural Dimensions

TSP incorporates and generalizes both tensor parallelism and sequence parallelism. In TP, model weights (e.g., Q/K/V projections, MLP matrices) are partitioned along feature axes so each device holds a weight shard and associated partial activations [2405.07719][2604.26294]. In SP, the sequence dimension of activations (tokens) is split across devices; each device processes a contiguous token block with a full model replica [2105.13120][2405.07719]. 

TSP “folds” these orthogonal axes onto a single device axis such that each rank holds both a weight and a sequence shard [2604.26294]. This structure applies to both dense and Mixture-of-Experts (MoE) architectures, unifies memory scaling, and frees devices for further axes such as data or expert parallelism.

Mechanisms for block partitioning include:
- 2D attention blockwise decomposition: partitioning the attention computation along both head (feature) and sequence (context) axes, creating a grid of $(H/d_{\textrm{hp}})\times(L/d_{\textrm{cp}})$ blocks, with each block assigned to a device or process group [2406.18485].
- Ring-based attention: implementing distributed self-attention by rotating key/value or activation slices in a ring topology to all devices handling a sequence-shard, guaranteeing complete context aggregation with limited communication per step [2105.13120][2405.07719].
- “Unified” group factorizations: arranging process groups into multidimensional meshes that flexibly interpolates between AllToAll (Ulysses) and ring-passing (Ring) for optimal hardware utilization and scaling [2405.07719].

## 2. Formal Memory and Communication Analysis

TSP achieves simultaneous reduction in parameter and activation memory by $1/D$ (where $D$ is the combined TSP group size) over conventional methods [2604.26294]. 

Table: Per-GPU memory in several parallelism strategies ([2604.26294])

| Parallelism | Parameter Memory         | Activation Memory    |
|-------------|-------------------------|---------------------|
| DP          | $LP_L\beta_P$           | $M_a^{\textrm{sel}}$|
| TP          | $LP_L\beta_P/D$         | $M_a^{\textrm{sel}}$|
| SP          | $LP_L\beta_P$           | $M_a^{\textrm{sel}}/D$|
| TP+SP       | $(LP_L\beta_P)/T$       | $M_a^{\textrm{sel}}/\Sigma$ |
| TSP         | $LP_L\beta_P/D$         | $M_a^{\textrm{sel}}/D$|

Here $LP_L$ is per-layer parameter count, $\beta_P$ is bytes/parameter, $M_a^{\textrm{sel}}$ is peak activation memory, $D$ is TSP degree.

Communication in TSP combines:
- Broadcast or ring-rotate of projection parameter shards (Q/K/V/MLP weights).
- All-gather of key/value activation blocks by sequence-shard for full attention context.
- Aggregation of partial FFN outputs during MLP forward/backward.

Dominant communication term (per layer, per GPU, under selective recompute) [2604.26294]:
$$
C^{\textrm{TSP}}_{\textrm{fwd}} = A\beta_P + M\beta_P\frac{D-1}{D} + \frac{2BSH\beta_P}{g}\frac{D-1}{D}
$$
where $A$ and $M$ depend on QKV/MLP parameters, $g$ is grouped-query ratio, $S$ is context length, $H$ is hidden dimension, $B$ is microbatch.

Compared to TP and SP (which have all-reduce and all-gather bottlenecks, respectively), TSP balances compute and communication, allowing both memory and throughput scaling. The activation communication cost for TSP matches that of SP at the same degree, but TSP incurs additional parameter exchanges (which can be overlapped with compute) [2604.26294].

## 3. Algorithmic Schedules and Implementations

**Attention Blocks:**  
- Each rank computes Q/K/V projections with its own local weights and activation shard.
- Parameter shards are broadcast or rotated to all other ranks so each device eventually generates projections for all parameter splits [2604.26294].
- All-gather is performed on local K/V projections along the sequence axis to reconstruct the global attention context per shard.
- Blockwise causal attention (e.g., FlashAttention) is applied locally, followed by projection and fusion of output blocks [2406.18485][2604.26294].

**Gated MLP/FFN Blocks:**  
- MLP weights are circulated in a ring so that each rank sequentially applies each parameter shard to its local input and accumulates the resulting partial outputs.
- No global all-reduce is needed since outputs are local to each activation shard [2604.26294].

**2D and Unified Group Schedules:**  
- Several systems arrange devices in a 2D or 4D mesh. For example, LoongTrain partitions along both head and sequence axes; Unified SP (USP) creates a mesh of ring and Ulysses groups, interpolating between pure AllToAll and pure Ring strategies [2406.18485][2405.07719].
- These layouts allow for optimal mapping to hardware (intra-node links, NIC count), minimize cross-node data movement, and are compatible with ZeRO, pipeline, and data parallelism [2406.18485][2405.07719][2105.13120].

**Overlapping Communication and Compute:**  
- Implementation best practices use dedicated communication streams for broadcast, all-gather, and ring sends/receives.
- Compute kernels (GEMMs, FlashAttention) are scheduled concurrent with weight or context communication [2604.26294][2406.18485].
- On modern accelerators, this overlap effectively hides much of the parameter-shard and sequence-context communication.

## 4. Comparative Performance and Scaling

Empirical benchmarks across modern architectures and hardware demonstrate distinct TSP scaling properties [2604.26294][2406.18485][2405.07719][2105.13120]:

- **Peak Memory:** TSP achieves the lowest per-GPU peak memory across a wide range of sequence lengths, matching TP at short context ($S$ small, parameter-dominates), and matching SP as $S$ grows (activation-dominates) [2604.26294].
- **Throughput:** TSP consistently outperforms matched TP+SP factorizations, with throughput advantage widening at larger device count ($D$), especially on long sequences ($S\gg h$).
- **Batch Size and Sequence Length Scaling:** SP and TSP enable training with $13.7\times$ larger batch size and $3\times$ longer context than TP (on 64 GPUs for BERT-Base), as well as support for $>100\,000$ token sequences using sparse attention kernels [2105.13120].
- **Hardware Utilization:** LoongTrain achieves up to $2.88\times$ Model FLOPs Utilization (MFU) over DeepSpeed-Ulysses or Megatron-CP for 1M-token LLM training, and near-linear scaling up to 512 GPUs [2406.18485]. Unified SP (USP) attains up to $86\%$ hardware utilization (MFU) on LLAMA3-8B at $208,000$ context [2405.07719].

Table: TSP and Baseline MFU for 7B LLMs ([2406.18485])

| Method            | Sequence Length | GPUs | Best MFU (%) | Relative Speedup |
|-------------------|----------------|------|--------------|------------------|
| DeepSpeed-Ulysses | 1M             | 64   | 36           | 1×               |
| Megatron-CP       | 1M             | 64   | 38           | 1.05×            |
| LoongTrain TSP    | 1M             | 64   | 55           | 1.5×             |

This suggests that practical, hardware-aware TSP yields both superior scaling and resource utilization for very long context LLM training versus orthogonal or baseline approaches.

## 5. Composability with Multidimensional Parallelism

TSP is designed for integration as a mesh axis in “4D” or higher dimensional parallelism frameworks. The canonical mesh ordering is:
1. Tensor Parallelism (TP) or TSP at the lowest level (shards model weights/activations)
2. Unified Sequence Parallelism (USP) or pure SP
3. ZeRO/Data Parallel (DP) for optimizers and gradients
4. Pipeline Parallelism (PP) for layer partitioning [2405.07719][2105.13120][2604.26294]

Key recommendations:
- Use DP first when batch allows; introduce SP or TSP to unlock larger context on limited batch size/hardware.
- Always pair SP/TSP with ZeRO-1/2 for optimizer/gradient memory efficiency. Consider ZeRO-3 if memory-bound.
- TSP frees GPUs (vs. TP+SP mesh) for additional DP/PP axes, maximizing overall hardware usage on long contexts [2604.26294][2405.07719].

USP and Loop-2D (e.g., LoongTrain) further exploit hardware topology by adjusting AllToAll vs. Ring trade, mapping high-bandwidth links to sequence or tensor axes [2406.18485][2405.07719].

## 6. Extensions and Related Methods

TSP and its generalizations are applicable beyond conventional transformer models. In tensor network-based sequence models, such as the uniform Matrix Product State (u-MPS), both tensor- and sequence-parallel contraction trees allow $O(\log n)$ depth parallelism with $O(nD^3)$ arithmetic, as opposed to $O(nD^2)$ for the sequential contraction. This forms a Pareto frontier between arithmetic intensity and sequence-level parallelism [2003.01039].

TSP is also extensible to expert-models (MoE), mixtures, and structured sampling; and can be composed with regular-expression conditioned sampling and richer conditional generative algorithms [2003.01039].

## 7. Limitations and Practical Considerations

- **Communication Overhead:** TSP introduces weight-movement in each forward pass (broadcast, ring, or all-gather of parameter shards), with an aggregate cost scaling as the sum of activation and weight terms. 
- **Partitioning and Mesh Constraints:** Folding both axes onto a single dimension limits the factorization choices vs. orthogonal TP+SP, but reduces device count per replica and enables denser intra-node placement [2604.26294].
- **Divisibility Requirements:** Sequence length $S$ and hidden units $h$ must be divisible by the parallel degree $D$ for ideal load balancing [2105.13120]. Imbalances introduce non-ideal communication and compute.
- **Kernel Support:** Efficient TSP depends on kernel fusion (e.g., FlashAttention), communication overlap, and optimized collectives (ROCm/RCCL, NVLink, IB/PCIe hierarchies).
- **Best Practice Tuning:** Hardware-specific tuning (e.g., Double-Ring concurrency in LoongTrain, All2All group placement in USP) is required for maximum MFU.

A plausible implication is that TSP, by subsuming traditional TP and SP into a unified axis, provides a building block for memory-constrained, long-context model training on emerging AI supercomputers [2604.26294][2406.18485][2405.07719][2105.13120][2003.01039].

Source: https://www.emergentmind.com/topics/tensor-and-sequence-parallelism-tsp