Symmetric Tensor Parallelism

• Symmetric tensor parallelism is a set of methods that leverages permutation invariance of tensor entries to reduce redundant computation and data movement in large-scale parallel systems.
• The approach uses symmetry-aware data partitioning and combinatorial designs to achieve communication-optimal scheduling and unique arithmetic operations across processors.
• This method improves scalability in scientific computing and machine learning by minimizing data exchange overhead and enhancing the efficiency of symmetric tensor contractions.

Symmetric tensor parallelism refers to the class of computational and data-distribution strategies that fully exploit the permutation invariance of symmetric tensors in high-performance parallel algorithms. By carefully partitioning both data and work while respecting tensor symmetry, these methods minimize communication, reduce redundant computation, and enable efficient scaling across massively parallel architectures. The area encompasses communication-optimal partitioning schemes, tight lower bounds for data movement, symmetry-aware scheduling for tensor contractions, and practical parallel implementations underpinning large-scale scientific computing and machine learning workloads.

1. Principles of Symmetric Tensor Parallelism

Symmetric tensor parallelism is defined by partitioning schemes and computational algorithms that explicitly leverage the invariance of tensor entries under index permutations. In an order-$d$ symmetric tensor $$A \in \mathbb{R}^{n \times n \times \dots \times n}$$, entry %%%%2%%%% remains unchanged under any permutation of indices, so the number of unique elements is reduced from $n^d$ to ${n+d-1 \choose d}$, and similarly, contractions can avoid redundant work. Parallelism in this context requires careful attention to both computation and data movement, as the reduction in arithmetic does not automatically yield minimum communication, and naively mapping symmetric computations to parallel hardware can lead to excessive data transfer (Solomonik et al., 2017).

Symmetric parallel algorithms must assign unique arithmetic operations (e.g., entrywise products $A_{i_1\dots i_d} \prod_k x_{i_k}$ in the tensor-times-same-vector kernel) such that each operation, respecting atomicity, is performed on exactly one processor, and all necessary inputs reside locally.

2. Communication Lower Bounds for Symmetric Tensor Contractions

Communication—both in terms of bandwidth (total words exchanged) and latency (number of messages)—is the core bottleneck in parallel tensor algorithms. For symmetric tensor contractions, the trade-off between reduced arithmetic complexity and minimum required data movement is governed by geometric and algebraic lower bounds. The foundational result for the symmetric tensor-times-same-vector (STTSV) operation on a fully symmetric $3$-tensor $A \in \mathbb{R}^{n \times n \times n}$—computing $y_i = \sum_{j,k} a_{ijk} x_j x_k$—establishes the following per-processor bandwidth lower bound for $P$ processors:

$\Omega( n P^{-1/3} )$

This result follows from extending classic geometric inequalities (Loomis–Whitney) to symmetric computations, showing that to perform its assigned subset of atomic products, each processor must access its local subset of $A$, $x$, and $y$ entries, and any further needed data requires communication (Daas et al., 18 Jun 2025). The sharpness of the bound is established by matching it with an explicit algorithm (see Section 3).

In the broader bilinear-algorithm framework, such lower bounds are determined by the expansion characteristics of the computation’s dependency graph, quantified via so-called expansion functions $E_F(d_A, d_B, d_C)$ that bound the number of computed products covered by subsets of inputs and outputs. Fully symmetric contractions may decrease floating-point operation counts by $d!$ but increase the exponent for minimum communication relative to nonsymmetric algorithms (Solomonik et al., 2017).

3. Communication-Optimal Symmetric Data Partitioning and Algorithms

To achieve the optimal lower bound, data must be partitioned in a symmetry-aware way. For the order-3 case, the strict lower tetrahedron $$\mathcal{T}_{\text{strict}} = \{(i,j,k): i > j > k\}$$, containing all unique tensor entries, is divided into "tetrahedral blocks" using combinatorial designs, specifically Steiner systems $S(m, r, 3)$. Each processor receives:

• The unique off-diagonal block $TB_3(R_p)$ for its assigned subset $R_p$.
• A fixed number of non-central diagonal blocks.
• At most one central diagonal block.

The vector inputs $x$ are partitioned and distributed such that each processor initially holds only its local parts, and gathers the missing blocks via schedule-optimized all-to-all exchanges within relevant processor groups. After local computation, the $y$ vector is reduced via equally structured exchanges. No tensor data moves in this protocol; only vectors are communicated (Daas et al., 18 Jun 2025).

Algorithmically, a processor performs local STTSV evaluations on its blocks, using the sparsity and symmetry of the blocks to reduce computation and storage. The full message schedule for $x$ and $y$ is decomposed into matchings of a bipartite communication graph, ensuring pairwise transfers in a minimal number of rounds, thereby precisely meeting the theoretical lower bound of $2 n P^{-1/3} - 2 n / P$ communicated words per processor.

For higher orders $d > 3$, symmetric extensions of the Loomis–Whitney inequality yield the communication bound $\Omega(n^{d/3}/P^{1/3})$, but practical block partitions require particular combinatorial constructions (e.g., higher-order Steiner systems) that are currently unavailable for many cases.

4. Symmetric Tensor Parallelism in HPC and Machine Learning

Symmetric tensor parallelism underlies both scientific algorithms, such as the higher-order power method (HO-PM) for computing tensor eigenpairs, and gradient steps in symmetric CP and Tucker decompositions. In these applications, symmetric tensor contractions are repeatedly invoked, so even a constant-factor improvement in communication has significant impact on scaling.

In deep learning, "symmetric" tensor parallelism refers to slicing both row and column dimensions and reducing along an additional "depth" dimension, as implemented in 3D tensor-parallel runtimes like Tesseract (Wang et al., 2021). For a weight matrix $$W \in \mathbb{R}^{d_\text{model} \times d_\text{ff}}$$, arranging $p$ processors in a $p_1 \times p_2 \times p_3$ mesh and partitioning as:

• $p_1$-way split along rows,
• $p_2$-way split along columns,
• $p_3$-way replication and reduction across the depth axis,

yields per-GPU memory and communication load of $|W|/p^{2/3}$ and $3 \alpha + \beta |W|/p^{2/3}$, in contrast to $|W|/p^{1/2}$ for 2D (SUMMA) or $|W|/p$ for 1D (Megatron-LM). This symmetric 3D partitioning achieves higher parallel efficiency and empirical speedups in both strong and weak scaling regimes (Wang et al., 2021).

5. Communication-Complexity Trade-offs and Algorithm Selection

The fundamental trade-off in symmetric tensor parallelism is between reduction in arithmetic and increases in I/O complexity. Using the bilinear (U,V,W) framework (Solomonik et al., 2017), contraction algorithms can be categorized as:

Algorithm family	Arithmetic cost	Communication lower bound
Unpacked (Upsilon)	$n^\omega$	$\Omega((n^\omega/P)^{2/3})$
Direct-packed (Psi)	$n^\omega/(s!t!)$	$\Omega((n^\omega/P)^{\max(s,t,v)/\omega})$
Sym-preserving (Phi)	$n^\omega/\omega!$	$\Omega((n^\omega/P)^{\kappa/\omega})$

where $\omega = s+t+v$ is the total tensor order, and $\kappa$ is the maximum mode sum involved in the contraction. For certain (symmetric) cases, especially at higher order, the exponent may rise from $2/3$ to $3/4$ or more, necessitating greater care to avoid excess communication.

Proper algorithm selection for parallel symmetric tensor contractions thus depends acutely on the problem structure, available memory, and machine topology. Extreme symmetry reduction is beneficial only when additional communication requirements remain feasible, or when system memory and processor count allow (Solomonik et al., 2017).

6. Applications Beyond Dense Tensors: Sparse and Hypergraph Computations

Sparse symmetric tensor contractions arise, for example, in hypergraph eigenvector centrality and related analytics. Efficient parallelization in these settings leverages custom symmetric storage formats, such as Compound Compressed Sparse Symmetric (CCSS), which merges hyperedge blowups into a forest-of-tries and avoids explicit enumeration of all symmetric nonzeros. Shared-memory parallel algorithms (e.g., ParaGiambu) proceed via thread-local depth-first search with memoization, assigning CCSS root-trees to threads and atomic accumulation of output (Shivakumar et al., 2023). Compression factors up to $26.4\times$ versus coordinate lists, and multi-core speedups of $2-54\times$ over baselines, have been demonstrated.

Generalizations to multi-node (MPI) settings are possible by partitioning CCSS forests across ranks and using all-reduce for border accumulations.

7. Future Directions and Partial Generalizations

While symmetric block-partitioning in three dimensions (order-3 tensors) and certain hypergraph/CP decompositions is communication-optimal and practical (due to the existence of Steiner systems $S(m,r,3)$), general extension to higher-order tensors relies on the existence of higher-dimensional combinatorial designs, which are only known for very restricted parameters. For $d > 3$, practical symmetric partitioning may require relaxing symmetry, or employing approximate/composite block structures. A plausible implication is that future work will focus on heuristics or partial-symmetry-preserving algorithms for these higher-order cases (Daas et al., 18 Jun 2025).

Additionally, in deep learning, symmetric 3D parallel techniques may be combined with other forms of parallelism (pipeline, data, expert/multiple weight partitioning) to further reduce bottlenecks as model sizes and cluster scales increase (Wang et al., 2021). Emerging architectures and nonuniform memory systems will likely motivate new symmetric scheduling strategies.

---

Key references:

(Daas et al., 18 Jun 2025, Shivakumar et al., 2023, Solomonik et al., 2017, Wang et al., 2021)