PASTA: Sparse Tensor Benchmark Suite

• PASTA is a benchmark suite that evaluates sparse tensor algorithms on single- and multi-core CPUs using optimized C/OpenMP implementations.
• It features key workloads like TEW, TS, TTV, TTM, and MTTKRP, highlighting challenges such as irregular memory access and synchronization overhead.
• PASTA serves as a reference for comparing machine performance and guiding hardware design using real-world tensors from collections like FROSTT.

PASTA, short for “PArallel Sparse Tensor Algorithm benchmark suite,” is a benchmark suite for sparse tensor algorithms on single- and multi-core CPUs. It was introduced to provide representative computational workloads for evaluating computer systems and to offer insight into how computer architecture and runtime behavior interact with sparse tensor operations. The suite is implemented in optimized sequential and parallel C/OpenMP code on the coordinate (COO) format and is evaluated on large, real-world sparse tensors drawn primarily from the FROSTT collection and related sources. The authors describe it as the first benchmark suite focused specifically on the sparse tensor world, with workloads derived from kernels that dominate CP decomposition, Tucker decomposition, tensor power methods, and tensor-network computations used in machine learning, quantum chemistry, healthcare analytics, social network analysis, recommendation systems, data mining, and signal processing (Li et al., 2019).

1. Motivation and scope

PASTA is motivated by the increasing use of tensor methods in applications where data are naturally multiway, such as user–item–time, subject–brain-voxel–stimulus, or IP–IP–time arrays. In these settings, the data are typically sparse, and scalable sparse tensor algorithms become critical both for performance and for interpretability of the resulting models (Li et al., 2019).

The benchmark targets two audiences. For application users, it provides representative workloads for comparing machines and libraries on sparse tensor computations. For system and architecture designers, it exposes how architectural features interact with sparse tensor kernels and thus provides guidance for better use of existing systems and for future design (Li et al., 2019).

The suite is built around several difficulties that are characteristic of sparse tensors rather than ordinary sparse matrices. These include the curse of dimensionality, mode orientation, the cost of tensor–matrix transformations such as matricization and tensorization, irregular dimensions and nonzero patterns, and the need to support arbitrary tensor orders with different loop structures and memory behaviors. This makes sparse tensor workloads irregular, low arithmetic-intensity, and highly sensitive to memory behavior, which is precisely why PASTA treats them as informative benchmarks (Li et al., 2019).

2. Core benchmark workloads

PASTA collects the sparse tensor kernels that dominate the computational cost of major tensor methods. CP decomposition is centered on MTTKRP, Tucker decomposition on repeated TTM, and tensor power methods on TTV; tensor-network models such as Tensor Train and Hierarchical Tucker repeatedly use TS, TTM, and tensor–tensor operations beyond the scope of this paper (Li et al., 2019).

Workload	Representative operation	Principal behavior
TEW	element-wise tensor addition, subtraction, multiplication, division	merge-like or streaming
TS	tensor with scalar	embarrassingly parallel
TTV	tensor-times-vector	fiber reduction
TTM	tensor-times-matrix	fiber expansion and rank-$R$ updates
MTTKRP	matricized tensor times Khatri–Rao product	CP bottleneck with write conflicts

Sparse Tensor Element-Wise operations, denoted TEW, include addition, subtraction, multiplication, and division between tensors of the same order. For addition, with $$\mathcal{X}, \mathcal{Y} \in \mathbb{R}^{I_1 \times \dots \times I_N}$$,

$$\mathcal{Z} = \mathcal{X} .+ \mathcal{Y}, \qquad z_{i_1 \dots i_N} = x_{i_1 \dots i_N} + y_{i_1 \dots i_N}.$$

PASTA distinguishes an equal-pattern case and a general case. In TEW-eq, where the tensors have equal shape and identical nonzero pattern, the output tensor is pre-allocated and the algorithm becomes a simple parallel loop over nonzeros. In the general case, the tensors are sorted in fixed lexicographic order and merged as two nonzero streams, with dynamic appends because the output sparsity is not known a priori. The latter makes TEW a pointer-chasing, low-intensity kernel with irregular control flow and dynamic allocation (Li et al., 2019).

Tensor–Scalar operations apply a scalar $s$ to every nonzero. For multiplication,

$$\mathcal{Y} = \mathcal{X} \times s, \qquad y_{i_1 \cdots i_N} = s \times x_{i_1 \cdots i_N}.$$

PASTA implements TS by pre-allocating an output tensor with the same nonzero pattern and streaming through nonzeros. Because there is no index transformation, TS is a prototypical bandwidth-bound kernel (Li et al., 2019).

Tensor–Times–Vector reduces tensor order by contracting one mode: $$\mathcal{Y} = \mathcal{X} \times_n \mathbf{v}, \qquad y_{i_1 \cdots i_{n-1} i_{n+1} \cdots i_N} = \sum_{i_n=1}^{I_n} x_{i_1 \cdots i_n \cdots i_N} \, v_{i_n}.$$ The implementation groups nonzeros into mode-$n$ fibers and constructs $\mathit{nfibs}$, the number of fibers, together with $\mathit{fptr}$, the array of fiber start offsets in COO order. Each fiber produces one output nonzero, and parallelization proceeds over independent fibers (Li et al., 2019).

Tensor–Times–Matrix replaces the contracted index by a rank index: $$\mathcal{Y} = \mathcal{X} \times_n U, \qquad y_{i_1 \cdots i_{n-1} r i_{n+1} \cdots i_N} = \sum_{i_n=1}^{I_n} x_{i_1 \cdots i_n \cdots i_N} \, u_{i_n r}.$$ Although in dense linear algebra this corresponds to $$\mathcal{X}, \mathcal{Y} \in \mathbb{R}^{I_1 \times \dots \times I_N}$$0, PASTA explicitly avoids matricizing and tensorizing because the associated copying overhead is large and especially problematic for sparse tensors. Its TTM algorithm also operates fiberwise, allocating $$\mathcal{X}, \mathcal{Y} \in \mathbb{R}^{I_1 \times \dots \times I_N}$$1 output nonzeros and performing a length-$$\mathcal{X}, \mathcal{Y} \in \mathbb{R}^{I_1 \times \dots \times I_N}$$2 rank-1 update for each nonzero in a fiber (Li et al., 2019).

Matricized Tensor Times Khatri–Rao Product is the core bottleneck in CP decomposition: $$\mathcal{X}, \mathcal{Y} \in \mathbb{R}^{I_1 \times \dots \times I_N}$$3 PASTA does not form the Khatri–Rao product explicitly. Instead, it iterates directly over nonzeros. For a third-order tensor in mode 1, the update is

$$\mathcal{X}, \mathcal{Y} \in \mathbb{R}^{I_1 \times \dots \times I_N}$$4

Because multiple nonzeros may update the same row and column of the output factor matrix, the benchmark implements two parallel strategies: atomic updates and privatization followed by reduction (Li et al., 2019).

In addition to these kernels, PASTA includes tensor–matrix transformations and lexicographic sorting of COO tensors, both of which can themselves stress memory hierarchies (Li et al., 2019).

3. COO representation and analytical performance model

All workloads are implemented on a single sparse tensor format, coordinate format. In COO, nonzeros are stored as parallel arrays: $$\mathcal{X}, \mathcal{Y} \in \mathbb{R}^{I_1 \times \dots \times I_N}$$5, containing tuples $$\mathcal{X}, \mathcal{Y} \in \mathbb{R}^{I_1 \times \dots \times I_N}$$6, and $$\mathcal{X}, \mathcal{Y} \in \mathbb{R}^{I_1 \times \dots \times I_N}$$7, containing the floating-point values. For a 3-way tensor, this is conceptually three index arrays plus one value array (Li et al., 2019).

This choice is deliberately mode-neutral. Any mode ordering can be supported by sorting, and arbitrary kernel implementations can be built without mode-specific data structure complexity. That simplicity supports the benchmark’s goals of diversity and extensibility. The cost is a relatively large memory footprint and irregular memory access: every nonzero repeats all indices, and kernels such as TTV and TTM require preprocessing to recover fiber structure (Li et al., 2019).

For a third-order tensor with 32-bit indices and 32-bit single-precision values, each nonzero costs $$\mathcal{X}, \mathcal{Y} \in \mathbb{R}^{I_1 \times \dots \times I_N}$$8 bytes in COO. The paper’s analytical model, based on a cubical tensor $$\mathcal{X}, \mathcal{Y} \in \mathbb{R}^{I_1 \times \dots \times I_N}$$9, shows that all benchmark kernels have arithmetic intensity well below $$\mathcal{Z} = \mathcal{X} .+ \mathcal{Y}, \qquad z_{i_1 \dots i_N} = x_{i_1 \dots i_N} + y_{i_1 \dots i_N}.$$0 flop/byte. Specifically, TEW has arithmetic intensity $$\mathcal{Z} = \mathcal{X} .+ \mathcal{Y}, \qquad z_{i_1 \dots i_N} = x_{i_1 \dots i_N} + y_{i_1 \dots i_N}.$$1, TS $$\mathcal{Z} = \mathcal{X} .+ \mathcal{Y}, \qquad z_{i_1 \dots i_N} = x_{i_1 \dots i_N} + y_{i_1 \dots i_N}.$$2, TTV $$\mathcal{Z} = \mathcal{X} .+ \mathcal{Y}, \qquad z_{i_1 \dots i_N} = x_{i_1 \dots i_N} + y_{i_1 \dots i_N}.$$3, TTM $$\mathcal{Z} = \mathcal{X} .+ \mathcal{Y}, \qquad z_{i_1 \dots i_N} = x_{i_1 \dots i_N} + y_{i_1 \dots i_N}.$$4, and MTTKRP $$\mathcal{Z} = \mathcal{X} .+ \mathcal{Y}, \qquad z_{i_1 \dots i_N} = x_{i_1 \dots i_N} + y_{i_1 \dots i_N}.$$5. The corresponding storage and memory traffic formulas make the same point: sparse tensor kernels are overwhelmingly memory-bandwidth bound and highly sensitive to cache behavior, prefetching, and memory hierarchy design (Li et al., 2019).

This analytical characterization also distinguishes the kernels. TEW and TS are the most bandwidth-bound. TTV gains locality from fiberwise reduction. TTM has the highest arithmetic intensity because rows of the factor matrix are reused within a fiber. MTTKRP sits between TTM and the lighter kernels, but its scalability is complicated by synchronization or reduction overheads (Li et al., 2019).

4. Parallelization strategy on multi-core CPUs

PASTA targets single-node multi-core CPUs through OpenMP. For the simplest kernels, parallelization is straightforward. TEW-eq and TS use a parallel loop over nonzero index $$\mathcal{Z} = \mathcal{X} .+ \mathcal{Y}, \qquad z_{i_1 \dots i_N} = x_{i_1 \dots i_N} + y_{i_1 \dots i_N}.$$6; each iteration reads input indices and values and writes to a distinct output location, so there are no dependencies (Li et al., 2019).

TTV and TTM are parallelized over fibers. The preprocessing arrays $$\mathcal{Z} = \mathcal{X} .+ \mathcal{Y}, \qquad z_{i_1 \dots i_N} = x_{i_1 \dots i_N} + y_{i_1 \dots i_N}.$$7 and $$\mathcal{Z} = \mathcal{X} .+ \mathcal{Y}, \qquad z_{i_1 \dots i_N} = x_{i_1 \dots i_N} + y_{i_1 \dots i_N}.$$8 give the bounds of each fiber in COO order, so each thread can process a disjoint set of fibers and write to a disjoint output region. This is synchronization-free and exposes coarse-grain parallelism (Li et al., 2019).

General TEW is more difficult. Because the output structure is not known in advance, naive parallelization is unsafe. PASTA partitions one tensor by slices while keeping each slice intact, then partitions the other tensor according to the same slice ranges. Each thread runs the sequential merge on its local subtensors and writes to a thread-local output buffer. This guarantees disjoint indices between partitions, but it also exposes a trade-off between synchronization-free execution and load imbalance, since a heavy slice cannot be split (Li et al., 2019).

MTTKRP is the most synchronization-sensitive kernel in the suite. Parallelization over nonzeros creates write conflicts when different nonzeros contribute to the same output row and rank. Atomics preserve correctness but often reduce performance. Privatization eliminates fine-grain synchronization by giving each thread a private dense $$\mathcal{Z} = \mathcal{X} .+ \mathcal{Y}, \qquad z_{i_1 \dots i_N} = x_{i_1 \dots i_N} + y_{i_1 \dots i_N}.$$9 buffer, followed by reduction, but it increases memory footprint and memory traffic (Li et al., 2019).

The paper explicitly notes that memory bandwidth and cross-socket effects limit scaling, and it identifies NUMA-aware techniques as a natural extension. This suggests that PASTA is intended not only as an application benchmark but also as a probe of runtime scheduling, synchronization cost, and memory locality in modern CPU systems (Li et al., 2019).

5. Experimental setup, datasets, and measured behavior

The evaluation platform is a dual-socket Intel Xeon E5-2698 v3 system based on the Haswell microarchitecture, with 32 physical cores total at $s$0 GHz, $s$1 KiB L1 data cache per core, and $s$2 GiB of system memory. The code is written in C with OpenMP and compiled with Intel icc 18.0.1. Unless otherwise stated, parallel runs use $s$3 threads. For TTM and MTTKRP, the rank is fixed at $s$4, consistent with low-rank decompositions where $s$5 (Li et al., 2019).

The datasets span third- and fourth-order tensors with widely varying dimensions and densities. Third-order examples include $s$6, $s$7, $s$8, $s$9, $$\mathcal{Y} = \mathcal{X} \times s, \qquad y_{i_1 \cdots i_N} = s \times x_{i_1 \cdots i_N}.$$0, $$\mathcal{Y} = \mathcal{X} \times s, \qquad y_{i_1 \cdots i_N} = s \times x_{i_1 \cdots i_N}.$$1, $$\mathcal{Y} = \mathcal{X} \times s, \qquad y_{i_1 \cdots i_N} = s \times x_{i_1 \cdots i_N}.$$2, and $$\mathcal{Y} = \mathcal{X} \times s, \qquad y_{i_1 \cdots i_N} = s \times x_{i_1 \cdots i_N}.$$3. Fourth-order examples include $$\mathcal{Y} = \mathcal{X} \times s, \qquad y_{i_1 \cdots i_N} = s \times x_{i_1 \cdots i_N}.$$4, $$\mathcal{Y} = \mathcal{X} \times s, \qquad y_{i_1 \cdots i_N} = s \times x_{i_1 \cdots i_N}.$$5, $$\mathcal{Y} = \mathcal{X} \times s, \qquad y_{i_1 \cdots i_N} = s \times x_{i_1 \cdots i_N}.$$6, $$\mathcal{Y} = \mathcal{X} \times s, \qquad y_{i_1 \cdots i_N} = s \times x_{i_1 \cdots i_N}.$$7, and $$\mathcal{Y} = \mathcal{X} \times s, \qquad y_{i_1 \cdots i_N} = s \times x_{i_1 \cdots i_N}.$$8. They range from $$\mathcal{Y} = \mathcal{X} \times s, \qquad y_{i_1 \cdots i_N} = s \times x_{i_1 \cdots i_N}.$$9M nonzeros in $$\mathcal{Y} = \mathcal{X} \times_n \mathbf{v}, \qquad y_{i_1 \cdots i_{n-1} i_{n+1} \cdots i_N} = \sum_{i_n=1}^{I_n} x_{i_1 \cdots i_n \cdots i_N} \, v_{i_n}.$$0 to $$\mathcal{Y} = \mathcal{X} \times_n \mathbf{v}, \qquad y_{i_1 \cdots i_{n-1} i_{n+1} \cdots i_N} = \sum_{i_n=1}^{I_n} x_{i_1 \cdots i_n \cdots i_N} \, v_{i_n}.$$1M in $$\mathcal{Y} = \mathcal{X} \times_n \mathbf{v}, \qquad y_{i_1 \cdots i_{n-1} i_{n+1} \cdots i_N} = \sum_{i_n=1}^{I_n} x_{i_1 \cdots i_n \cdots i_N} \, v_{i_n}.$$2, and from density $$\mathcal{Y} = \mathcal{X} \times_n \mathbf{v}, \qquad y_{i_1 \cdots i_{n-1} i_{n+1} \cdots i_N} = \sum_{i_n=1}^{I_n} x_{i_1 \cdots i_n \cdots i_N} \, v_{i_n}.$$3 in $$\mathcal{Y} = \mathcal{X} \times_n \mathbf{v}, \qquad y_{i_1 \cdots i_{n-1} i_{n+1} \cdots i_N} = \sum_{i_n=1}^{I_n} x_{i_1 \cdots i_n \cdots i_N} \, v_{i_n}.$$4 down to $$\mathcal{Y} = \mathcal{X} \times_n \mathbf{v}, \qquad y_{i_1 \cdots i_{n-1} i_{n+1} \cdots i_N} = \sum_{i_n=1}^{I_n} x_{i_1 \cdots i_n \cdots i_N} \, v_{i_n}.$$5 in $$\mathcal{Y} = \mathcal{X} \times_n \mathbf{v}, \qquad y_{i_1 \cdots i_{n-1} i_{n+1} \cdots i_N} = \sum_{i_n=1}^{I_n} x_{i_1 \cdots i_n \cdots i_N} \, v_{i_n}.$$6 (Li et al., 2019).

The principal metrics are wall-clock time and speedup. The measured results follow the analytical model closely. Parallel TEW-eq achieves speedups of roughly $$\mathcal{Y} = \mathcal{X} \times_n \mathbf{v}, \qquad y_{i_1 \cdots i_{n-1} i_{n+1} \cdots i_N} = \sum_{i_n=1}^{I_n} x_{i_1 \cdots i_n \cdots i_N} \, v_{i_n}.$$7 to $$\mathcal{Y} = \mathcal{X} \times_n \mathbf{v}, \qquad y_{i_1 \cdots i_{n-1} i_{n+1} \cdots i_N} = \sum_{i_n=1}^{I_n} x_{i_1 \cdots i_n \cdots i_N} \, v_{i_n}.$$8 on 32 cores, while TS multiplication reaches $$\mathcal{Y} = \mathcal{X} \times_n \mathbf{v}, \qquad y_{i_1 \cdots i_{n-1} i_{n+1} \cdots i_N} = \sum_{i_n=1}^{I_n} x_{i_1 \cdots i_n \cdots i_N} \, v_{i_n}.$$9 to $n$0. General TEW is much harder to scale, giving about $n$1 to $n$2, because dynamic append operations, slice-based partitioning, and irregular sparsity patterns limit both locality and load balance (Li et al., 2019).

TTV scales better, with reported speedups of $n$3 to $n$4, consistent with its higher arithmetic intensity and better fiber locality. TTM performs best, at roughly $n$5 to $n$6, because the extra rank-$n$7 work increases reuse and amortizes memory latency. Its sequential runtime is $n$8–$n$9 slower than TTV, but that additional computation yields more parallel headroom (Li et al., 2019).

For MTTKRP, privatization gives speedups up to $\mathit{nfibs}$0, but performance is less stable. On some extreme tensors, including $\mathit{nfibs}$1, the cost of allocating and reducing large thread-local buffers can dominate and produce slight slowdowns. Atomics can occasionally be better in such skewed cases, but are generally inferior to privatization on most tensors (Li et al., 2019).

6. Role in the sparse tensor ecosystem

PASTA fills a gap between general-purpose benchmarks and specialized tensor libraries. Classic benchmark suites such as SPEC, PARSEC, Rodinia, MediaBench, and EEMBC do not focus on sparse tensor computations. Tensor Toolbox and TensorLab provide rich tensor functionality, but they are MATLAB-based and are not benchmark suites designed for architecture comparison. High-performance libraries such as SPLATT, Cyclops Tensor Framework, DFacTo, GigaTensor, HyperTensor, and GenTen provide optimized implementations of particular methods or operations, often centered on CP decomposition and MTTKRP, but they do not provide a broad, unified suite of kernels, datasets, and interfaces expressly for benchmarking (Li et al., 2019).

PASTA is therefore deliberately constructed as a benchmark: diverse kernels, real datasets, well-specified algorithms, and a single CPU baseline implementation in C/OpenMP. In practical use, the workflow is direct. One obtains the code, builds it with a C compiler supporting OpenMP, chooses workloads such as TEW, TS, TTV, TTM, or MTTKRP, selects included datasets or provides new COO tensors, configures thread count, rank $\mathit{nfibs}$2, and operating modes, and optionally selects among available strategies such as atomics or privatization in MTTKRP. The suite then reports execution time in sequential and parallel modes, from which speedups can be derived (Li et al., 2019).

The authors also identify clear extension paths. COO provides a simple, robust baseline, but formats such as compressed sparse fiber and hierarchical COO are named as future additions that could reduce index storage and improve locality while introducing mode-orientation trade-offs. Likewise, the CPU implementation is presented as a baseline for later extension to GPUs, FPGAs, and distributed systems (Li et al., 2019).

In this sense, PASTA is best understood as a reference benchmark for sparse tensor computation rather than merely a collection of kernels. It turns the dominant operations of modern sparse tensor methods into a systematic workload suite that exposes memory pressure, irregularity, synchronization cost, and limited arithmetic intensity in a way that conventional HPC benchmarks do not (Li et al., 2019).