Sparsity Roofline Model

• Sparsity Roofline is a visual analytic framework that extends the traditional Roofline model by incorporating index data transfer and sparsity-induced compute/memory barriers.
• It quantifies kernel performance by jointly evaluating sparse arithmetic intensity, floating-point throughput, and memory bandwidth limitations to predict efficiency.
• The model guides hardware-software co-design in neural networks and tensor computations, enabling optimal sparsity configurations without relying on direct kernel benchmarking.

The Sparsity Roofline is a visual and analytic framework developed to assess the theoretical and practical performance bounds of sparse computations on modern hardware, with specific applicability to sparse tensor decompositions, neural networks, and data analytics workloads. It extends the traditional Roofline model by explicitly incorporating the costs of index data transfer, irregular access patterns, and sparsity-induced compute/memory barriers. The model enables practitioners and hardware architects to predict, analyze, and compare the throughput and efficiency of sparse kernels and architectures by jointly characterizing floating-point throughput, memory bandwidth limitations, and network accuracy for sparsity patterns and formats, all without requiring direct kernel benchmarking (Li et al., 2020, Shinn et al., 2023).

1. Sparse Arithmetic Intensity and Index Overhead

The Sparsity Roofline model generalizes arithmetic intensity—originally the ratio of floating-point operations (FLOPs) to total bytes transferred—to sparse kernels by including all index-related I/O. For a sparse tensor kernel, the sparse arithmetic intensity $I_s$ is defined as

$$I_s = \frac{\mathrm{FLOP}}{\mathrm{Bytes_{read}} + \mathrm{Bytes_{write}}}$$

where bytes must be counted for both floating-point values and indices in the selected sparse data format (e.g., COO or HiCOO). In coordinate (COO) format, each nonzero requires one floating-point value plus one index per tensor mode. In hierarchical coordinate (HiCOO) format, block index compression reduces total I/O, yielding 20–40% higher $I_s$ by minimizing per-nonzero index cost, as quantitated for typical tensor algebra kernels (Li et al., 2020).

For sparse neural network layers, the computation extends to block-structured or N:M sparsity patterns, with index overhead calculated per sparse format (e.g., CSR, block, or N:M encoding). The inference traffic includes both data (weights, activations) and index metadata, substantially decreasing arithmetic intensity compared to dense counterparts—unless index compression or regular block sparsity is deployed (Shinn et al., 2023).

2. Machine Ceilings: Compute, Memory, and MLP Boundaries

The performance ceilings in the Sparsity Roofline are set by both hardware peak compute throughput ($F_{\mathrm{peak}}$) and sustainable memory bandwidth ($B_{\mathrm{mem}}$): $$F \le \min \left( F_{\mathrm{peak}},\, I_s \times B_{\mathrm{mem}} \right)$$ $F_{\mathrm{peak}}$ is computed as $M \times C \times W \times f$, where $M$ is the number of sockets, $C$ the core count, $W$ the vector width in flops per cycle, and $f$ the frequency. $B_{\mathrm{mem}}$ is empirically measured (via STREAM or ERT) and reflects achievable DRAM bandwidth under regular strides.

Sparse kernels are further subject to memory-level parallelism (MLP) ceilings. Irregular index access patterns reduce effective bandwidth and cause inelastic stalls—such as TLB and cache misses—so some sparse kernels saturate a lower MLP-limited roofline than the $B_{\mathrm{mem}}$ line, particularly observed in atomic-heavy tensor algebra workloads (e.g., MTTKRP) on GPUs (Li et al., 2020).

3. Roofline Visualization: Operational Intensity and Performance Bounds

In the Sparsity Roofline, each kernel or network is plotted in the $(I_s, F)$ space. Key features:

• Horizontal line: $F_{\mathrm{peak}}$ (compute bound)
• Sloped lines: $B_{\mathrm{mem}}$ and (if relevant) MLP ceiling
• Kernel performance: Each implementation's point must lie below these ceilings; proximity to the sloped line indicates memory bound, while horizontal proximity denotes compute bound.

For sparsity-aware neural network evaluation, the Roofline visualization shifts the axes to accuracy vs. theoretical speedup ($S_{\mathrm{th}}$) over a baseline, enabling practitioners to trace the trade-off between performance and accuracy for different sparsity patterns and densities (Shinn et al., 2023).

4. Use in Sparse Neural Network Modeling

The model readily supports layer-wise and end-to-end inference latency prediction. For each layer: $$T_{\mathrm{dense}} = \max \left( \frac{F_{\mathrm{dense}}}{P_{\mathrm{op}}}, \frac{B_{\mathrm{dense}}}{P_{\mathrm{bw}}} \right)$$

$$T_{\mathrm{sparse}} = \max \left( \frac{F_{\mathrm{sparse}}}{P_{\mathrm{op}}}, \frac{B_{\mathrm{sparse}}}{P_{\mathrm{bw}}} \right)$$

Total network speedup is $$S_{\mathrm{th}} = T_{\mathrm{dense, model}} / T_{\mathrm{sparse, model}}$$. This calculation is exact under idealized conditions (perfect DRAM caching, equally optimized kernels), with deviations arising under suboptimal implementations or hardware-unfriendly sparsity (Shinn et al., 2023).

In block-structured and N:M pruning, accuracy and speedup Pareto frontiers are visualized. For ConvNeXt and Swin transformers, small blocks (2x2 or 4x4) yield up to $2\times$ speedup with minimal accuracy loss, while wider blocks degrade accuracy. For next-generation Tensor Cores supporting alternative N:M patterns, the Roofline predicts throughput gains and accuracy implications without kernel prototyping.

5. Effects of Data Format: COO vs. HiCOO, and Index Compression

Data format choice highly influences $I_s$ via index compression and block structuring. Empirical findings show:

• COO kernels typically exhibit lower $I_s$, placing them deeper in the memory-bound regime and achieving only 20–30% of theoretical memory-bandwidth ceiling due to random index traffic.
• HiCOO, with dense blocking and compressed indices, shifts kernel $I_s$ rightward, enabling 50–70% memory-bandwidth utilization, notably in TEW, TS, TTV operations.
• In atomic-heavy kernels (MTTKRP), GPU MLP ceilings dominate, with HiCOO advantageous for reducing misses but still constrained by atomic-operation bottlenecks (Li et al., 2020).

Kernel Type	$I_s$ (COO)	$I_s$ (HiCOO)
TEW	0.083	0.083
TS	0.125	0.125
TTV	0.167	0.18
TTM	0.5	0.5
MTTKRP	0.25	0.3

HiCOO’s index compression and block structuring are thus instrumental in approaching memory-bound ceilings for practical sparse workloads.

6. Model Assumptions and Limitations

The framework assumes perfect DRAM caching (all counted bytes loaded exactly once) and uniformly optimized kernel implementations. When both dense and sparse kernels access hardware with equal efficiency, theoretical speedup matches measured speedup. Breakdown occurs when:

• Implementation overheads are non-negligible (e.g., poor load balancing, cache tiling inefficiencies)
• Format-specific limitations (e.g., index overhead, atomic operation bottlenecks) dominate
• Hardware-specific constraints (e.g., Tensor Core support, GPU memory-level parallelism) impact realized performance.

A plausible implication is that the model enables early identification of promising format/sparsity configurations for further kernel optimization and hardware support, focusing developer effort where theoretical speedup and accuracy preservation coincide.

7. Applications and Practitioner Workflow

Sparsity Roofline empowers ML practitioners and hardware architects to guide network sparsification and hardware co-design. For any kernel or network, practitioners can:

1. Enumerate operation/layer shapes and sparsity patterns
2. Count FLOPs and transferred bytes (including index costs)
3. Calculate theoretical latency and speedup using only hardware peaks ($P_{\mathrm{op}}$, $P_{\mathrm{bw}}$)
4. Plot accuracy versus speedup to identify optimal Pareto configurations for kernel or hardware investment

This process bypasses kernel benchmarking, relying solely on analytic computation, and is validated on real-world networks and hardware (ConvNeXt, Swin Transformer, Tensor Core-equipped GPUs) (Shinn et al., 2023).

The model’s core utility lies in quantifying the joint trade-offs among sparsity, accuracy, and hardware efficiency, and in exposing the performance impact of index formats, block structures, and architecture-specific limitations. It offers a methodology for hardware-software co-design across scientific computing, deep learning inference, and data analytics leveraging sparse computation.