SpArch: Efficient Architecture for Sparse Matrix Multiplication (2002.08947v1)

Abstract: Generalized Sparse Matrix-Matrix Multiplication (SpGEMM) is a ubiquitous task in various engineering and scientific applications. However, inner product based SpGENN introduces redundant input fetches for mismatched nonzero operands, while outer product based approach suffers from poor output locality due to numerous partial product matrices. Inefficiency in the reuse of either inputs or outputs data leads to extensive and expensive DRAM access. To address this problem, this paper proposes an efficient sparse matrix multiplication accelerator architecture, SpArch, which jointly optimizes the data locality for both input and output matrices. We first design a highly parallelized streaming-based merger to pipeline the multiply and merge stage of partial matrices so that partial matrices are merged on chip immediately after produced. We then propose a condensed matrix representation that reduces the number of partial matrices by three orders of magnitude and thus reduces DRAM access by 5.4x. We further develop a Huffman tree scheduler to improve the scalability of the merger for larger sparse matrices, which reduces the DRAM access by another 1.8x. We also resolve the increased input matrix read induced by the new representation using a row prefetcher with near-optimal buffer replacement policy, further reducing the DRAM access by 1.5x. Evaluated on 20 benchmarks, SpArch reduces the total DRAM access by 2.8x over previous state-of-the-art. On average, SpArch achieves 4x, 19x, 18x, 17x, 1285x speedup and 6x, 164x, 435x, 307x, 62x energy savings over OuterSPACE, MKL, cuSPARSE, CUSP, and ARM Armadillo, respectively.

• The paper introduces SpArch, a specialized hardware architecture that optimizes Sparse Generalized Matrix-Matrix Multiplication (SpGEMM) through novel data handling techniques.
• Key innovations in SpArch, such as a streaming merger and condensed matrix representation, dramatically reduce DRAM access compared to state-of-the-art solutions.
• SpArch achieves significant performance gains, up to 1285× speedup and 435× energy savings, over popular libraries like MKL and cuSPARSE.

SpArch: A Specialized Architecture for Efficient Sparse Matrix Multiplication

The paper "SpArch: Efficient Architecture for Sparse Matrix Multiplication" investigates the development of a dedicated architecture aimed at enhancing the performance of Sparse Generalized Matrix-Matrix Multiplication (SpGEMM). SpGEMM is a fundamental operation that frequently arises in scientific computing, machine learning, and various engineering applications. Traditional computing platforms such as CPUs and GPUs struggle with SpGEMM due to the irregular memory access patterns and poor data locality stemming from the sparse nature of the matrices involved.

The authors introduce SpArch, an accelerator that jointly optimizes data reuse for both input and output matrices. The architecture employs an outer product approach, a choice driven by its potential for excellent input data reuse. Though the outer product method inherently encourages superb input data retention, it generates multiple partial product matrices that undermine efficiency owing to the increased output data handling and associated DRAM access.

Key Innovations of SpArch

1. Streaming-based Merger Pipeline: SpArch incorporates a parallelized streaming-based merger to integrate pipeline stages for multiplying and merging matrices. By immediately merging partial matrices on-chip as they are produced, this method substantially enhances the throughput and limits intermediate data traffic to DRAM.
2. Condensed Matrix Representation: The authors propose a pioneering condensed matrix representation technique to reduce the number of partial matrices generated during computation, achieving a reduction by three orders of magnitude. This contributes to a 5.4× reduction in DRAM access.
3. Huffman Tree Scheduler: A Huffman tree-based scheduling methodology efficiently determines the merging order of partial matrices to minimize DRAM access. This improvement results in a further 1.8× reduction in data transfer demand on DRAM compared to previous methods.
4. Row Prefetcher with Optimal Buffer Replacement: SpArch resolves the increased input matrix read requirements induced by the condensed representation using a novel row prefetcher. It optimizes buffer replacement, resulting in a 1.5× decrease in DRAM access for input data.

Performance and Results

The paper evaluates SpArch's performance across 20 benchmarks, demonstrating an average 2.8× reduction in total DRAM access relative to contemporary state-of-the-art solutions such as OuterSPACE. The architecture achieves significant speedups of up to 1285× and energy savings reaching 435× when benchmarked against popular libraries like MKL, cuSPARSE, CUSP, and Armadillo.

Theoretical and Practical Implications

SpArch exemplifies a promising advancement in domain-specific architectures for sparse computations. The detailed exploration of data reuse optimization provides insights into how specialized hardware can outperform traditional approaches by orders of magnitude in both speed and energy efficiency. In terms of practical implications, the architecture is particularly suited for scientific applications requiring high-performance sparse linear algebra operations, especially in areas with memory-bound operations.

Future Directions

Looking ahead, the principles outlined in SpArch could extend to other sparse computational kernels, further enhancing performance in domains such as machine learning inference for sparse neural networks and large-scale scientific modeling. Additionally, the amalgamation of SpArch's techniques with emerging memory systems and other domain-specific hardware could continue to bridge the gap between software inefficiencies and hardware capabilities in high-performance computing environments.

In conclusion, SpArch not only showcases the power of thoughtfully designed architectural solutions in addressing specific computational challenges but also underscores the necessity of integrating specialized data handling techniques to fully harness the potential of current and future hardware advancements in the field of sparse matrix computations.