Parallel Sparse Matrix-Matrix Multiplication and Indexing: Implementation and Experiments (1109.3739v2)

Abstract: Generalized sparse matrix-matrix multiplication (or SpGEMM) is a key primitive for many high performance graph algorithms as well as for some linear solvers, such as algebraic multigrid. Here we show that SpGEMM also yields efficient algorithms for general sparse-matrix indexing in distributed memory, provided that the underlying SpGEMM implementation is sufficiently flexible and scalable. We demonstrate that our parallel SpGEMM methods, which use two-dimensional block data distributions with serial hypersparse kernels, are indeed highly flexible, scalable, and memory-efficient in the general case. This algorithm is the first to yield increasing speedup on an unbounded number of processors; our experiments show scaling up to thousands of processors in a variety of test scenarios.

• The paper evaluates scalable parallel implementations of sparse matrix-matrix multiplication (SpGEMM) and indexing (SpRef), introducing a novel approach that uses SpGEMM to solve the SpRef problem efficiently on distributed memory systems.
• A key methodological contribution is a distributive SpGEMM algorithm generalized for diverse use cases with a 2D block distribution and serial hypersparse kernels, achieving scalable speedup over thousands of processors.
• These methods offer significant benefits for large-scale computational problems in fields like graph algorithms, linear solvers, and network analysis, improving performance and setting the foundation for future communication-avoiding algorithms.

Parallel Sparse Matrix-Matrix Multiplication and Indexing: Implementation and Experiments

The paper presented in this paper evaluates scalable parallel implementations of sparse matrix-matrix multiplication (SpGEMM) and generalized sparse matrix indexing (SpRef) in the context of distributed memory systems. Sparse matrices are instrumental in numerous computational applications, particularly in graph algorithms and certain linear solvers like algebraic multigrid. This paper introduces advancements in SpGEMM and leverages its efficiency to solve the complex problem of sparse matrix indexing, establishing a novel and proficient approach to the SpRef operation.

The primary contributions of this research are threefold:

1. A straightforward and effective implementation of SpRef grounded in SpGEMM.
2. A distributive SpGEMM algorithm generalized for diverse use cases and adaptable processor layouts.
3. Empirical evaluations yielding extensive performance data for both SpGEMM and SpRef.

The parallel SpGEMM methodology uses a two-dimensional block data distribution, paired with serial hypersparse kernels to enhance flexibility, scalability, and memory efficiency. Remarkably, the algorithm uniquely achieves increasing speedup that remains scalable over thousands of processors, a significant improvement over previous methods constrained by the number of processors.

Technical Achievements and Implications

1. Algorithmic Framework: The research identifies SpGEMM as not only a critical primitive for traditional graph algorithms and linear solvers, but also a building block for more complex matrix operations like SpRef. The paper details SpGEMM's potential to streamline sparse matrix indexing, employing a 2D grid decomposition which dramatically outperforms conventional one-dimensional alternatives, both in terms of computational and communication efficiency.
2. Complexity and Efficiency: The theoretical performance is discussed in-depth, with the algorithm exhibiting a computational complexity that is well-optimized for the hypersparsity of submatrices involved in a 2D partition. The innovative use of DCSC (Doubly Compressed Sparse Column) format further underscores this efficiency by reducing memory overhead drastically compared to conventional CSC (Compressed Sparse Column) structures.
3. Parallel Experiments and Scalability: Through rigorous experimental validation, the paper confirms the practicality of these methods. Tests conducted on large-scale parallel computing platforms corroborate the anticipated scalability, indicating near-linear performance at initial concurrency and transitioning smoothly to stable, scalable behavior thereafter. This approach benefits a range of matrix operations that were previously limited by prohibitive scalability and memory costs.
4. Applications: The practical implications of these findings span widely across various computational problems. SpGEMM's applications, in particular, are substantial in solving problems related to algebraic multigrid methods, network analysis, and advanced matrix operations integral to scientific and engineering computational tasks.

Given the pervasive role of sparse computations in fields such as numerical simulations, network analysis, and machine learning, these results are notably beneficial. They promise improved performance and reduced resource usage for large-scale applications that are increasingly common in high-performance computing environments. Furthermore, this research sets the foundation for future explorations in communication-avoiding algorithms and hierarchical parallelism, both of which are critical in anticipating next-generation architectures dominated by multicore processing units. Overall, this paper offers a methodologically sound and scalable solution to a longstanding computational problem, paving the way for more efficient future developments in sparse matrix computing.