Parallel Breadth-First Search on Distributed Memory Systems (1104.4518v2)

Abstract: Data-intensive, graph-based computations are pervasive in several scientific applications, and are known to to be quite challenging to implement on distributed memory systems. In this work, we explore the design space of parallel algorithms for Breadth-First Search (BFS), a key subroutine in several graph algorithms. We present two highly-tuned parallel approaches for BFS on large parallel systems: a level-synchronous strategy that relies on a simple vertex-based partitioning of the graph, and a two-dimensional sparse matrix-partitioning-based approach that mitigates parallel communication overhead. For both approaches, we also present hybrid versions with intra-node multithreading. Our novel hybrid two-dimensional algorithm reduces communication times by up to a factor of 3.5, relative to a common vertex based approach. Our experimental study identifies execution regimes in which these approaches will be competitive, and we demonstrate extremely high performance on leading distributed-memory parallel systems. For instance, for a 40,000-core parallel execution on Hopper, an AMD Magny-Cours based system, we achieve a BFS performance rate of 17.8 billion edge visits per second on an undirected graph of 4.3 billion vertices and 68.7 billion edges with skewed degree distribution.

• The paper develops two innovative approaches—a level-synchronous strategy and a two-dimensional partitioning method—to significantly reduce communication overhead in distributed BFS.
• It employs a hybrid MPI-threading model, achieving up to 17.8 billion edge visits per second on large-scale graph datasets.
• The study introduces a detailed performance model that evaluates memory latency and bandwidth, paving the way for future enhancements in distributed graph processing.

Overview of Parallel Breadth-First Search on Distributed Memory Systems

The paper "Parallel Breadth-First Search on Distributed Memory Systems" by Aydın Buluç and Kamesh Madduri presents innovative algorithms and implementation strategies for executing Breadth-First Search (BFS) efficiently on large-scale distributed memory systems. The focus is on addressing the challenges posed by graph-based computations—critical sub-tasks in numerous applications that involve vast-scale or complex relational data such as social networks or biological systems.

Core Contributions

The authors develop two primary approaches for parallel BFS: a straightforward level-synchronous strategy and a more sophisticated two-dimensional matrix-partitioning-based method. Both are tested and optimized extensively on distributed memory systems and include modes enhanced with intra-node multithreading. These contributions are particularly aimed at reducing communication overhead, which is a critical bottleneck in scaling BFS performance across distributed systems.

1. Level-Synchronous Approach: This approach partitions the graph based on vertices, allowing parallel traversal without intricate communication among processors. Initial experiments with this method have shown competitive performance in specific load regimes.
2. Two-Dimensional Split: By representing and dealing with the graph as a sparse matrix, the authors leverage a 2D partitioning scheme. This structure substantially minimizes communication overhead by restricting collective communication operations to subsets of processors. This technique is most beneficial in scenarios with high communication-to-computation ratios and illustrates impressive performance scaling up to 40,000 cores.
3. Hybrid Multithreading Strategy: Combining MPI and threading within each node, this approach substantially increases performance by balancing intra- and inter-node computations while also reducing communication overhead.

Performance Results

The paper provides comprehensive experimental results on the Hopper and Franklin supercomputers. The algorithms demonstrate superior performance metrics, achieving a maximum of 17.8 billion edge visits per second on a dataset with 68.7 billion edges. Notably, the two-dimensional partitioning approach reduces communication times by up to a factor of 3.5. These results highlight that the developed methods are not constrained by communication costs, which can traditionally dominate runtime in distributed BFS implementations.

Theoretical and Practical Implications

The authors present a performance model that incorporates memory access latency and bandwidth—a relevant metric for understanding execution time on current architectures. This model facilitates an evaluation of different algorithmic strategies and supports future research on refining graph algorithm execution on distributed systems. The approach not only improves the baseline of BFS performance but also encourages consideration of BFS as a benchmark for assessing system capabilities, particularly within the context of the Graph 500 list of supercomputer rankings.

Future Developments

While the paper establishes a comprehensive groundwork, there are unexplored avenues:

• Memory and Communication Optimizations: Further refinements could involve exploring graph partitioning's impact on communication volume and exploring potential reductions in overall data transfer.
• Alternate Programming Models: Future works might investigate the extent to which other programming paradigms, such as PGAS, could simplify and enhance distributed graph computations while retaining performance.
• Scalable Collective Operations: Optimizing collective operations in distributed environments, considering the supercomputer architectures of future systems, remains an open field with significant potential impact.

In conclusion, the paper advances the state-of-the-art in distributed memory BFS algorithms and underscores the importance of communication optimizations in achieving high performance for graph algorithms at scale. This paper provides a foundational base for both practical applications and theoretical explorations in parallel graph processing.