Level-Synchronous BFS: Parallel Graph Traversal

• Level-Synchronous BFS is a parallel graph traversal algorithm that uses explicit synchronization at each level to ensure correctness and exploit extensive parallelism.
• It integrates methodologies across shared-memory, distributed-memory, and GPU architectures, employing dynamic core allocation, optimized scans, and sparse matrix techniques.
• This approach delivers scalable performance and energy efficiency improvements, with implementations demonstrating significant speedups on modern many-core systems.

Level-synchronous Breadth-First Search (BFS) is a key parallel graph traversal method structured around explicit synchronization points between levels of discovery. At each BFS level, the algorithm identifies all vertices at a fixed distance from the source, simultaneously explores their neighbors, and synchronizes before continuing to the next layer. This design both simplifies algorithmics and exposes substantial parallelism suitable for shared-memory and distributed-memory architectures, as well as modern accelerators such as GPUs with specialized matrix-multiplication hardware. Recent research has established matching theoretical lower bounds, identified leading edge implementations exploiting architecture-specific primitives, and assessed performance trade-offs in practice.

1. Level-Synchronous BFS Framework and Algorithmic Structure

Level-synchronous BFS proceeds by maintaining, for each distance $l$ from a given source $s$, a frontier array (often denoted CurLevelVertices, $F$, or $x^{(\ell)}$) containing all vertices at level $l$. The computation for each level consists of four key subroutines: (a) scanning outgoing edges from the current frontier, (b) collecting neighbors not previously seen, (c) deduplicating this candidate set, and (d) forming the next-level array. All threads execute these steps in parallel for the current level before synchronizing and proceeding.

This structure occurs in shared-memory (Tithi et al., 2022), distributed-memory (Buluc et al., 2011, Buluc et al., 2017), and accelerator-based (Elbek et al., 26 Dec 2025) settings. The core design—parallel work interleaved with global barriers between levels—results in a simple and analyzable execution model:

• Each level is a parallel phase with synchronization; all computation for level $l$ precedes any for level $l+1$.
• Status arrays (distance/predecessor) are updated via "benign races," i.e., concurrent threads may attempt to write the same value to a not-yet-visited vertex, since all writes are semantically equivalent due to BFS invariants.

In matrix algebra terms, level-synchronous BFS can be recast as repeated sparse-matrix-vector multiplications (SpMSpV) over the Boolean semiring, with the frontier vector encoding active vertices (Elbek et al., 26 Dec 2025).

2. Theoretical Work, Span, and Lower Bounds

The critical performance parameters for level-synchronous BFS are the graph size ($n = |V|$, $m = |E|$), graph diameter $D$, the per-level work $s$0, and the available parallelism $s$1. Serial BFS has cost $s$2. For parallel execution, the optimal time is lower-bounded by:

$s$3

This reflects the sum of perfect work distribution ($s$4) and the minimal $s$5 synchronization inherent to spawning $s$6 tasks per level, for $s$7. Traditional level-synchronous shared-memory BFSes achieve

$s$8

which is not optimal when frontier sizes are much smaller than $s$9 for some levels. The S3BFS algorithm (Tithi et al., 2022) dynamically matches the number of cores $F$0 active at level $F$1 to the available work $F$2, achieving

$F$3

asymptotically for all graph shapes, saturating the lower bound in both work and span.

3. Parallelization Strategies: Shared Memory, Distributed Memory, and Accelerators

Shared Memory

In shared-memory systems, level-synchronous BFSes employ fine-grained parallelism for both vertex and edge exploration. S3BFS eliminates locks and atomic instructions (except benign CAS on distance arrays), relying on divide-and-conquer spawning with per-level adaptive core count ($F$4). This both minimizes idle core time and reduces bandwidth/cache-pressure, translating directly into asymptotic energy efficiency and empirical savings—up to 30% energy reduction on levels with $F$5, and scalable throughput on many-core coprocessors (Tithi et al., 2022).

Work-efficient parallel prefix-sum (scan) subroutines are required to partition work and accumulate edge counts. S3BFS optimizes scan by (i) capping the number of threads to $F$6 for arrays of length $F$7, and (ii) using blockwise and inter-block scans for optimal work/log-span balance.

Distributed Memory

In distributed settings, level-synchronous methods proceed by having each processing node own a partition (1D or 2D) of the vertex and adjacency space. At each level, after exploring local adjacency, processors exchange newly discovered vertices using collective communication. The communication costs per level are dominated by all-to-all (1D) or semi-collective (2D mesh) operations. For 1D partitioning, the per-level communication time scales as

$F$8

where $F$9 is the network latency per message, and $x^{(\ell)}$0 is the per-word cost for all-to-all. 2D partitioning reduces collective scale (each step uses only $x^{(\ell)}$1 participants), thus achieving lower latency and better load balance beyond a few thousand cores (Buluc et al., 2011, Buluc et al., 2017).

Hybrid MPI+threads schemes partition vertices among ranks but use multithreading within nodes, markedly reducing the MPI communicator count and associated latency. This leads to superior scaling—e.g., 17.8 GTEPS on 40,000 cores for BFS on R-MAT graphs (Buluc et al., 2011).

GPU/Tensor Core Acceleration

Modern GPU-based level-synchronous BFS pipelines, such as BLEST (Elbek et al., 26 Dec 2025), reinterpret each BFS level as a Boolean SpMSpV. The adjacency matrix is packed and traversed using frontier-encoded bitmaps. The Binarised Virtual Slice Sets (BVSS) data layout and associated graph-reordering schemes ensure well-balanced, high-throughput work distribution at warp-level granularity. Specialized matrix-multiply-accumulate (MMA) units (Tensor Cores) execute batches of dot-products (pull-style expansion), achieving substantial reductions in kernel launch and atomic memory operation overhead through algorithmic kernel fusion and lazy vertex updates.

BLEST shows mean speedups of $x^{(\ell)}$2 (BerryBees), $x^{(\ell)}$3 (Gunrock), and $x^{(\ell)}$4 (GSWITCH) on large-graph benchmarks (Elbek et al., 26 Dec 2025).

4. Data Structures, Memory Layouts, and Partitioning

The memory representation of adjacency and visited status is central to efficient level-synchronous BFS. CSR (compressed sparse row) is prevalent for its fast row access on locality-constrained architectures but is memory-intensive for large 2D decompositions, as each processor incurs per-row overhead proportional to $x^{(\ell)}$5 (Buluc et al., 2017). DCSC (doubly compressed sparse columns) stores only nonzero columns/hypersparse blocks, considerably reducing memory at the cost of a modest (10–30%) per-edge processing increase.

In accelerator settings, the focus is on data layout for vectorized/matrix-multiply execution. The BVSS design in BLEST clusters source vertices into bit-packed slices, further partitioned into virtual sets matched to hardware thread-block size, minimizing frontier-oblivious work assignment and maximizing Tensor Core occupancy (Elbek et al., 26 Dec 2025). Graph reordering—Jaccard-based for social graphs, Reverse Cuthill–McKee for geometric graphs—is employed to optimize mask packing and cache line usage.

A summary table of key data structure trade-offs is as follows:

Structure	Memory Scalability	Access Speed	Scalability Issues
CSR	$x^{(\ell)}$6	Constant	Excess per-row overhead at large $x^{(\ell)}$7
DCSC	$x^{(\ell)}$8	Logarithmic	~10–30% slower per edge, fits larger graphs
BVSS (BLEST)	GPU-oriented, bitpacked	Blocked/reduced	Requires reordering to minimize divergence and balance slices

5. Performance, Energy, and Scalability Considerations

Optimal level-synchronous BFS implementations align the amount of active parallel processing with the available work per level, reducing both time and energy usage. In the shared-memory regime, dynamically adapting thread count at each BFS level, as in S3BFS, eliminates unnecessary core usage, producing measurable energy savings—over 30% on small-frontier levels, and overall speedups exceeding $x^{(\ell)}$9 compared to fixed-allocation strategies (Tithi et al., 2022).

On distributed-memory supercomputers, the dominant scaling bottleneck is communication latency from collectives at each level; 2D partitioning and hybrid threading relieve this to the extent that, on large graphs (e.g., $l$0, $l$1), BFS achieves 17.8 GTEPS using hybrid 1D partitioning at 40,000-core scale (Buluc et al., 2011); cost is primarily in all-to-all exchanges and pointer array memory (Buluc et al., 2017).

For GPUs, batch-oriented techniques, kernel fusion, and lazy updates reduce kernel launch and atomic update costs, with BLEST maintaining speedups over Gunrock and achieving the best results reported on both social (dense) and geometric (sparse) graphs (Elbek et al., 26 Dec 2025).

Strong and weak scaling studies reveal that improving load balance, minimizing unnecessary thread activation, and judicious memory layout are the major levers of scalable BFS performance.

6. Comparative Analysis with Prior and Variant Methods

Traditional level-synchronous BFS with static work division and global synchronization at each level delivers predictable execution but is suboptimal when the frontier is small relative to $l$2. Direction-optimizing BFS (alternating top-down and bottom-up) reduces unnecessary edge scans by switching traversal direction when the frontier exceeds a threshold, yielding an order-of-magnitude speedup at scale (Buluc et al., 2017).

Cilk reducers and GPU BFS pipelines relying on atomics or fixed-thread launches (e.g., Gunrock, Merrill & Grimshaw) exhibit high lock contention and lack adaptability to work fluctuations, leading to inferior span and energy characteristics (Tithi et al., 2022, Elbek et al., 26 Dec 2025).

The S3BFS design (Tithi et al., 2022) is characterized by its matching of theoretical lower bounds, dynamic throttling of active cores, lock-free design (only benign CAS), and proportional energy usage. BLEST demonstrates the ability to fully utilize hardware matrix multiply units for sparse, ordered BFS computation on GPU, leveraging sparsity-aware data partitioning and batch updates for maximum throughput (Elbek et al., 26 Dec 2025).

7. Challenges, Limitations, and Ongoing Work

Several open challenges persist in level-synchronous BFS:

• For highly skewed or dynamic graphs, static reordering (e.g., Jaccard-based or RCM) may be suboptimal; dynamic policy selection is under active research for BLEST (Elbek et al., 26 Dec 2025).
• The $l$3 cost of advanced reorderings can dominate preprocessing on massive instances; scalable heuristics are needed for multi-billion-edge graphs.
• On distributed-memory architectures, 1D partitions suffer from maximal collective participation and load imbalance; 2D and hierarchical decompositions alleviate but do not fully remove communication bottlenecks (Buluc et al., 2017, Buluc et al., 2011).
• While direction-optimizing alternatives yield dramatic gains, they are not strictly level-synchronous and require additional logic and data structures.

Continued progress will require further integration of communication-avoiding schemes, hardware-adaptive scheduling, and graph-analytics-specific memory layouts.

---

Key References:

• "An Optimal Level-synchronous Shared-memory Parallel BFS Algorithm with Optimal parallel Prefix-sum Algorithm and its Implications for Energy Consumption" (Tithi et al., 2022)
• "Parallel Breadth-First Search on Distributed Memory Systems" (Buluc et al., 2011)
• "Distributed-Memory Breadth-First Search on Massive Graphs" (Buluc et al., 2017)
• "BLEST: Blazingly Efficient BFS using Tensor Cores" (Elbek et al., 26 Dec 2025)