EP-BFS: Semi-External Breadth-First Search

• EP-BFS is a semi-external BFS algorithm for massive directed graphs that builds BFS trees using O(n) memory and compact in-memory sketches combined with sequential disk scans.
• It employs a threshold mechanism for edge filtering and iterative pruning to reduce I/O overhead and improve cache efficiency during in-memory reductions.
• Empirical evaluations demonstrate that EP-BFS significantly outperforms traditional methods by reducing runtime and disk I/O on large real-world datasets.

EP-BFS is a semi-external breadth-first search algorithm for massive directed, disk-resident graphs, introduced in "Efficient Semi-External Breadth-First Search" (Wan et al., 17 Jul 2025). It is designed to compute a BFS tree while using only $O(n)$ main memory, with the explicit goal of keeping the in-memory sketch small, minimizing I/Os, and avoiding the heavy internal overhead associated with fully external BFS. The algorithm operates by maintaining a compact partial tree and edge sketch in RAM, scanning residual edges sequentially from disk, invoking an in-memory reduction procedure only when necessary, and progressively pruning vertices and edges whose final BFS positions have already been fixed (Wan et al., 17 Jul 2025).

1. Computational setting and motivation

EP-BFS is formulated in the semi-external memory model. The disk is divided into blocks of size $B$, each I/O transfers one block between disk and RAM, and the main memory can hold at most $M$ elements. Unlike the fully external model, the semi-external model assumes $M \ge c \times n$, where $n=|V(G)|$ and $c$ is a small constant, so memory is sufficient to hold at least a spanning tree of the graph. The graph $G=(V,E)$ is disk-resident, while RAM stores only a small sketch $\mathcal{A}\subseteq G$ together with per-vertex attributes. The paper uses the minimum memory space requirement (MMSR) to denote the space used for this sketch and its node attributes; with 32-bit vertices and attributes, a “reasonable” MMSR is stated as roughly

$$\text{MMSR} \;\lesssim\; 64 \times 2n + 3 \times 32n \text{ bits},$$

that is, a few words per vertex plus $O(n)$ edges (Wan et al., 17 Jul 2025).

The motivation is the inadequacy of both standard in-memory BFS and classical external-memory BFS on contemporary web and social graphs. Internal-memory BFS has time complexity $B$0 but requires the entire graph in RAM, which is infeasible for datasets such as WDC-2014 with 1.7 billion nodes and 64 billion edges, or eu-2015 with 1.07 billion nodes and 91.8 billion edges. Fully external BFS can reduce I/O asymptotically, but the best known algorithm for general directed graphs, EM-BFS, relies on a buffered repository tree and has I/O complexity

$B$1

with substantial internal overhead. EP-BFS is positioned as the semi-external alternative: it retains $B$2-scale memory while pursuing substantially better practical performance on billion-scale directed graphs (Wan et al., 17 Jul 2025).

2. Graph-theoretic basis and precursor methods

The algorithm is built around a structural characterization of BFS trees. The graph is conceptually augmented with a dummy root $B$3 connected to all vertices, so disconnected components are handled by enqueuing children of $B$4 when the BFS queue becomes empty. For a spanning tree $B$5, the breadth-first order of a node $B$6 is denoted $B$7. The key notion is a V-BFS edge: an edge $B$8 such that $B$9, and if $M$0 is the parent of $M$1 in $M$2, then $M$3 and $M$4. The paper states that a spanning tree $M$5 of $M$6 is a BFS-tree if and only if there is no V-BFS edge in $M$7 as classified by $M$8. Semi-external BFS is therefore cast as the problem of repeatedly restructuring an arbitrary spanning tree until no V-BFS edges remain (Wan et al., 17 Jul 2025).

Two baseline semi-external algorithms are used as reference points. EE-BFS maintains only a spanning tree in memory and locally restructures it whenever a scanned edge is a V-BFS edge; its worst-case time is $M$9, where LLSP$M \ge c \times n$0 is the length of the longest simple path. EB-BFS stores a spanning tree together with a batch of up to $M \ge c \times n$1 edges, runs an in-memory BFS on the union, and achieves time $M \ge c \times n$2 and I/O $M \ge c \times n$3. EP-BFS preserves the same small-memory regime as EB-BFS but modifies the sketch, the triggering logic for in-memory processing, and the pruning strategy to reduce practical cost substantially (Wan et al., 17 Jul 2025).

Method	In-memory sketch	Stated behavior
EE-BFS	Spanning tree $M \ge c \times n$4	Edge-by-edge restructuring; worst-case $M \ge c \times n$5 time
EB-BFS	$M \ge c \times n$6 plus batch $M \ge c \times n$7 of size up to $M \ge c \times n$8	Batch in-memory BFS; $M \ge c \times n$9 time
EP-BFS	Forest $n=|V(G)|$0, remaining tree edges on disk, batch $n=|V(G)|$1	Same asymptotic bound as EB-BFS, lower practical cost through thresholds, pruning, and cache-oriented layout

3. Algorithmic organization

EP-BFS decomposes the current spanning tree into an in-memory forest $n=|V(G)|$2 and a disk-resident set $n=|V(G)|$3 of remaining tree edges. It also maintains an in-memory edge batch $n=|V(G)|$4, with the capacity constraint $n=|V(G)|$5. For each vertex $n=|V(G)|$6, it stores two mutable attributes: $n=|V(G)|$7, the current BFS index, and $n=|V(G)|$8, the current parent. Additional structures include the array BON, which stores the BFS order of unpruned nodes and doubles as a queue during the in-memory reduction; Adj and ES, which represent adjacency lists with contiguous outgoing edges per vertex; and threshold values $n=|V(G)|$9, $c$0, $c$1, and $c$2 (Wan et al., 17 Jul 2025).

The initialization phase scans the graph, partitions edges into sublists $c$3 of up to $c$4 edges each, builds an initial forest, and treats nodes of zero in-degree or zero out-degree specially via $c$5 and $c$6. When the in-memory sketch fills, a simple reduction recomputes a BFS forest of $c$7. A subsequent TreeReduce consolidates the structure into a spanning tree rooted at the dummy root and initializes the parent and BFS-order attributes. The main loop then repeatedly scans the active residual edge set $c$8, deciding for each edge whether it is irrelevant, should be retained for the next iteration, or should be inserted into the current batch $c$9 (Wan et al., 17 Jul 2025).

The threshold mechanism is the distinctive control device. Each batch $G=(V,E)$0 has a threshold $G=(V,E)$1, initialized to $G=(V,E)$2. When a scanned edge $G=(V,E)$3 is detected as a V-BFS edge with respect to the current tree, $G=(V,E)$4 is updated to $G=(V,E)$5, and the edge is inserted into $G=(V,E)$6. Edges are otherwise filtered according to the relation of their endpoints’ BFS orders to $G=(V,E)$7, $G=(V,E)$8, $G=(V,E)$9, and $\mathcal{A}\subseteq G$0. This reduces both the number of edges admitted into the in-memory sketch and the number of times the in-memory procedure must be invoked (Wan et al., 17 Jul 2025).

The in-memory procedure itself is EP-Reduce. It performs BFS on $\mathcal{A}\subseteq G$1, using BON as the queue and rebuilding $\mathcal{A}\subseteq G$2 from the resulting parent pointers. Because outgoing edges of each vertex are stored contiguously in ES, scanning a node’s adjacency during this phase is cache-friendly. After each iteration, EP-BFS updates the global thresholds through the Find routine, prunes vertices whose positions are already fixed via vPrune, and shrinks the residual edge set via ErPrune. Over time the active graph $\mathcal{A}\subseteq G$3 decreases, which lowers both scan volume and the number of future in-memory reductions. All disk I/O is sequential because the residual edge set is reorganized into adjacency lists on disk (Wan et al., 17 Jul 2025).

4. Correctness, complexity, and memory profile

The central correctness statement is Theorem 4.1 in the paper: EP-BFS returns the BFS tree correctly with at most LLSP$\mathcal{A}\subseteq G$4 iterations. The proof relates EP-BFS to repeated applications of reduced versions of EB-BFS and shows that the thresholding and pruning steps preserve the eventual BFS-tree while discarding only vertices and edges whose positions are already fixed. The dummy-root reconstruction step then restores the full tree on all vertices, including those with zero in-degree or zero out-degree (Wan et al., 17 Jul 2025).

The stated worst-case time complexity is

$\mathcal{A}\subseteq G$5

because EP-Reduce takes $\mathcal{A}\subseteq G$6 time per invocation, there can be at most $\mathcal{A}\subseteq G$7 invocations per iteration, and there are at most LLSP$\mathcal{A}\subseteq G$8 iterations. The I/O complexity is

$\mathcal{A}\subseteq G$9

since each iteration scans the residual edge set sequentially. Asymptotically this matches EB-BFS, but EP-BFS reduces the constant factors through fewer in-memory reductions, pruning of the residual graph, and contiguous adjacency storage (Wan et al., 17 Jul 2025).

The memory design is explicitly minimal for the semi-external setting. EP-BFS stores only two attributes per node, $$\text{MMSR} \;\lesssim\; 64 \times 2n + 3 \times 32n \text{ bits},$$0 and $$\text{MMSR} \;\lesssim\; 64 \times 2n + 3 \times 32n \text{ bits},$$1, together with at most $$\text{MMSR} \;\lesssim\; 64 \times 2n + 3 \times 32n \text{ bits},$$2 in-memory edges. For $$\text{MMSR} \;\lesssim\; 64 \times 2n + 3 \times 32n \text{ bits},$$3, the paper describes this as effectively $$\text{MMSR} \;\lesssim\; 64 \times 2n + 3 \times 32n \text{ bits},$$4 memory. On large graphs, the observed sketch size remained a small fraction of the full graph size: for eu-2015, the graph size is 683 GB and the EP-BFS sketch size $$\text{MMSR} \;\lesssim\; 64 \times 2n + 3 \times 32n \text{ bits},$$5 is 32.2 GB; for WDC-2014, the graph size is 480 GB and the sketch size is 51.7 GB; for Friendster, the graph size is 19.3 GB and the sketch size is 2.06 GB (Wan et al., 17 Jul 2025).

5. Empirical evaluation

The experimental study compares EP-BFS with EE-BFS, EB-BFS, and GridGraph on 14 real graphs and multiple synthetic graph families. The implementation uses Java 8 for EP-BFS, EE-BFS, and EB-BFS, and evaluates performance primarily on an HDD, with SSD experiments used to isolate the effect of device speed. The default parameter is $$\text{MMSR} \;\lesssim\; 64 \times 2n + 3 \times 32n \text{ bits},$$6, meaning that the in-memory sketch can hold $$\text{MMSR} \;\lesssim\; 64 \times 2n + 3 \times 32n \text{ bits},$$7 edges (Wan et al., 17 Jul 2025).

On real graphs, EP-BFS consistently reduces runtime and I/O relative to EB-BFS, and on the largest graphs it is often the only approach finishing within the 24-hour time limit. For example, on uk-2002, EB-BFS takes 4,097 seconds and 75.5 GB of I/O, whereas EP-BFS takes 138 seconds and 12.7 GB; on twitter-2010, EB-BFS takes 11,165 seconds and 175 GB of I/O, whereas EP-BFS takes 682 seconds and 40.8 GB. On uk-2014, clueweb12, gsh-2015, eu-2015, and WDC-2014, EB-BFS times out, while EP-BFS completes with runtimes from 25,535 to 73,526 seconds and I/O from 1,558 to 4,124 GB. The abstract summarizes the overall effect as “up to 10 times faster,” while individual datasets reported in the paper exhibit substantially larger gaps in some cases (Wan et al., 17 Jul 2025).

Dataset	EB-BFS	EP-BFS
uk-2002	4,097 s; 75.5 GB I/O	138 s; 12.7 GB I/O
twitter-2010	11,165 s; 175 GB I/O	682 s; 40.8 GB I/O
WDC-2014	timeout	32,496 s; 1,558 GB I/O

The paper attributes the improvement to three effects: fewer invocations of the in-memory process, lower per-iteration I/O due to graph pruning, and better cache behavior from contiguous adjacency storage. Random edge-sampling experiments on clueweb12 show that EP-BFS remains viable even when EB-BFS requires more than 20 hours on sparsified instances. In comparisons against GridGraph on subgraphs of twitter-2010, GridGraph times out for subgraphs with more than 3% of edges, whereas EP-BFS processes the full twitter-2010 graph in 682 seconds using about 1.26 GB memory. Synthetic experiments further show that EP-BFS scales better than EB-BFS as average degree and graph size increase, that its runtime is relatively stable across $$\text{MMSR} \;\lesssim\; 64 \times 2n + 3 \times 32n \text{ bits},$$8, and that HDD and SSD results are similar because the algorithm relies almost entirely on sequential I/O and is often CPU-bound rather than device-bound (Wan et al., 17 Jul 2025).

6. Terminology and related uses

The label “EP-BFS” is not uniform across graph-algorithm literature. In the semi-external memory setting, it specifically denotes the algorithm of "Efficient Semi-External Breadth-First Search" (Wan et al., 17 Jul 2025). In separate lines of work, closely related labels have been used for different concepts: reinforcement/backup fault-tolerant BFS structures in which some edges are reinforced and others serve as backups (Parter et al., 2015); dual-failure and multiple-source fault-tolerant BFS structures that preserve exact BFS distances under edge failures (Parter, 2015, Gupta et al., 2017); and elimination- or parameter-oriented BFS variants used to generate AT-free graph orders through domination-convexity tie-breaking (Beisegel, 2018).

This suggests that EP-BFS is best interpreted contextually rather than as a globally standardized term. In large-scale graph processing, the name refers to a semi-external BFS algorithm with $$\text{MMSR} \;\lesssim\; 64 \times 2n + 3 \times 32n \text{ bits},$$9-memory sketches, threshold-guided edge admission, iterative pruning, and sequential scans. In structural graph theory and survivable network design, similarly named constructions address different objectives entirely: exact distance preservation under failures, or search orders enforcing special convexity or elimination properties.