---
title: 'EP-BFS: Semi-External Breadth-First Search'
url: https://www.emergentmind.com/topics/ep-bfs
type: topic
---

# EP-BFS: Semi-External Breadth-First Search

EP-BFS is a semi-external breadth-first search algorithm for massive directed, disk-resident graphs, introduced in "Efficient Semi-External Breadth-First Search" [2507.12925]. It is designed to compute a BFS tree while using only \(O(n)\) main memory, with the explicit goal of keeping the in-memory sketch small, minimizing I/Os, and avoiding the heavy internal overhead associated with fully external BFS. The algorithm operates by maintaining a compact partial tree and edge sketch in RAM, scanning residual edges sequentially from disk, invoking an in-memory reduction procedure only when necessary, and progressively pruning vertices and edges whose final BFS positions have already been fixed [2507.12925].

## 1. Computational setting and motivation

EP-BFS is formulated in the semi-external memory model. The disk is divided into blocks of size \(B\), each I/O transfers one block between disk and RAM, and the main memory can hold at most \(M\) elements. Unlike the fully external model, the semi-external model assumes \(M \ge c \times n\), where \(n=|V(G)|\) and \(c\) is a small constant, so memory is sufficient to hold at least a spanning tree of the graph. The graph \(G=(V,E)\) is disk-resident, while RAM stores only a small sketch \(\mathcal{A}\subseteq G\) together with per-vertex attributes. The paper uses the minimum memory space requirement (MMSR) to denote the space used for this sketch and its node attributes; with 32-bit vertices and attributes, a “reasonable” MMSR is stated as roughly
\[
\text{MMSR} \;\lesssim\; 64 \times 2n + 3 \times 32n \text{ bits},
\]
that is, a few words per vertex plus \(O(n)\) edges [2507.12925].

The motivation is the inadequacy of both standard in-memory BFS and classical external-memory BFS on contemporary web and social graphs. Internal-memory BFS has time complexity \(O(n+m)\) but requires the entire graph in RAM, which is infeasible for datasets such as WDC-2014 with 1.7 billion nodes and 64 billion edges, or eu-2015 with 1.07 billion nodes and 91.8 billion edges. Fully external BFS can reduce I/O asymptotically, but the best known algorithm for general directed graphs, EM-BFS, relies on a buffered repository tree and has I/O complexity
\[
O\!\left((n + \frac{m}{B})\log_2 \frac{n}{B} + sort(m)\right),
\]
with substantial internal overhead. EP-BFS is positioned as the semi-external alternative: it retains \(O(n)\)-scale memory while pursuing substantially better practical performance on billion-scale directed graphs [2507.12925].

## 2. Graph-theoretic basis and precursor methods

The algorithm is built around a structural characterization of BFS trees. The graph is conceptually augmented with a dummy root \(r\) connected to all vertices, so disconnected components are handled by enqueuing children of \(r\) when the BFS queue becomes empty. For a spanning tree \(T\), the breadth-first order of a node \(u\) is denoted \(bfo(u,T)\). The key notion is a **V-BFS edge**: an edge \((u,v)\in E(G)\) such that \(bfo(u,T) < bfo(v,T)\), and if \(w\) is the parent of \(v\) in \(T\), then \(w\neq r\) and \(bfo(w,T)\le bfo(u,T)\). The paper states that a spanning tree \(T\) of \(G\) is a BFS-tree if and only if there is no V-BFS edge in \(G\) as classified by \(T\). Semi-external BFS is therefore cast as the problem of repeatedly restructuring an arbitrary spanning tree until no V-BFS edges remain [2507.12925].

Two baseline semi-external algorithms are used as reference points. EE-BFS maintains only a spanning tree in memory and locally restructures it whenever a scanned edge is a V-BFS edge; its worst-case time is \(O(n \times m \times \text{LLSP}(G))\), where LLSP\((G)\) is the length of the longest simple path. EB-BFS stores a spanning tree together with a batch of up to \(Kn\) edges, runs an in-memory BFS on the union, and achieves time \(O(m \times \text{LLSP}(G))\) and I/O \(O\!\big(\frac{m}{B}\times \text{LLSP}(G)\big)\). EP-BFS preserves the same small-memory regime as EB-BFS but modifies the sketch, the triggering logic for in-memory processing, and the pruning strategy to reduce practical cost substantially [2507.12925].

| Method | In-memory sketch | Stated behavior |
|---|---|---|
| EE-BFS | Spanning tree \(T\) | Edge-by-edge restructuring; worst-case \(O(nm\cdot \text{LLSP}(G))\) time |
| EB-BFS | \(T\) plus batch \(\mathbb{E}\) of size up to \(Kn\) | Batch in-memory BFS; \(O(m\cdot \text{LLSP}(G))\) time |
| EP-BFS | Forest \(\mathbb{T}\), remaining tree edges on disk, batch \(\mathbb{E}\) | Same asymptotic bound as EB-BFS, lower practical cost through thresholds, pruning, and cache-oriented layout |

## 3. Algorithmic organization

EP-BFS decomposes the current spanning tree into an in-memory forest \(\mathbb{T}\) and a disk-resident set \(E_{\mathbb{T}}\) of remaining tree edges. It also maintains an in-memory edge batch \(\mathbb{E}\), with the capacity constraint \(|\mathbb{T}|+|\mathbb{E}|\le (K+1)n\). For each vertex \(u\), it stores two mutable attributes: \(u.\mathcal{B}\), the current BFS index, and \(u.\mathcal{P}\), the current parent. Additional structures include the array `BON`, which stores the BFS order of unpruned nodes and doubles as a queue during the in-memory reduction; `Adj` and `ES`, which represent adjacency lists with contiguous outgoing edges per vertex; and threshold values \(\mathcal{F}[i]\), \(\mathcal{F}_R\), \(\mathcal{F}_C\), and \(\mathcal{F}_{CC}\) [2507.12925].

The initialization phase scans the graph, partitions edges into sublists \(E_1,\dots,E_k\) of up to \((1+K)n\) edges each, builds an initial forest, and treats nodes of zero in-degree or zero out-degree specially via \(E_i\) and \(E_o\). When the in-memory sketch fills, a simple reduction recomputes a BFS forest of \(\mathbb{T}\cup\mathbb{E}\). A subsequent `TreeReduce` consolidates the structure into a spanning tree rooted at the dummy root and initializes the parent and BFS-order attributes. The main loop then repeatedly scans the active residual edge set \(E_R\), deciding for each edge whether it is irrelevant, should be retained for the next iteration, or should be inserted into the current batch \(\mathbb{E}\) [2507.12925].

The threshold mechanism is the distinctive control device. Each batch \(i\) has a threshold \(\mathcal{F}[i]\), initialized to \(+\infty\). When a scanned edge \((u,v)\) is detected as a V-BFS edge with respect to the current tree, \(\mathcal{F}[i]\) is updated to \(u.\mathcal{B}\), and the edge is inserted into \(\mathbb{E}\). Edges are otherwise filtered according to the relation of their endpoints’ BFS orders to \(\mathcal{F}[i]\), \(\mathcal{F}_R\), \(\mathcal{F}_C\), and \(\mathcal{F}_{CC}\). This reduces both the number of edges admitted into the in-memory sketch and the number of times the in-memory procedure must be invoked [2507.12925].

The in-memory procedure itself is `EP-Reduce`. It performs BFS on \(\mathbb{G}=(V(\mathbb{T}),E(\mathbb{T})\cup\mathbb{E})\), using `BON` as the queue and rebuilding \(\mathbb{T}\) from the resulting parent pointers. Because outgoing edges of each vertex are stored contiguously in `ES`, scanning a node’s adjacency during this phase is cache-friendly. After each iteration, EP-BFS updates the global thresholds through the `Find` routine, prunes vertices whose positions are already fixed via `vPrune`, and shrinks the residual edge set via `ErPrune`. Over time the active graph \(G_R\) decreases, which lowers both scan volume and the number of future in-memory reductions. All disk I/O is sequential because the residual edge set is reorganized into adjacency lists on disk [2507.12925].

## 4. Correctness, complexity, and memory profile

The central correctness statement is Theorem 4.1 in the paper: EP-BFS returns the BFS tree correctly with at most LLSP\((G)\) iterations. The proof relates EP-BFS to repeated applications of reduced versions of EB-BFS and shows that the thresholding and pruning steps preserve the eventual BFS-tree while discarding only vertices and edges whose positions are already fixed. The dummy-root reconstruction step then restores the full tree on all vertices, including those with zero in-degree or zero out-degree [2507.12925].

The stated worst-case time complexity is
\[
O(m \times \text{LLSP}(G)),
\]
because `EP-Reduce` takes \(O(n)\) time per invocation, there can be at most \(\lceil m/n\rceil\) invocations per iteration, and there are at most LLSP\((G)\) iterations. The I/O complexity is
\[
O\!\left(\frac{m}{B}\times \text{LLSP}(G)\right),
\]
since each iteration scans the residual edge set sequentially. Asymptotically this matches EB-BFS, but EP-BFS reduces the constant factors through fewer in-memory reductions, pruning of the residual graph, and contiguous adjacency storage [2507.12925].

The memory design is explicitly minimal for the semi-external setting. EP-BFS stores only two attributes per node, \(\mathcal{B}\) and \(\mathcal{P}\), together with at most \(O((K+1)n)\) in-memory edges. For \(K=1\), the paper describes this as effectively \(\Theta(n)\) memory. On large graphs, the observed sketch size remained a small fraction of the full graph size: for eu-2015, the graph size is 683 GB and the EP-BFS sketch size \(|\mathcal{A}|\) is 32.2 GB; for WDC-2014, the graph size is 480 GB and the sketch size is 51.7 GB; for Friendster, the graph size is 19.3 GB and the sketch size is 2.06 GB [2507.12925].

## 5. Empirical evaluation

The experimental study compares EP-BFS with EE-BFS, EB-BFS, and GridGraph on 14 real graphs and multiple synthetic graph families. The implementation uses Java 8 for EP-BFS, EE-BFS, and EB-BFS, and evaluates performance primarily on an HDD, with SSD experiments used to isolate the effect of device speed. The default parameter is \(K=1\), meaning that the in-memory sketch can hold \((K+1)n\) edges [2507.12925].

On real graphs, EP-BFS consistently reduces runtime and I/O relative to EB-BFS, and on the largest graphs it is often the only approach finishing within the 24-hour time limit. For example, on uk-2002, EB-BFS takes 4,097 seconds and 75.5 GB of I/O, whereas EP-BFS takes 138 seconds and 12.7 GB; on twitter-2010, EB-BFS takes 11,165 seconds and 175 GB of I/O, whereas EP-BFS takes 682 seconds and 40.8 GB. On uk-2014, clueweb12, gsh-2015, eu-2015, and WDC-2014, EB-BFS times out, while EP-BFS completes with runtimes from 25,535 to 73,526 seconds and I/O from 1,558 to 4,124 GB. The abstract summarizes the overall effect as “up to 10 times faster,” while individual datasets reported in the paper exhibit substantially larger gaps in some cases [2507.12925].

| Dataset | EB-BFS | EP-BFS |
|---|---|---|
| uk-2002 | 4,097 s; 75.5 GB I/O | 138 s; 12.7 GB I/O |
| twitter-2010 | 11,165 s; 175 GB I/O | 682 s; 40.8 GB I/O |
| WDC-2014 | timeout | 32,496 s; 1,558 GB I/O |

The paper attributes the improvement to three effects: fewer invocations of the in-memory process, lower per-iteration I/O due to graph pruning, and better cache behavior from contiguous adjacency storage. Random edge-sampling experiments on clueweb12 show that EP-BFS remains viable even when EB-BFS requires more than 20 hours on sparsified instances. In comparisons against GridGraph on subgraphs of twitter-2010, GridGraph times out for subgraphs with more than 3% of edges, whereas EP-BFS processes the full twitter-2010 graph in 682 seconds using about 1.26 GB memory. Synthetic experiments further show that EP-BFS scales better than EB-BFS as average degree and graph size increase, that its runtime is relatively stable across \(K\in[0.05,2]\), and that HDD and SSD results are similar because the algorithm relies almost entirely on sequential I/O and is often CPU-bound rather than device-bound [2507.12925].

## 6. Terminology and related uses

The label “EP-BFS” is not uniform across graph-algorithm literature. In the semi-external memory setting, it specifically denotes the algorithm of "Efficient Semi-External Breadth-First Search" [2507.12925]. In separate lines of work, closely related labels have been used for different concepts: reinforcement/backup fault-tolerant BFS structures in which some edges are reinforced and others serve as backups [1504.04169]; dual-failure and multiple-source fault-tolerant BFS structures that preserve exact BFS distances under edge failures [1505.00692], [1704.06907]; and elimination- or parameter-oriented BFS variants used to generate AT-free graph orders through domination-convexity tie-breaking [1807.05065].

This suggests that EP-BFS is best interpreted contextually rather than as a globally standardized term. In large-scale graph processing, the name refers to a semi-external BFS algorithm with \(O(n)\)-memory sketches, threshold-guided edge admission, iterative pruning, and sequential scans. In structural graph theory and survivable network design, similarly named constructions address different objectives entirely: exact distance preservation under failures, or search orders enforcing special convexity or elimination properties.

Source: https://www.emergentmind.com/topics/ep-bfs