DiskJoin: SSD-based Vector Similarity Join

• DiskJoin is a disk-resident, SSD-optimized system enabling large-scale approximate vector similarity self-joins by reorganizing data into buckets.
• It employs efficient bucketization to group vectors for sequential SSD access and uses graph-based task ordering to maximize cache reuse.
• Probabilistic pruning and Belady-based cache management significantly reduce candidate comparisons and disk traffic while maintaining high recall.

Searching arXiv for DiskJoin and closely related systems to ground the article with current paper metadata and related references. DiskJoin is a single-machine, SSD-based system for large-scale vector similarity self-join (SSJ). It is designed to find all pairs of vectors within a distance threshold $\epsilon$ when the dataset does not fit in main memory, including billion-scale datasets, and it does so by making disk I/O rather than compute the primary design target. The system adopts an approximate SSJ formulation with target recall, reorganizes vectors into buckets stored consecutively on SSD, constructs a bucket graph over potentially matching bucket pairs, and combines task ordering, Belady-based cache management, and probabilistic pruning to reduce read amplification, repetitive access, and candidate comparisons (Chen et al., 25 Aug 2025).

1. Problem definition and computational setting

DiskJoin studies similarity self-join over a vector dataset $\mathcal{X}=\{x_1,\dots,x_N\}$, where each $x_n$ is a $d$-dimensional vector, under Euclidean distance. The SSJ task is defined as finding all vector pairs $(x_n,x_{n'})$ such that $\Vert x_n-x_{n'}\Vert \le \epsilon$ and $1\le n\neq n' \le N$ (Chen et al., 25 Aug 2025). Because exact SSJ in high dimensions suffers from the curse of dimensionality, and classical pruning methods such as triangle inequality become weak, the paper explicitly chooses an approximate SSJ formulation whose objective is to maintain high recall while reducing candidate comparisons. Recall is defined as

$$r=\frac{|\mathcal{R} \cap \mathcal{R}'|}{|\mathcal{R}|}.$$

The central systems claim is that, once the dataset exceeds RAM, the main bottleneck is disk I/O rather than computation. The paper attributes this to several interacting factors: the data are stored on SSD with only a small fraction fitting in memory; traditional disk-based vector indexes often access vectors individually; individual vectors are often smaller than the 4KB page size; and query-at-a-time processing causes both read amplification and repetitive access (Chen et al., 25 Aug 2025). The profiling comparison reported for DiskANN on BigANN100M illustrates the scale of this effect: DiskANN total time is 130,107 sec with 115,854 GB disk traffic, whereas DiskJoin total time is 1,185 sec with 265 GB disk traffic.

A recurrent misconception addressed by the paper is that similarity join can be treated as a straightforward variant of nearest-neighbor search. DiskJoin’s design rejects that view. The paper’s stated bottom line is that similarity join is not nearest-neighbor search: SSJ is a batch problem over a static dataset, so the system can reorganize both the data and the computation to maximize sharing of disk reads and cache reuse (Chen et al., 25 Aug 2025). This suggests that an SSD-resident SSJ engine should be optimized around collective access patterns rather than per-query index traversal.

2. Data organization, bucketization, and the bucket graph

DiskJoin is built around access batching. Instead of processing vectors one query at a time, it groups similar vectors into buckets. Each bucket $b$ has a center vector $c_b$ and a radius $\mathcal{X}=\{x_1,\dots,x_N\}$0, defined as the maximum distance from any vector in the bucket to the center (Chen et al., 25 Aug 2025). The vectors in each bucket are stored consecutively on disk, enabling sequential reads. According to the paper, this reduces read amplification by reading a bucket as a unit rather than reading vectors individually, increases data reuse because vectors in the same bucket tend to share $\mathcal{X}=\{x_1,\dots,x_N\}$1-neighbors, and makes disk access much more sequential and thus SSD-friendly.

The bucketization procedure is explicitly designed to be memory-light and I/O-efficient. It consists of three steps: sample bucket centers by randomly sampling a small set $\mathcal{X}=\{x_1,\dots,x_N\}$2 of vectors, with the sample about $\mathcal{X}=\{x_1,\dots,x_N\}$3 of the dataset and obtained by streaming the dataset once; assign vectors to the nearest sampled center using an in-memory HNSW index on the sampled centers; and write buckets back to disk using in-memory buffers that are flushed when full, thereby avoiding write amplification from writing tiny vector fragments directly to disk (Chen et al., 25 Aug 2025). The paper states that HNSW construction costs $\mathcal{X}=\{x_1,\dots,x_N\}$4, query cost is $\mathcal{X}=\{x_1,\dots,x_N\}$5, and the memory footprint is only about $\mathcal{X}=\{x_1,\dots,x_N\}$6 of the dataset size. The disk layout is independent of $\mathcal{X}=\{x_1,\dots,x_N\}$7 and the target recall $\mathcal{X}=\{x_1,\dots,x_N\}$8, so it can be reused across queries.

Once bucketization is complete, DiskJoin constructs a bucket graph in which each node is a bucket and an edge indicates that the two buckets may contain vector pairs within distance $\mathcal{X}=\{x_1,\dots,x_N\}$9. For buckets $x_n$0 and $x_n$1 with centers $x_n$2 and $x_n$3 and radii $x_n$4 and $x_n$5, a necessary condition for potential $x_n$6-neighbors is

$x_n$7

This necessary condition follows from triangle inequality, but the paper emphasizes that in high dimensions it is too weak, which motivates the probabilistic pruning stage (Chen et al., 25 Aug 2025).

Component	Definition	Role
Bucket	Center vector $x_n$8 and radius $x_n$9	Access batching and sequential SSD reads
Bucket graph	Node = bucket; edge = possible $d$0-match	Encodes candidate bucket-pair tasks
Disk layout	Vectors in the same bucket stored consecutively	Reduces read amplification

A plausible implication is that DiskJoin’s performance depends on the quality of this disk-resident bucket layout at least as much as on the subsequent join kernel. The paper’s design makes bucketization a first-class systems primitive rather than a preprocessing convenience.

3. Task orchestration, graph reordering, and cache management

The second major idea in DiskJoin is task orchestration. After the bucket graph is built, DiskJoin processes bucket-pair tasks in an order that improves cache reuse, decomposing orchestration into task ordering and cache management (Chen et al., 25 Aug 2025). The key systems observation is that the bucket graph gives future access knowledge once the edge processing order is fixed; this permits an optimal eviction policy for that fixed order.

The paper formalizes the scheduling problem as Minimum Edge Cover with Cache (MECC). Given an undirected graph $d$1 and an initially-empty cache with capacity $d$2, the cache supports $d$3 and $d$4; an edge $d$5 is covered if both $d$6 and $d$7 are present in the cache at the same time; and the objective is to find the sequence of load operations with the minimum length that covers all edges of $d$8 (Chen et al., 25 Aug 2025). The paper proves that the MECC problem is NP-hard by reduction from Independent Set. DiskJoin therefore does not solve scheduling optimally; it uses a heuristic.

For cache management, DiskJoin uses Belady’s algorithm. Given a fixed edge processing order, the access sequence of buckets is

$d$9

with length $(x_n,x_{n'})$0. At each cache miss, if the cache is full, the system evicts the bucket whose next use is farthest in the future. The implementation uses a max-heap $(x_n,x_{n'})$1 over cached buckets, an array $(x_n,x_{n'})$2 storing all future access positions for each bucket, and a counter array $(x_n,x_{n'})$3 tracking how many times each bucket has been seen so far. The stated complexity is $(x_n,x_{n'})$4 because the sequence length is $(x_n,x_{n'})$5 and the heap size is $(x_n,x_{n'})$6 (Chen et al., 25 Aug 2025).

Task ordering is handled by graph reordering. The paper’s intuition is that if buckets $(x_n,x_{n'})$7 and $(x_n,x_{n'})$8 have many common out-neighbors, then processing them close together will permit reuse of those neighbors in cache. It defines a sliding window of size

$(x_n,x_{n'})$9

where $\Vert x_n-x_{n'}\Vert \le \epsilon$0 is the average out-degree, and seeks to maximize

$\Vert x_n-x_{n'}\Vert \le \epsilon$1

for the reordered node sequence $\Vert x_n-x_{n'}\Vert \le \epsilon$2 (Chen et al., 25 Aug 2025). The greedy heuristic starts with the node of largest out-degree and repeatedly chooses the next node that maximizes overlap of its neighbors with the current sliding window. Its optimized implementation maintains priority scores

$\Vert x_n-x_{n'}\Vert \le \epsilon$3

and has complexity

$\Vert x_n-x_{n'}\Vert \le \epsilon$4

The reported effect is substantial: reordering improves cache hit rate by about 50%, and together with Belady improves hit rate by about 20% more compared with the baseline cache behavior, with the combined method achieving cache hit rates above 75% even with only 10% memory (Chen et al., 25 Aug 2025). This suggests that the scheduling and caching layer is not an auxiliary optimization but a coequal component of the algorithmic design.

4. Probabilistic pruning and approximation behavior

DiskJoin’s approximation mechanism is probabilistic pruning. The paper treats this as the key technique for reducing the number of candidate bucket pairs generated by the bucket graph. The motivation is explicit: the triangle-inequality condition

$\Vert x_n-x_{n'}\Vert \le \epsilon$5

produces too many candidate bucket pairs in high dimensions, so DiskJoin prunes far candidate buckets probabilistically while trying to preserve a target recall $\Vert x_n-x_{n'}\Vert \le \epsilon$6 (Chen et al., 25 Aug 2025).

The geometric model assumes that vectors are uniformly distributed in the hypersphere $\Vert x_n-x_{n'}\Vert \le \epsilon$7 for a bucket $\Vert x_n-x_{n'}\Vert \le \epsilon$8. For a bucket with $\Vert x_n-x_{n'}\Vert \le \epsilon$9 candidate buckets, if the algorithm prunes the $1\le n\neq n' \le N$0 furthest ones, the missed-neighbor fraction is bounded by the relative volume of the pruned regions. The paper writes this as

$1\le n\neq n' \le N$1

and then applies a referenced geometric bound involving $1\le n\neq n' \le N$2, $1\le n\neq n' \le N$3, and the scaled boundary distance $1\le n\neq n' \le N$4 (Chen et al., 25 Aug 2025). When $1\le n\neq n' \le N$5, there is no intersection and the bucket does not contribute to the sum.

Algorithmically, the rule is simple. Algorithm 3 sorts candidate buckets in descending order of distance to the current bucket and accumulates

$1\le n\neq n' \le N$6

It stops when

$1\le n\neq n' \le N$7

At that point, the buckets encountered so far are pruned (Chen et al., 25 Aug 2025). In the paper’s interpretation, the farthest candidate buckets are pruned until the estimated probability of missing true neighbors reaches the allowed error budget $1\le n\neq n' \le N$8.

The practical effect reported is large: pruning reduces the number of candidate bucket pairs, the number of distance computations, and disk access volume, while causing only a tiny recall loss, about 0.1% in experiments (Chen et al., 25 Aug 2025). A common misunderstanding would be to treat DiskJoin as an exact external-memory join algorithm; the paper is explicit that it is approximate rather than exact. Its quality criterion is therefore controlled recall under a target $1\le n\neq n' \le N$9, not exhaustive pair enumeration.

5. Experimental evaluation and performance characteristics

The evaluation uses three real-world datasets: Deep100M with 100 million 96-dimensional image embeddings from GoogLeNet, BigANN100M with 100 million 128-dimensional SIFT descriptors, and SPACEV1B with 1.402 billion 100-dimensional document embeddings from Microsoft SpaceV Superior (Chen et al., 25 Aug 2025). The baselines are DiskANN, adapted by using each vector as a query and increasing $$r=\frac{|\mathcal{R} \cap \mathcal{R}'|}{|\mathcal{R}|}.$$0 until $$r=\frac{|\mathcal{R} \cap \mathcal{R}'|}{|\mathcal{R}|}.$$1-neighbors are found; ClusterJoin, implemented as a single-node in-memory version for fair comparison; and RSHJ, an in-memory LSH-based similarity join. The paper notes that there is no prior system specifically targeting disk-based vector similarity join.

Two AWS servers are used. Server 1 has a 48-core Intel Xeon Platinum 8375C @ 2.90GHz, 96GB RAM, and 2 × 1.3TB NVMe SSDs, and is used for Deep100M and BigANN100M. Server 2 has a 72-core Intel Xeon E5-2686 v4 @ 2.30GHz, 512GB RAM, and 8 × 1.8TB NVMe SSDs, and is used for SPACEV1B. The default settings are memory = 10% of dataset size, target recall $$r=\frac{|\mathcal{R} \cap \mathcal{R}'|}{|\mathcal{R}|}.$$2, and $$r=\frac{|\mathcal{R} \cap \mathcal{R}'|}{|\mathcal{R}|}.$$3 chosen so each vector has about 100 neighbors on average. OS page caching is disabled using O_DIRECT (Chen et al., 25 Aug 2025).

The main experimental result is that DiskJoin significantly outperforms alternatives. Compared to DiskANN, DiskJoin is consistently 2–3 orders of magnitude faster, with reported speedups ranging from 52× to 1137×. At 95% recall on Deep100M, the reported speedup reaches 1137×. Compared to ClusterJoin and RSHJ, DiskJoin is still significantly faster; RSHJ fails on larger datasets because of its memory and time complexity, and ClusterJoin scales poorly and becomes near-quadratic in distance computations (Chen et al., 25 Aug 2025).

The paper identifies three principal causes of the speedup. First, disk traffic is much lower: DiskJoin reads almost exactly the useful data, with read amplification near 1, specifically 1.0039 on Deep and 1.0026 on BigANN, whereas DiskANN has read amplification around 6–7×. Second, task reordering and Belady improve cache hit rate, and DiskJoin’s cache hit rate exceeds 75% with only 10% memory. Third, candidate pruning greatly reduces the number of bucket pairs and therefore distance computations and I/O (Chen et al., 25 Aug 2025).

Several sensitivity results further delimit the operating regime. As memory increases from 5% to 20% of dataset size, DiskJoin remains 2–3 orders of magnitude faster than DiskANN, but gains diminish beyond 10% memory because DiskJoin becomes compute-bound rather than I/O-bound. As the distance threshold increases, runtime grows sublinearly up to about 500 neighbors on average, though cost increases because more bucket pairs survive pruning. Best performance is achieved with about 0.1% of the dataset size as the number of buckets; too few buckets yield coarse partitioning and more candidate pairs, whereas too many buckets make buckets too small and reduce I/O efficiency. Orchestration overhead is about 5% of total runtime. Robustness across random bucket-center sampling is reported with mean recall 0.903, standard deviation 0.005, mean runtime 276.02 sec, and standard deviation 12.62 sec. Under fragmented file systems, performance remains robust until fragmentation becomes severe enough that extents fall below 4KB, which causes read amplification (Chen et al., 25 Aug 2025).

6. Scope, limitations, and extensions

DiskJoin’s stated contributions are a new system for disk-based vector similarity join; bucket-wise access batching; task orchestration through an NP-hard scheduling formulation solved via graph reordering and Belady cache eviction; probabilistic pruning based on geometric volume bounds and arccos-based error accumulation; scalable bucketization using random sampling and HNSW; and strong empirical results with 50× to 1000× speedups and practical billion-scale performance on a single machine (Chen et al., 25 Aug 2025). Within that scope, the system is especially effective when the dataset is too large for memory but fits on SSD, the vectors are high-dimensional embeddings, the task is offline batch similarity join rather than latency-sensitive online search, a small-to-moderate amount of memory is available, and sequential bucket-based access can be exploited.

The limitations are also explicit. DiskJoin is approximate, not exact. Performance depends on the quality of bucketization and probabilistic pruning. The method assumes a bucketed disk layout, and severe fragmentation can hurt. Extremely high-dimensional or very dense-neighborhood settings can still increase cost. The paper further suggests that, after I/O is sufficiently reduced, the system can become compute-bound, so further speedups may require GPU acceleration (Chen et al., 25 Aug 2025). These limitations are structural rather than incidental: they follow directly from the design choice to trade exactness for high recall and to reorganize static data for offline processing.

The paper also outlines an extension from self-join to cross-join. DiskJoin can be extended to cross-join between two datasets by bucketizing each dataset separately, making the bucket graph bipartite, and reordering the larger dataset while caching the smaller one. Both cross-join variants outperform DiskANN, and reordering the larger dataset is reported to be slightly better (Chen et al., 25 Aug 2025). This suggests that the bucket-graph and orchestration abstractions are not restricted to self-join, but can be transferred to related batch similarity workloads.

In aggregate, DiskJoin presents a systems argument about SSD-resident vector analytics: when the workload is similarity join over a static dataset, the decisive optimizations are bucket-wise storage, bucket-graph dependency analysis, cache-aware task ordering, and controlled probabilistic pruning rather than per-query index traversal. Under that framing, the system’s principal significance lies in turning SSD-based similarity join from an I/O-bound problem into one in which disk traffic is almost completely tamed on a single machine (Chen et al., 25 Aug 2025).