Ropebwt3 SMEM Algorithm

• Ropebwt3's SMEM algorithm is a method for finding super-maximal exact matches in query strings using a run-length encoded Burrows-Wheeler Transform and FM-index.
• It integrates advanced data structures such as dynamic B⁺-trees, DS-BWT for bidirectional search, and checkpointed rank tables to achieve high scalability.
• The approach offers rapid query speeds with O(m) time per query and outperforms previous methods through optimized skip heuristics and memory-efficient indexing.

Ropebwt3’s Super-Maximal Exact Match (SMEM) algorithm is an efficient, large-scale method for finding all super-maximal exact matches within a query string against a massive reference collection. It builds on a run-length–encoded Burrows-Wheeler Transform (BWT) and FM-index, enabling efficient search, compression, and scalability for ultra-redundant datasets such as pangenome assemblies. At its core, the algorithm operationalizes recent theoretical advances in SMEM finding in a form tailored to repetitive biological data, integrating advanced data structures and bidirectional extensions, and applying heuristics to skip redundant searches and support terabase-scale data indexing and querying (Li, 2024).

1. Formal Foundations and SMEM Definitions

Let $\Sigma$ denote the alphabet (for DNA, $\Sigma = \{A, C, G, T, N\}$), augmented by a sentinel $\$$to define$\Sigma' = \Sigma \cup {\$\}$. Considering$T = P_0\$ 0P_1\$_1 \cdots P_{m-1}\${m-1}$$as the concatenated reference string of total length$$n$, the suffix array$S$$records the start position of each lexicographically ordered suffix. The BWT,$$B$\Sigma = \{A, C, G, T, N\}$0B[i] = T[S(i) - 1]$\Sigma = \{A, C, G, T, N\}$1” symbol.

The interval $\Sigma = \{A, C, G, T, N\}$2 in the FM-index corresponds to the occurrence range of query $\Sigma = \{A, C, G, T, N\}$3 in $\Sigma = \{A, C, G, T, N\}$4. An exact match $\Sigma = \{A, C, G, T, N\}$5 denotes $\Sigma = \{A, C, G, T, N\}$6. This is maximal (MEM) if it cannot be extended in either string, and super-maximal (SMEM) if its interval on $\Sigma = \{A, C, G, T, N\}$7, $\Sigma = \{A, C, G, T, N\}$8, is not properly contained in the interval of any other MEM within $\Sigma = \{A, C, G, T, N\}$9. This structure ensures that SMEMs form a set of non-overlapping, non-contained matches.

2. Core Data Structures in Ropebwt3

Ropebwt3’s efficiency derives from sophisticated use of several intertwined data structures:

• Run-Length–Encoded BWT: The transform $\$0 is compacted by encoding consecutive identical symbols (runs), highly effective for repetitive (e.g., pangenomic) sequences.
• Rope / B⁺-Tree: For dynamic indexing, the run-length encoding is stored in a B⁺-tree; leaves hold the encoded substrings, while internal nodes maintain cumulative symbol counts for all $\$1 symbols. Both rank queries $\$2 and insertions require $\$3 time, with $\$4 as the current number of runs.
• Fermi-Binary Format: In the finalized static index, run encodings are Elias-delta compressed; block headers (typically 4 KB) store cumulative $\$5 counts. This arrangement allows memory mapping and permits rank queries within a block by short scans, commonly with constant average time.
• FM-Index Sampling: To resolve $\$6 intervals to actual locations in $\$7, ropebwt3 stores sampled suffix array values $\$8 at multiples of $\$9 (commonly $to define$0). Locating a reference occurrence requires $to define$1 LF-mapping traversals.

3. Bidirectional Search and Double-Strand BWT (DS-BWT)

To efficiently support bidirectional extension—a requirement for MEM and SMEM finding—ropebwt3 constructs the double-strand BWT (DS-BWT) by indexing $to define$2 concatenated with its reverse-complement. For any string $to define$3, both $to define$4 and its reverse-complement $to define$5 share identical occurrence counts, facilitating symmetric exploration.

A “bi-interval” in this context is a tuple $to define$6. Backward and forward extensions are central operations, defined as follows:

• Backward Extension: Updates $to define$7 when prepending symbol $to define$8 to $to define$9 using rank operations and cumulative counts involving the complement DNA alphabet order.
• Forward Extension: Implemented as a swapped, complemented backward extension.

Backward and forward extensions operate using compiled inner loops over the five DNA bases, which, together with DS-BWT, achieves both bidirectional search and high performance.

4. The SMEM-Finding Procedure

Ropebwt3’s SMEM identification algorithm follows the method of Gagie et al. (DLT 2024), employing interval extension and “skipping” optimization to minimize redundant computation when traversing query $\}$0.

The key procedural steps are:

1. Minimum-Length Verification: Starting at query position $\}$1, extend backward for $\}$2 symbols and test if the interval remains above the minimum occurrence threshold $\}$3.
2. Maximal Forward Extension: From the verified interval, extend forward as long as the occurrence count remains above $\}$4. The resulting $\}$5 interval is a MEM of sufficient length.
3. Skip Covered Subintervals: Using a backward scan from $\}$6 to $\}$7, the procedure determines the shortest remaining unmatched interval, skipping contained MEMs and thus ensuring that [i, e) is an SMEM, not contained in any other interval.

This approach, supported by pseudocode in (Li, 2024), outputs all SMEMs efficiently with the guarantee that no two SMEM intervals on $\}$8 overlap or contain one another.

5. Algorithmic Optimizations and Heuristics

Several optimizations enable high throughput and memory efficiency in ropebwt3:

• Early Stopping: Extensions are immediately abandoned when the occurrence count $\}$9 falls below $T = P_0\$0.
• DS-BWT Symmetry: Single BWT index serves both extension directions, halving the required memory compared to forward/reverse indexing.
• Compiled Inner Loops: DNA alphabet size enables unrolled/compiled loops for intra-block searches, expediting rank operations.
• Checkpointed Rank Tables: In the static fermi format, cumulative symbol counts at block boundaries allow fast rank queries with minimal scanning.
• Gagie’s “Skip-To” Acceleration: For SMEM finding, processed intervals after each SMEM are skipped entirely, preventing duplicate or overlapping searches.

6. Time Complexity, Space Requirements, and Scalability

Ropebwt3 achieves strong scalability for both index construction and querying:

• Construction: Building the run-length–encoded BWT (using library-produced partial BWT merges) takes $T = P_0\$1 time and $T = P_0\$2 space.
• Querying: Each SMEM query of length $T = P_0\$3 incurs $T = P_0\$4; $T = P_0\$5 for the dynamic tree or $T = P_0\$6 in the static fermi block format. For pangenomes, $T = P_0\$7 so $T = P_0\$8 is small.
• FM-index Storage: $T = P_0\$9 words for the BWT, $_1 \cdots P_{m-1}\$0 samples for the suffix array, and $_1 \cdots P_{m-1}\$1 [number of blocks]$_1 \cdots P_{m-1}\$2 for checkpoint tables. Empirically, this amounts to less than 20 GB for 7.3 terabases.
• Application: Ropebwt3 indexed 7.3 terabases in approximately 26 days using at most 170 GB RAM and supports SMEM queries (short and long reads) at rates exceeding 1 million base pairs per second per thread.

7. Comparative Analysis with Prior SMEM Algorithms

Ropebwt3 displays significant advancements over previous SMEM solutions:

Approach	Core Mechanism	Relative Performance
Li et al. (2012)	Dual BWT, linear-scan SMEM	DS-BWT + skip ≈2× faster, ½ RAM
Bannai & Inoue et al. (SODA 2018)	$_1 \cdots P_{m-1}\$3-index, $_1 \cdots P_{m-1}\$4 MEM	No SMEM skipping; less tailored
MONI (Rossi et al., CPM 2022)	$_1 \cdots P_{m-1}\$5-index, locates SMEMs+extensions	Ropebwt3 ≈4× faster, less RAM
Movi/PMLS (Zakeri et al., 2024)	Pseudo-matching-lengths	Not exact; ropebwt3 is exact

Ropebwt3’s DS-BWT combined with Gagie’s interval-skipping SMEM algorithm yields efficient O(m) time per query and, due to its run-length encoding, is particularly effective for highly redundant genomic data. Previous methods either lacked efficient skipping or required higher memory by maintaining dual BWTs or did not guarantee exactness. Ropebwt3 guarantees both correctness (exact SMEMs and SA intervals) and practical scalability on terabase-scale datasets (Li, 2024).