Extended Burrows–Wheeler Transform (eBWT)

• The extended Burrows–Wheeler Transform (eBWT) is a generalization of the classic BWT that operates on multisets of strings, including circular sequences and finite automata, to support advanced indexing and compression.
• Algorithmic frameworks such as SAIS-style induced sorting, grammar-based methods, and prefix-free parsing enable nearly linear time computation and reduced memory usage for processing massive sequence collections.
• eBWT unifies various combinatorial structures—such as necklaces, de Bruijn words, and automata indexing—to improve pattern matching and compression, though optimal run-minimization remains a complex challenge.

The extended Burrows–Wheeler Transform (eBWT) is a mathematical and algorithmic generalization of the classical Burrows–Wheeler Transform (BWT), designed to operate on multisets (collections) of strings rather than a single input. Unlike the standard BWT, which applies to a single string with a unique sentinel, the eBWT encompasses a broader set of combinatorial structures, including circular strings, necklaces, collections of sequences, and even finite automata. This flexibility has led to foundational roles for the eBWT in areas such as pan-genomic indexing, alignment-free sequence analysis, compressed data structures, and automata indexing.

1. Formal Definitions and Structural Properties

The eBWT operates on a multiset $$\mathcal{S} = \{S_1, S_2, ..., S_t\} \subseteq \Sigma^+$$, typically assumed to consist of primitive strings (not proper powers). The essential steps are:

• For each $S_x \in \mathcal{S}$ of length $\ell_x$, enumerate all its cyclic rotations (conjugates) $\text{conj}_k(S_x) = S_x[k+1..\ell_x] S_x[1..k]$, $k=0,\dots,\ell_x-1$.
• Collect all such conjugates across the multiset.
• Impose the $\omega$-order: order all rotations by the lexicographic order of their infinite periodic extensions, i.e., $U^\omega = UUU...$ for a string $U$.
• Once all rotations are sorted, define the eBWT as the array $eBWT[1..N]$, $N = \sum_x \ell_x$, whose $i$-th entry is the symbol immediately preceding the start of the $i$-th rotation in its own cycle (with wrap-around).

This construction is independent of the ordering of input strings, a key property established by Mantaci et al. and exploited in several indexing and compression applications (Boucher et al., 2021, Bannai et al., 2019, Higgins, 2019, Olbrich, 27 Apr 2025, Ingels et al., 5 Jun 2025). The eBWT specializes to the ordinary BWT given a singleton set $\mathcal{S} = \{T$\$(with$T$arbitrary and \$ a unique sentinel).

2. Algorithmic Frameworks and Construction Methods

Efficient computation of the eBWT leverages advanced suffix sorting and induced sorting paradigms:

• SAIS-style Induced Sorting: Adapt SAIS, originally designed for linear-time suffix array construction, to multisets of cycles under $\omega$-order. Key modifications include classifying substrings by cyclic L/S/LMS typing and recursive substrings replacement; full eBWT computation is accomplished in $O(N)$ time for integer alphabets (Bannai et al., 2019, Boucher et al., 2021). Compared to prior Lyndon rotation–based approaches, the SAIS-based construction is simpler, omits explicit sentinels, supports arbitrary multiset orderings, and generalizes to the classical BWT as a degenerate case.
• Grammar-based and Lyndon SLP approaches: For highly repetitive datasets, constructing the eBWT from a compressed grammar (Lyndon SLP) yields significant space and time advantages. Strings are factorized into Lyndon words and context-free grammars (SLPs) are used to represent repetitions; the eBWT is then extracted by propagating lexicographic order through the nonterminal expansions and performing a specific run-length–encoded traversal (Olbrich, 27 Apr 2025). This approach offers nearly linear construction for massive databases, minimize memory usage, and supports high parallelism in grammar formation.
• Prefix-free Parsing (PFP) combination: On very large, repetitive collections, PFP is used to compress the input into a set of phrases and a parse array. eBWT is then computed on the parse, and finally, the full transformation is synthesized from the parse-eBWT and the phrase dictionary (Boucher et al., 2021). This pipeline reduces memory footprint substantially and enables construction on collections of hundreds of thousands of sequences.
• Incremental and dynamic extensions: Recent work demonstrates fully dynamic eBWT constructions where new sequences can be added or removed with controlled time and space complexities, leveraging dynamic string data structures and efficient index weaving (Osterkamp et al., 2024).

3. Algebraic and Combinatorial Generalizations

The eBWT encapsulates and unifies several well-known BWT extensions:

• Necklaces and de Bruijn words: The eBWT’s structure naturally extends to multisets of necklaces (cyclically equivalent primitive words) and is closely tied to the combinatorics of de Bruijn sequences. Inverting the eBWT provides mechanisms for generating all de Bruijn sequences of fixed span and controlling the number of distinct factors in strings (Higgins, 2019). The mapping can also be described as a union of partial order-preserving permutations on the input space, linking the transform to syntactic semigroups of cyclic languages.
• Automata-theoretic generalizations: The eBWT framework can be extended to nondeterministic finite automata (NFA) via co-lexicographic orders on state sets. The eBWT is then a function of the state labels under the coarsest forward-stable co-lex order, facilitating effective FM-index–like pattern matching, with complexity determined by the width of the partial order (Becker et al., 10 Mar 2025).
• Cartesian-tree indexing: The eBWT admits direct extension to complex equivalence classes of rotations, such as those defined by sharing the same Cartesian tree (relevant in time-series and music analysis). Construction algorithms support multiple circular texts, dynamic inclusion, and nearly succinct space (Osterkamp et al., 2024).

4. Statistical and Compression Properties

The compressibility of the eBWT is governed by the number of "runs" in the transformation—maximal sequences of repeated symbols:

• The number of runs generated by the eBWT is highly sensitive to the decomposition of the input; poor decompositions can lead to linearly many runs (resulting in weak compression), while well-chosen decompositions can reduce the number of runs to a constant bound with respect to the alphabet size, regardless of the input length (Ingels et al., 5 Jun 2025).
• The number of possible decompositions of a word into substrings for input to the eBWT grows exponentially with string length, leading to hard combinatorial optimization problems when seeking run-minimal decompositions in the context of compression schemes.
• A key consequence is that, although the eBWT dramatically generalizes the BWT's cluster-forming and run-minimizing capabilities, achieving optimal compressibility may require sophisticated heuristics or constraints on decomposition strategies.

5. Bioinformatics: Indexing, SNP Calling, and Clustering

The eBWT underpins advanced alignment-free and reference-free sequence analysis, with particular impact in genomics:

• SNP discovery: By applying the eBWT to multisets of DNA reads, bases from identical or near-identical genomic loci are clustered in contiguous segments in the transformed sequence. Theoretical analysis models the distribution of cluster sizes using a Poisson process dependent on read coverage and error rate. Maximal LCP (Longest Common Prefix) intervals in the transformed array correspond to clusters of reads covering the same genomic context. This mechanism allows for O(1)-space, O(N)-time identification of genomic variants (e.g., SNPs) without reference genomes, using LF-mapping and cluster-based voting procedures to derive variant calls with high sensitivity and precision at modest coverage levels (Prezza et al., 2018).
• Positional clustering: eBWT-based clustering accurately predicts the number of read-copies of any genome position present in a cluster, enabling statistical filtering to discard ambiguous or spurious clusters. This property has facilitated next-generation pan-genomic indices and rapid variation discovery in large-scale datasets (Prezza et al., 2018).
• Empirical performance: eBWT-based pipelines have demonstrated superior tradeoffs between sensitivity, precision, RAM usage, and run time relative to state-of-the-art tools for reference-free variant discovery, particularly in moderate-to-highly covered human genome simulations.

6. Applications in Indexing, Pattern Matching, and Data Structures

The eBWT serves as the backbone of modern compressed indices:

• FM-index extensions: The transform enables the construction of FM-indices on sets of sequences, collections of circular strings, or the state-space of labeled automata; all support LF- and backward-search procedures generalized from the single-string case (Boucher et al., 2021, Becker et al., 10 Mar 2025).
• Pattern matching in graphs and automata: Generalizations of the eBWT to NFAs allow succinct indices for pattern matching in variation graphs and pangenomes. Efficiency depends on properties such as the width of the co-lex order on automaton states.
• Pan-genomic and assembly-free analysis: The order independence and combinatorial generality of the eBWT underpin efficient representations of large sequence collections, enabling scalable pan-genomic toolkit development (Boucher et al., 2021, Olbrich, 27 Apr 2025).

7. Limitations, Open Problems, and Future Directions

While the eBWT unifies many BWT-like structures, several challenges remain:

• Decomposition complexity: For optimizing run-length encoding, the exponential growth in decomposition choices renders global minimization intractable; practical compressors must balance block size, decomposition constraints, and computational cost (Ingels et al., 5 Jun 2025).
• Automata eBWT construction: For NFAs, constructing the coarsest forward-stable co-lex order dominates complexity. Linear-space representations now exist, but near-linear–time algorithms for arbitrary NFAs remain an open frontier (Becker et al., 10 Mar 2025).
• Worst-case grammar size: Grammar-based methods can degrade to $O(N)$ size in low-repetitiveness regimes, affecting practical performance (Olbrich, 27 Apr 2025).
• Parallelism and scalability: While certain phases of grammar-based and PFP-based constructions are highly parallelizable, the final scan for BWT symbol production may become a sequential bottleneck. Work is ongoing to fully parallelize single-string and multi-string eBWT construction.
• Robust support for full FM-index functionality in eBWT-based indices, including efficient construction of complete locate and large-alphabet support, is an area of ongoing research (Boucher et al., 2021).

The continued generalization of the Burrows–Wheeler Transform by the eBWT framework supports both theoretical exploration (algebraic, automata-theoretic, combinatorial) and large-scale practical deployments in text and sequence indexing, biological data analysis, and run-minimal compression.