--- title: 'Dollarless eBWT: Sentinel-Free Transform' url: https://www.emergentmind.com/topics/dollarless-ebwt type: topic --- # Dollarless eBWT: Sentinel-Free Transform The dollarless eBWT (extended Burrows–Wheeler Transform) encompasses a class of transformations that extend or reconstruct Burrows–Wheeler-style text permutations, originally devised for string compression and indexing, but explicitly without the need for end-marker (sentinel, dollar "$") characters. This approach has significant theoretical, algorithmic, and practical ramifications for text collections, compressed indexing, and genome analysis, providing both efficiency improvements and new guarantees over classical constructions. ## 1. Formal Definitions and Core Properties The dollarless eBWT generalizes the classic BWT by omitting any special end-of-string character. Consider a multiset of primitive (non-periodic) strings $\mathcal M = \{T_1, \dots, T_m\}$ over a finite, ordered alphabet $\Sigma$, where $|T_i| = n_i$ and total length $N = \sum_i n_i$. For each $T_i$, all $n_i$ cyclic rotations are considered, and indices are treated modulo $n_i$. The generalized conjugate array is obtained by collecting all these cyclic rotations and sorting them under the so-called $\omega$-order, characterized by: - For $U, V \in \Sigma^*$, $U \prec_\omega V$ if (a) $U, V$ have the same primitive root and $U <_{\text{lex}} V$, or (b) $U^\omega <_{\text{lex}} V^\omega$ otherwise, with $U^\omega$ denoting infinite repetition of $U$. The dollarless eBWT output string $B[1..N]$ is formed by reading, for each sorted rotation, the character preceding its starting position, with the positions of first-rotations also recorded. Notably, no sentinels or external markers are ever introduced at any step—rotations are entirely circular [2106.11191]. For a single string $T$ (i.e., $m=1$), this process yields the circular BWT, coinciding with the classical BWT modulo rotation choice. ## 2. Algorithmic Construction: Linear-Time, Sentinel-Free Approaches Dollarless eBWT can be computed in linear time via an adaptation of the SAIS (Suffix Array Induced Sorting) paradigm. The main steps are: - **Cyclic S/L/LMS Typing:** Each cyclic position is labelled as S-type or L-type according to whether its rotation precedes or follows its neighbor under $\omega$-order. LMS-positions are those S-positions whose predecessor is L-type. - **Bucket-Based Induction:** Rotations are bucketed by their first character. LMS-positions are first placed in their correct buckets, then multiple left-to-right and right-to-left passes induce L- and S-type orderings, respectively. This guarantees LMS-substrings are sorted correctly under $\omega$-order. - **Recursion on Named Substrings:** Distinct LMS-substrings are replaced by unique names, a shorter instance is constructed, and recursion proceeds until all names are distinct. - **Final Induction:** When recursion returns, positions are mapped back to the original array, and further induction fills in the complete rotation ordering. Finally, length-1 strings are trivially handled. The entire procedure runs in $O(N)$ time and $O(N)$ space for $N$ total input length, and never requires appending a sentinel or computing Lyndon rotations. This construction underpins both single-string and multi-string (collection) eBWT algorithms [2106.11191]. ## 3. Combinatorics of Sentinel Insertion and Characterizations Classical BWT is usually defined for strings $v\$ $ (with appended sentinel), but not every string over $(\Sigma \cup \{\$\})^{n+1}$ is a BWT image. For a string $w \in \Sigma^n$ (without sentinel), a central combinatorial problem is: in which positions $i$ does insertion of a sentinel (between $w_{i-1}$ and $w_i$) make $w'$ a valid BWT-image? This is formalized as a "nice position". Formally, for a permutation $\pi_w$ induced (by stable sorting) from $w$, define "pseudo-cycles" $S \subseteq \{1, \dots, n\}$, each split as $S_L < S_R$, such that $\pi(S) = (S_L - 1) \cup S_R$. The critical interval $R(S) = [\max(S_L)+1, \min(S_R)]$ and insertion positions $i$ outside any $R(S)$ for all pseudo-cycles are exactly the "nice" ones, i.e., those yielding valid BWT images. All nice $i$ have the same parity, and a variety of combinatorial bounds on the number and location of such positions have been established [1908.09125]. This characterizes which strings $w$ can be viewed as BWT images of some $v$ with a sentinel. The answer depends entirely on the structure of $\pi_w$—its cycles and pseudo-cycles. If at least one nice position exists, the original string can be recovered as the dollarless eBWT without explicit insertion of the sentinel: select any nice position, virtually insert the sentinel for computation, then remove it from the output to yield a fully compatible, invertible, and indexable transform. ## 4. Practical Algorithms: Construction and Inversion In direct implementation, absent a dedicated sentinel-free construction algorithm, one can: - Concatenate the collection with sentinels, form the standard EBWT via the suffix array over $T=S_1\$S_2\$ \dots S_t\$$, - Identify all sentinel positions in the output BWT and store them in a bitvector $E[1..N]$, - Remove all sentinels from the stored transform to obtain the dollarless eBWT $L$. The pair $(L, E)$ provides a fully invertible representation: to invert, merge $L$ and the marker positions given by $E$ to reconstruct the full BWT, then perform standard LF-mapping to recover the original text or collection [1809.07320]. For each character abutting a removed sentinel in the BWT, adjacent runs may now coalesce, yielding a more compressible representation for downstream indexing or storage. This practical approach enables O(N) time and space construction and inversion, with the only overhead being the additional marker bitvector whose size is $O(t \log(N/t) + t)$ bits for $t$ strings. For large collections or highly repetitive texts, prefix-free parsing (PFP) can serve as an effective scalable variant. Here, phrases beginning and ending at hash-based triggers are parsed, the dollarless eBWT of the parse is computed, then the overall transform is reconstructed blockwise. The method reduces both memory and time for very large genomics datasets [2106.11191]. ## 5. Compression and Indexing Benefits Empirical studies show measurable benefits of omitting sentinels in practice. For example, on a human chromosome 19 readset (concatenated text with $N$ large), constructing the standard EBWT with one \$ per read yields roughly 220 million runs. Eliminating all \$’s from the output reduces the run count in $L$ by approximately 19%, from 220 million to 178 million [1809.07320]. This reduction arises as runs of real characters adjacent to sentinels can now merge, providing more favorable run-length encodings essential for compressed index structures. Subsequent application of tree-based eXtended BWTs (XBWT) using a labeled tree based on the reference genome further compresses the representation. In the same setting, the XBWT reduced the run count by an additional 15%, to approximately 150 million runs, establishing its utility for highly repetitive, reference-driven collections [1809.07320]. The dollarless eBWT enables the direct application of FM-index and other compressed full-text indexing strategies on the transformed text, as LF-mapping, rank, and select operations carry over unchanged. Asymptotic time and space complexities remain equivalent to the sentinel-based approaches. ## 6. Variants and Extensions: Large-Scale and Reference-Aware Approaches The theoretical and algorithmic frameworks for dollarless eBWT motivate further optimizations: - **Scalable Construction with PFP:** Using prefix-free parsing with rolling hash triggers allows the algorithm to handle datasets well beyond main memory, leveraging bucketwise and blockwise merging for both transform computation and inversion [2106.11191]. - **Reference-Aware Transform (XBWT):** When a reference genome is available and reads are aligned, a labeled tree structure permits application of Ferragina et al.’s XBWT. This structure clusters sequences based on path-labels rather than only local suffix contexts, yielding further compressibility and supporting full-text indexed search [1809.07320]. Both avenues benefit from the absence of sentinels, as they mitigate redundant boundary symbols and enable unification of adjacent runs. ## 7. Summary Table: Dollarless eBWT Construction Approaches | Approach | Sentinel Needed? | Time Complexity | Memory Overhead | |------------------------|------------------|-----------------|----------------------| | Direct SAIS-eBWT [2106.11191] | No | $O(N)$ | $O(N)$ | | SA/EBWT+strip [1809.07320] | Yes (virtual) | $O(N)$/$O(N\log N)$ | $O(N)$ text, $O(t\log(N/t))$ bits for E | | Prefix-Free Parsing | No | $O(N)$ | $O(|D| + N_p)$ | *Here, $N$ is total input length, $t$ is string count, $|D|$ is dictionary size, $N_p$ is total phrase count.* ## References - "When a Dollar Makes a BWT" [1908.09125] - "Computing the original eBWT faster, simpler, and with less memory" [2106.11191] - "Compressing and Indexing Aligned Readsets" [1809.07320] Source: https://www.emergentmind.com/topics/dollarless-ebwt