Practical Linear-Time Computation of Smallest Suffixient Sets

Abstract: Suffixient arrays are recent structures that have attracted attention because they offer relevant pattern matching functionality in less asymptotic space than the Run-Length BWT, the de-facto standard to index highly repetitive string collections. Various algorithms exist for building them from the suffix array data structures. We present the first construction algorithm that is (i) linear-time, (ii) one-pass over the structures, and (iii) implemented and practical. This makes the construction particularly useful on large text collections, which we demonstrate empirically by showing that it dominates the space/time tradeoff map of the implemented constructions.

• The paper introduces a linear-time, one-pass algorithm that efficiently constructs smallest suffixient sets for repetitive datasets.
• It leverages stack-based tracking and on-the-fly PSV/NSV computation to reduce space usage by up to 19× compared to prior methods.
• Empirical evaluations on real-world benchmarks demonstrate significant improvements in speed and space, setting a new standard for string indexing.

Practical Linear-Time Computation of Smallest Suffixient Sets

Introduction

The study of string indexing on highly repetitive collections—such as pan-genomic datasets, versioned source code collections, and massive document repositories—has driven the development of succinct and efficient structures for pattern matching. Suffixient sets and corresponding suffixient arrays have emerged as an alternative to run-length encoded Burrows-Wheeler Transform (RLBWT) based indexes, providing similar retrieval functionality while using less space when the compressibility measure $\chi$ (the size of the smallest suffixient set) is considerably smaller than the classic BWT runs parameter $r$. Prior work has yielded multiple algorithms for constructing smallest suffixient sets, but practical, linear-time, one-pass, and implemented solutions remained elusive. This paper introduces the first construction algorithm fulfilling all these properties and establishes empirical dominance in the space/time tradeoff landscape, making it highly relevant for large-scale, repetitive text indexing.

Background and Motivation

Suffixient sets, formalized by Depuydt et al., are minimal subsets of suffixes such that, sorted colexicographically, they enable efficient searching for occurrences of patterns in a text $T$ with significantly reduced space requirements compared to previous indexes. The crucial property is that, for every right-maximal substring and its one-character extension, a suffixient position ensures coverage for maximal exact matching. Suffixient arrays, constructed from these sets, support fast occurrence finding and maximal exact substring detection, making them instrumental for compressed indexes on highly repetitive texts.

Recent efforts have produced increasingly efficient constructions, including online, one-pass, and sublinear-time variants, yet practical implementations of linear-time and one-pass algorithms were missing prior to this work. Theoretical advances left open the question of bridging algorithmic efficiency with real-world practicality, especially with respect to scalability and resource constraints.

Algorithmic Contributions

The central contribution is a linear-time, one-pass construction algorithm for smallest suffixient sets—named LC—that is proven correct, space-efficient, and empirically implemented. The approach advances prior efforts by leveraging structural properties of $c$-run breaks in the BWT domain and refining candidate selection through stack-based tracking of nearest smaller values, eliminating redundant computation and storage.

Characterization of Suffixient Positions

The algorithm leverages the insight that smallest suffixient set construction reduces to detecting certain run-breaks (termed last $c$-candidates) based on maximality conditions in the colexicographically ordered suffix array. Using succinct definitions and an equivalence with run-break maxima (Proposition 1), the algorithm avoids quadratic overhead associated with evaluating local maxima within potentially overlapping interval "boxes," as in earlier approaches.

Algorithmic Realization

The plain last-candidate (PLC) algorithm, initially derived, achieves linear time by maintaining candidate status through sequential scanning, requiring the computation of previous and next smaller values arrays (PSV, NSV). The final, optimized LC algorithm reduces working space by computing PSV and NSV values on-the-fly only for run-breaks, using lightweight stack techniques, thus matching the best known space while improving practical running time.

Experimental Evaluation

Evaluation was performed on large-scale real-world benchmarks from the Pizza&Chili collection, comprising highly repetitive DNA, textual, and source code datasets. The performance of LC was compared to implemented alternatives: PLC, FM, LF, and the online algorithm Onl.

Figure 1: Space usage and running time comparison between all construction algorithms on DNA sequences from Pizza&Chili collection. Logscale in the space coordinate.

LC consistently achieved the lowest or equal space consumption compared to all other methods, using 3.1–3.3$\times$ less space than PLC and FM, and up to 19$\times$ less than Onl, with a slight exception for selected datasets. LC also outperformed prior implementations in running time—being 14–39% faster than LF and 6–32% faster than PLC and FM. The Onl approach is sometimes faster, but its memory requirements (up to $\sim20\times$ higher) render it impractical for large instances.

Figure 2: Space usage and running time comparison between all construction algorithms on einstein.en.txt and sources sequences from Pizza&Chili collection. Logscale in the space coordinate.

When applied to non-DNA, less repetitive collections, similar patterns hold: LC and related constructions scale well with input length, while the online method performs best when $\chi$ is extremely small but at the aforementioned space cost.

Implications and Theoretical Significance

The practical achievement of simultaneous linear time, one-pass, and low space in an implemented algorithm sets a new benchmark for suffixient set constructions. The algorithm's capacity to operate with minimal main memory, especially with on-the-fly computations and stack-based PSV/NSV generation, facilitates scaling to datasets previously infeasible for suffixient set-based solutions.

From a theoretical perspective, the work strengthens the case for $\chi$ as an operational repetitiveness measure, since efficient suffixient set computation is now feasible for industrial-scale applications. The advancements also lay groundwork for further integration with streaming paradigms, enabling suffix-array–based construction in compressed or out-of-core settings with main memory proportional to compressibility characteristics, not raw input size.

Future Directions

Several open points emerge from this study. One immediate direction is the adaptation of the FM algorithm to support one-pass execution by means of dynamic PSV/NSV computation for run-breaks, potentially yielding even better practical performance. Coupling the presented approach with prefix-free parsing (PFP) methods opens new avenues for optimizing construction in massively repeated datasets, typical of modern pan-genomic and codebase archiving efforts.

Reducing main memory consumption for box height maintenance—currently $r$0 in worst case, though typically lower—remains a goal for fully streaming, compressibility-sensitive implementations. This would further solidify the method's practical utility for ultra-large, highly repetitive data.

Conclusion

The work delivers the first practical, linear-time, one-pass, and implemented algorithm for constructing smallest suffixient sets, empirically demonstrating dominance across meaningful tradeoffs in space and time. It effectively broadens the accessible spectrum for compressed pattern matching in highly repetitive data, advancing both the theoretical and applied frontiers of string indexing and large-scale text analytics.