Compact multi-text index for circular Cartesian tree matching

Abstract: Cartesian tree matching (CTM) is a structural pattern matching approach that identifies sequences with the same Cartesian tree topology, making it suitable for data with natural variability where exact comparisons carry little semantic meaning. While theoretical algorithms for CTM have been studied extensively, systematic empirical evaluations of practical implementations remain rare. This article presents an implementation of the Cartesian Extended Burrows-Wheeler Transform (ceBWT), a BWT-based index structure for CTM. The implementation supports both a dynamically extendable and a statically compressed index variant.

• The paper introduces a ceBWT index that efficiently supports circular Cartesian tree matching over multiple texts in noisy data applications like genomics and music.
• It develops both dynamic and static variants, with the static version achieving O(m) query speed and a space efficiency of 0.50 bytes per symbol.
• Engineering improvements such as reduced replication overhead and multithreading optimizations enable robust and scalable index construction.

Compact Multi-Text Index for Circular Cartesian Tree Matching: An Authoritative Summary

Introduction and Motivation

The paper "Compact multi-text index for circular Cartesian tree matching" (2606.18696) investigates practical and empirical aspects of Cartesian tree matching (CTM) over multiple sequences, focusing on scenarios with natural variability in data where strict value matching is semantically weak (e.g., genomics and music). CTM operates on the recursive structure of extrema in strings, abstracting away both exact values and finer-grained rank information, distinguishing it from order-preserving matching (OPM) and traditional exact matching models. Despite recent developments in efficient theoretical CTM indexing algorithms, comprehensive benchmark-driven evaluations and scaling analyses have been absent. This work fills that gap by presenting an implementation of the Cartesian Extended Burrows–Wheeler Transform (ceBWT), a BWT-based index tailored for circular CTM matching, and systematically analyzing its construction, query, and scalability properties.

Theoretical Foundations and Index Variants

Structural Matching Models

The paper details CTM, where matching is defined via isomorphic Cartesian tree topologies derived recursively from minima locations. CTM is strictly coarser than OPM; it focuses solely on recursive extremal structure, making it uniquely suited for applications where measurement noise or domain-specific structural shapes (e.g., musical motifs, genomic motifs) are more relevant than absolute values or pairwise order.

Index Construction

The ceBWT index extends previous CTM indexes by supporting both statically compressed and incrementally extendable dynamic variants. It operates by storing conjugates (all cyclic rotations) of input sequences in $\omega$-preorder so that CTM matches correspond to contiguous intervals. These intervals are efficiently located through backward search using either dynamic wavelet trees (for the dynamic index) or collections of static bit vectors (for the static index encoding $F$ and $L$ arrays).

• Dynamic ceBWT: Supports on-the-fly merging of new sequences, storing arrays in dynamic wavelet trees. Pattern matching is achieved in $O(m \frac{\lg\,\sigma\,\lg\,n}{\lg\,\lg\,n})$ with $O(n\lg\sigma)$ space.
• Static ceBWT: Compact encoding using bit vector collections, matches in $O(m)$ with space $3n+o(n)$ bits. Incremental merges are disabled but storage efficiency is maximized.

A major engineering contribution is the input replication reduction: construction uses twofold copies for initial texts and threefold for inserts, improving efficiency over fourfold replication without violating correctness. Multithreading optimizations are also introduced.

Figure 1: Main data structures used by the ceBWT index in both dynamic and static variants.

Empirical Evaluation: Construction, Query, and Scalability

Construction Throughput and Index Size

Empirical benchmarks on genomic (E. coli K-12 MG1655, rice Oryza sativa, various NCBI genomes) and music datasets (tokenized MIDI) demonstrate consistent scaling, with throughput and memory usage responding to sequence count and length as expected from theoretical asymptotics. The static index achieves best-in-class space efficiency (0.50 bytes/symbol), outperforming RLCSA and SDSL-based baselines, while dynamic ceBWT incurs additional space overhead due to auxiliary structures and dynamic wavelet trees.

Figure 2: Throughput (symbols/second) during index construction, comparing ceBWT variants and baselines.

Figure 3: Memory footprint of the main static and dynamic index data structures for repetitive versus non-repetitive genomes of identical length.

Pattern Search Performance

ceBWT supports fast repeated pattern matching once the index is constructed, with static ceBWT queries being substantially faster ($O(m)$) than dynamic ceBWT queries, and competitive with exact-matching (FM-index, RLCSA) and OPM baselines. Query speed for static ceBWT remains robust at increasing pattern lengths and outperforms the compact order-preserving FM-index on the evaluated MIDI workloads, despite CTM solving a weaker structural matching problem.

Figure 4: Scaling of the pattern search using the dynamic and the static index structure for different pattern lengths, measured on 50 patterns per length on 16 real genomes.

Figure 5: Throughput (symbols/second) for pattern search across algorithm variants.

Impact of Repetitiveness

Highly repetitive genomic subsequences exhibit moderate improvements in construction speed and modest differences in memory footprint, but a pronounced increase in match frequency, supporting the claim that CTM is sensitive to structural repetitiveness.

Figure 6: Number of matches found for repetitive versus non-repetitive genomes across varying pattern lengths.

Algorithmic Comparison and Practical Implications

ceBWT indices are contrasted with:

• KMP-based CTM baselines: Traditional CTM search algorithms compute matches ad hoc, requiring linear time for each pattern. Once a ceBWT index is built, repeated pattern searches are orders of magnitude faster, validating the practical value for applications with frequent or complex query demands.
• Exact (FM-index, RLCSA, r-index) and OPM indices: ceBWT achieves competitive or superior space efficiency on multi-text workloads, and query speeds are within a practical range, despite matching a more general structural relation.

  Figure 7: Memory-footprint comparison of ceBWT, SDSL, r-index, RLCSA, and OPM-FM index on tokenized MIDI files.

  

  Figure 8: Pattern-search runtime comparison among ceBWT and baselines for 2,000 tokenized MIDI patterns.

Implementation Details and Engineering Considerations

The implementation leverages dynamic and static succinct data structures (dynamic wavelet trees, SDSL bit vectors), with extensive benchmarking and profiling to identify and remedy scalability bottlenecks. Threaded index construction yields performance improvements up to a small number of workers; further scaling is limited by sequential merge overheads, emphasizing a need for future algorithmic parallelization beyond naive concurrency.

The index supports multiple storage variants (full, CA, minimal), controlling the precision and semantics of pattern count, extract, and remove operations, allowing trade-offs between space and query granularity.

Conclusion

The ceBWT index presents a robust, scalable, and space-efficient solution for circular Cartesian tree matching over multi-text collections, with practical utility in genomics, music information retrieval, and domains where structural patterns are paramount. Static ceBWT achieves high storage efficiency and fast query speeds, outperforming competing structural and exact-matching indexes in several conditions. Dynamic ceBWT enables incremental updates but at a cost in scalability and space. Repetitiveness strongly influences match counts, with modest effects on memory and throughput.

Implications and Future Directions

Theoretical implications include confirmation of CTM's utility as a structural matching model and validation of ceBWT's algorithmic scalability. Practically, ceBWT offers a persistent index for repeated queries, enabling real-time analytics in music and genomics. Future developments should focus on optimizing dynamic structures (rank/select performance, incremental updates), harnessing domain-specific alphabets, developing hybrid static-dynamic architectures, and advancing parallel indexing algorithms. Deeper study into the relationship between repetitiveness and CTM index performance is warranted, particularly for highly repetitive, real-world datasets.

See (2606.18696) for detailed benchmarks and further discussion.