Relative Lempel-Ziv Compression

• Relative Lempel-Ziv is a dictionary-based compression method that parses target sequences into maximal matching phrases using a fixed reference, ideal for genomic and archival data.
• It employs a greedy left-to-right factorization to ensure fast random access and effective self-indexing while maintaining a simple, bounded-depth structure.
• Recent enhancements like RLZAP and HRLZ adapt the original scheme to handle mismatches and structural variations, offering improved compression ratios and query performance.

Searching arXiv for relevant papers on Relative Lempel-Ziv and its variants. Relative Lempel-Ziv (RLZ) is a dictionary-based compression method in which a target string is compressed relative to a fixed reference string by parsing the target into substrings that occur in the reference. In the genomic setting, RLZ exploits the fact that within a species, individual genomes are often nearly identical; in web and archive settings, it exploits large-scale repetition across documents. The method is characterized by a greedy left-to-right factorization, fast random access, and a design space centered on reference selection, phrase representation, and data structures for search and access. Subsequent work extended the basic scheme to compressed self-indexing, archive compression, adaptive pointer coding, hierarchical multi-reference compression, small-space approximation to LZ77, and conversion from RLZ parses to exact LZ77 parses (Gagie et al., 2011, Cox et al., 2016, Bille et al., 2022, Kosolobov et al., 2019, Gagie, 2022).

1. Definition and core parsing model

In its canonical form, RLZ fixes a reference string and parses a target string greedily into phrases that are longest prefixes of the remaining target suffix that occur as substrings of the reference. For a reference genome $G[1..n]$ and target $T[1..N]$, the RLZ parse is the unique factorization

$T = T_1 T_2 \dots T_r$

such that each phrase $T_i$ is the longest prefix of the remaining suffix that occurs in $G$, and is represented by a pair $(\mathrm{pos}(i), \ell(i))$ (Gagie et al., 2011). Equivalent formulations are given for a general text $T[1..n]$ relative to a fixed reference $R[1..\ell]$, where each copying phrase is encoded as $(p_j,\ell_j)$, and if no non-empty match exists the encoder emits a literal (Kosolobov et al., 2019).

A closely related formulation appears in large web collections, where the dictionary $D[1..m]$ is a fixed global sample with $T[1..N]$0. The text is factorized into phrases $T[1..N]$1, where each phrase is either the longest string starting at the current position that occurs contiguously in $T[1..N]$2, or a single literal character if no nonempty prefix occurs in $T[1..N]$3 (Hoobin et al., 2011). In archive compression, the same principle is stated for a concatenation $T[1..N]$4 of documents and a semi-static dictionary $T[1..N]$5, with factors represented either as dictionary copies $T[1..N]$6 or literal bytes (Petri et al., 2016).

The essential distinction from classical LZ77 is that RLZ restricts copying to a fixed external reference rather than the already-seen prefix of the text itself. This restriction yields a depth-1 structure with no recursive nesting of pointers, which is central to its access and indexing properties (Gagie et al., 2011). A plausible implication is that RLZ trades some of the adaptivity of LZ77 for better control over memory usage and simpler access semantics.

2. Encoding, references, and baseline data structures

A standard implementation of RLZ builds an index on the reference and then scans the target once. One suffix-array–based pseudocode uses a loop that, at each position $T[1..N]$7, finds the longest match of $T[1..N]$8 in $T[1..N]$9, emits either $T = T_1 T_2 \dots T_r$0 or a literal, and advances by $T = T_1 T_2 \dots T_r$1 (Kosolobov et al., 2019). In the self-index formulation, the reference $T = T_1 T_2 \dots T_r$2 is stored in a compressed suffix array supporting pattern search in $T = T_1 T_2 \dots T_r$3 time and random access in $T = T_1 T_2 \dots T_r$4 time, while the phrase array occupies $T = T_1 T_2 \dots T_r$5 bits (Gagie et al., 2011).

Reference construction is application-dependent. In genomic databases, the first genome may be selected as the reference, with the remaining genomes parsed against it (Gagie et al., 2011). In web collections and archives, a representative sample is built by periodically extracting substrings from the corpus and concatenating them into a dictionary (Hoobin et al., 2011, Petri et al., 2016). One scalable recipe samples substrings at regular stride to obtain a dictionary of size $T = T_1 T_2 \dots T_r$6, and then builds a suffix array on that dictionary in $T = T_1 T_2 \dots T_r$7 time and $T = T_1 T_2 \dots T_r$8 words of RAM (Hoobin et al., 2011).

For random access in the basic RLZ representation, one stores phrase boundaries in a bitvector and stores the phrase pointers in an array. Given a position $T = T_1 T_2 \dots T_r$9, one identifies the phrase covering $T_i$0, computes the offset within that phrase, and returns the corresponding symbol from the reference (Cox et al., 2016, Gagie et al., 2011). In the self-index setting, if $T_i$1 falls inside the reference $T_i$2, access is answered directly by the compressed suffix array; if $T_i$3 falls inside a target $T_i$4, one locates the covering phrase and returns $T_i$5, for total time $T_i$6 (Gagie et al., 2011).

The memory profile is governed primarily by the reference length. RLZ indexes only the reference, so memory is $T_i$7 for a suffix-array or FM-index–based implementation, plus streaming working memory and the output buffer (Kosolobov et al., 2019). This bounded-reference property is one of RLZ’s defining practical features.

3. Random access, search, and compressed self-indexing

RLZ was developed not only as a compressor but also as a basis for search over highly repetitive collections. In the genomic self-index, one stores the reference genome $T_i$8 in a compressed self-indexed form and parses each additional genome against $T_i$9. Search is decomposed into occurrences in $G$0, secondary occurrences in parsed targets, and primary occurrences crossing phrase boundaries in targets (Gagie et al., 2011).

The core structures for this self-index include the phrase pointers, a 2-sided range-report structure for secondary occurrences, a 2D range-report structure for primary occurrences, and an FM-index on the sequence of phrase dictionary ranks. Gagie et al. show that $G$1 can be stored in

$G$2

bits (Gagie et al., 2011). Random access to any position takes $G$3 time, substring extraction takes $G$4, and pattern search has overall time

$G$5

where $G$6, $G$7, and $G$8 denote occurrences in the reference, primary occurrences in targets, and secondary occurrences in targets, respectively (Gagie et al., 2011).

This indexing perspective clarifies why RLZ became important in genomic databases. Because each phrase copies only from the fixed reference, the depth of nesting is 1, and the search structures operate over phrase boundaries and copied intervals rather than recursively traversing a general LZ77 parse (Gagie et al., 2011). This suggests that RLZ is particularly well suited to workloads that require both compression and query support.

4. Phrase variants and the RLZ family

The original greedy RLZ formulation is effective but sensitive to small local differences. In particular, single-character substitutions force two phrase breaks and inflate the number of phrases $G$9 (Cox et al., 2016). A first refinement, due to Deorowicz and Grabowski, allows each phrase to end with one mismatch character. In this representation, each phrase consists of a maximal match followed by a mismatch literal; the data structures store the starting positions in the reference, the mismatch characters, and a bitvector marking phrase ends (Cox et al., 2016). The effect is that isolated single substitutions become one phrase plus one mismatch literal instead of two pure-copy phrases, thereby halving the overhead on isolated SNPs (Cox et al., 2016).

A second refinement, due to Ferrada et al., uses relative pointers rather than absolute source positions. Defining

$(\mathrm{pos}(i), \ell(i))$0

where $(\mathrm{pos}(i), \ell(i))$1 is the source position in the reference and $(\mathrm{pos}(i), \ell(i))$2 is the phrase start in the target, yields an array of relative pointers that often remains identical over many consecutive phrases. These relative pointers can be run-length compressed by partitioning $(\mathrm{pos}(i), \ell(i))$3 into maximal runs of equal values, storing the run values and a bitvector of run starts (Cox et al., 2016). When target and reference differ only by isolated SNPs, the relative-pointer array exhibits long equal runs, reducing space (Cox et al., 2016).

RLZAP generalizes this line of work to handle short insertions, deletions, and multi-character substitutions. It precomputes matching statistics $(\mathrm{pos}(i), \ell(i))$4 and $(\mathrm{pos}(i), \ell(i))$5, then parses left to right using two interleaved steps: an adaptive step and an explicit step. Adaptive phrases encode the current pointer as a delta from the previous explicit pointer using $(\mathrm{pos}(i), \ell(i))$6; explicit phrases store the full pointer in $(\mathrm{pos}(i), \ell(i))$7 bits (Cox et al., 2016). A bounded lookahead allows the parser to skip over short insertions or deletions and restart under a small pointer shift, while runs of trailing unmatched characters are treated as literals at the end of phrases (Cox et al., 2016).

For fast random access, RLZAP maintains a bitvector $(\mathrm{pos}(i), \ell(i))$8 for phrase starts, a bitvector $(\mathrm{pos}(i), \ell(i))$9 marking explicit phrases, tables for explicit and adaptive pointers, and concatenated literal storage with a rank-sum structure on the per-phrase literal counts. Retrieval proceeds by locating the phrase containing a queried position, determining whether the offset lies in the literal tail, and if not computing the source position in the reference via the recovered relative pointer (Cox et al., 2016).

The cumulative effect of these variants is a family of encodings that preserve the central RLZ paradigm while adapting phrase representation to the mutation patterns of the data.

5. Performance characteristics and empirical trade-offs

Empirical results show that RLZ’s effectiveness depends strongly on the similarity between target and reference, the reference construction method, the phrase representation, and the access model. In early genomic experiments, vanilla RLZ achieved compression ratios on the order of $T[1..n]$0–$T[1..n]$1, corresponding to $T[1..n]$2–$T[1..n]$3 bits per base, with random access to an arbitrary base in $T[1..n]$4–$T[1..n]$5 (Gagie et al., 2011). In a later RLZ-based self-index, storing $T[1..n]$6 human genomes in $T[1..n]$7–$T[1..n]$8 GB instead of $T[1..n]$9 GB raw and pattern search in $R[1..\ell]$0–$R[1..\ell]$1 ms for microbial-size patterns were reported as representative outcomes (Gagie et al., 2011).

In web collections, RLZ demonstrates a strong trade-off between dictionary size, compression ratio, and decode throughput. On GOV2, with dictionary fraction $R[1..\ell]$2, the method achieved compression ratio $R[1..\ell]$3 and $R[1..\ell]$4 decoded documents per second; with $R[1..\ell]$5, it achieved $R[1..\ell]$6 and $R[1..\ell]$7 documents per second (Hoobin et al., 2011). On Wikipedia, $R[1..\ell]$8 gave $R[1..\ell]$9 and $(p_j,\ell_j)$0 documents per second, while $(p_j,\ell_j)$1 gave $(p_j,\ell_j)$2 and $(p_j,\ell_j)$3 documents per second (Hoobin et al., 2011). The same study states that standard block compression with ZLIB or LZMA at $(p_j,\ell_j)$4 MB block sizes reached $(p_j,\ell_j)$5–$(p_j,\ell_j)$6 but only $(p_j,\ell_j)$7 docs/s (Hoobin et al., 2011).

In archive compression, random-access latency is explicitly modeled as

$(p_j,\ell_j)$8

For HDD, the dominant factor affecting access speed is the compression rate achieved, even when this involves larger dictionaries and larger blocks; on SSD the same effects are present, but not as markedly (Petri et al., 2016). On GOV2 in RANDOM mode, RLZ-ZZ with $(p_j,\ell_j)$9 MiB and $D[1..m]$0 KiB achieved about $D[1..m]$1 compression and about $D[1..m]$2 docs/s on HDD, while RLZ-PV with $D[1..m]$3 MiB and $D[1..m]$4 KiB achieved about $D[1..m]$5 and about $D[1..m]$6 docs/s on SSD (Petri et al., 2016).

RLZAP reports improvements over previous RLZ variants on genome collections and related structures. On three yeast or bacterial genome collections and DLCP arrays of three human genomes, target size decreased from $D[1..m]$7 to $D[1..m]$8 MiB on Cere, from $D[1..m]$9 to $T[1..N]$00 MiB on E. Coli, from $T[1..N]$01 to $T[1..N]$02 MiB on Para, and from $T[1..N]$03 to $T[1..N]$04 MiB on DLCP (Cox et al., 2016). For substrings of length at least $T[1..N]$05 on DNA, RLZAP is approximately $T[1..N]$06 slower per character than the compared RLZ variant, for example $T[1..N]$07 ns to $T[1..N]$08 ns on Cere, but still at most about $T[1..N]$09 ns per character; on DLCP it is slightly faster, for example $T[1..N]$10 ns to $T[1..N]$11 ns for length at least $T[1..N]$12 (Cox et al., 2016). The same work states a space–time trade-off of about $T[1..N]$13–$T[1..N]$14 less space at the cost of a modest constant-factor slowdown on pure DNA, together with $T[1..N]$15 fewer L2/L3 misses on DLCP (Cox et al., 2016).

6. Extensions: approximation, conversion, and hierarchical references

A major theoretical and algorithmic extension is ReLZ, which uses RLZ as a first-stage preprocessor to approximate the LZ77 parsing in small space. RLZ first produces a sequence of phrases; these phrases are treated as metasymbols, and a second-level LZ parse is computed on the shorter metasymbol string. The resulting parse is then translated back to the original sequence (Kosolobov et al., 2019). ReLZ achieves the entropy bound

$T[1..N]$16

for $T[1..N]$17, and the paper also proves a lower bound showing that the number of phrases in ReLZ can be $T[1..N]$18 times larger than the number of phrases in LZ (Kosolobov et al., 2019). Experiments report an approximation factor below $T[1..N]$19 in all tested scenarios, and sometimes below $T[1..N]$20, relative to the size of LZ (Kosolobov et al., 2019).

A complementary result addresses exact conversion from RLZ to LZ77. Given a text $T[1..N]$21 prefixed by a reference $T[1..N]$22 and the $T[1..N]$23-phrase RLZ parse of $T[1..N]$24 with respect to $T[1..N]$25, one can compute the exact LZ77 parse of $T[1..N]$26 in $T[1..N]$27 time and $T[1..N]$28 total space (Gagie, 2022). The construction uses random access over the RLZ parse, Karp–Rabin fingerprints, sorted co-lexicographic and lexicographic orders of RLZ phrases and suffixes, and a static 2D range-minimum query structure (Gagie, 2022). This result is significant because it separates the small-space construction advantages of RLZ from the stronger parsing model of LZ77.

Another extension replaces the single reference with a hierarchy of references. In Hierarchical Relative Lempel-Ziv Compression, a rooted tree is formed on the strings, each non-root string is compressed using RLZ with its parent as reference, and only the root is stored in plain text (Bille et al., 2022). Decompression traverses the tree in BFS order starting at the root. The hierarchy is optimized by assigning weights $T[1..N]$29 equal to the number of phrases in the RLZ parsing of source and destination strings, and then computing a minimum-weight spanning arborescence in the complete directed graph (Bille et al., 2022). To reduce the cost of constructing this graph, a sparse graph derived using locality sensitive hashing can be used instead (Bille et al., 2022).

The empirical outcome of this hierarchical design is a twofold improvement in compression on bacterial genome data sets, with negligible effect on decompression time compared to the standard single reference approach (Bille et al., 2022). On the E.coli set, the total number of phrases decreased from $T[1..N]$30 for single-reference RLZ to $T[1..N]$31 for optimal HRLZ and $T[1..N]$32 for approximate HRLZ, while decompression time was $T[1..N]$33, $T[1..N]$34, and $T[1..N]$35 seconds, respectively (Bille et al., 2022).

7. Theoretical interpretation, applications, and scope

RLZ sits at the intersection of practical repetitive-text compression and entropy-oriented analysis. One theoretical perspective is that RLZ provides a tunable trade-off between memory usage and compression quality through the size and quality of the reference (Kosolobov et al., 2019). Another is given by a chain-rule analysis of Lempel-Ziv complexity for paired sequences. The 2025 work on LZ78 complexity states that the joint per-symbol LZ78 complexity of a pair $T[1..N]$36 can be sandwiched between the per-symbol sum of $T[1..N]$37 and $T[1..N]$38, up to vanishing redundancies, and explicitly notes that this underlies the theoretical justification for universal compression of a target $T[1..N]$39 relative to a reference $T[1..N]$40, so-called Relative Lempel-Ziv compression (Merhav, 15 Jun 2025). This is a theoretical foundation for relative or conditional compression in an individual-sequence setting.

The principal application domain is genomic databases. RLZ was introduced in response to the prospect of databases containing thousands of genomes, where individuals’ genomes are almost exact copies of each other; the method stores one genome uncompressed or as an FM-index and compresses the others relative to it (Gagie et al., 2011). RLZAP extends this to variant-rich collections by handling substitutions, short indels, and multi-character mismatches while retaining fast random access (Cox et al., 2016). HRLZ addresses the case where one global reference is insufficiently representative for all strings in the collection (Bille et al., 2022).

RLZ is also applied beyond genomics. In large document repositories and archives, a representative sample can serve as a semi-static dictionary that enables random-access retrieval without decompressing an entire adaptive-compression block (Hoobin et al., 2011, Petri et al., 2016). RLZAP explicitly notes that any collection of very similar sequences with small local shifts, such as versioned text corpora and software repositories, can benefit from its adaptivity (Cox et al., 2016).

A common misconception is that RLZ is merely a weaker form of LZ77. The literature instead presents it as a distinct design point: a depth-1 relative parser with bounded indexing cost, strong random-access support, and specialized extensions for repetitive collections (Gagie et al., 2011, Kosolobov et al., 2019). Another misconception is that a single-reference design is intrinsic to RLZ; hierarchical RLZ shows that multiple references can be integrated coherently while preserving linear-time decompression (Bille et al., 2022). Conversely, the lower bound for ReLZ shows that RLZ-based approximations to LZ77 cannot always guarantee a constant-factor approximation in phrase count (Kosolobov et al., 2019). The resulting picture is not of a single algorithm but of a mature family of relative parsing methods whose behavior is governed by the interaction among reference choice, phrase model, and access requirements.