---
title: Relative Lempel-Ziv Compression
url: https://www.emergentmind.com/topics/relative-lempel-ziv
type: topic
---

# Relative Lempel-Ziv Compression

Searching arXiv for relevant papers on Relative Lempel-Ziv and its variants.
Relative Lempel-Ziv (RLZ) is a dictionary-based compression method in which a target string is compressed relative to a fixed reference string by parsing the target into substrings that occur in the reference. In the genomic setting, RLZ exploits the fact that within a species, individual genomes are often nearly identical; in web and archive settings, it exploits large-scale repetition across documents. The method is characterized by a greedy left-to-right factorization, fast random access, and a design space centered on reference selection, phrase representation, and data structures for search and access. Subsequent work extended the basic scheme to compressed self-indexing, archive compression, adaptive pointer coding, hierarchical multi-reference compression, small-space approximation to LZ77, and conversion from RLZ parses to exact LZ77 parses [1111.1355] [1605.04421] [2208.11371] [1903.01909] [2211.13254].

## 1. Definition and core parsing model

In its canonical form, RLZ fixes a reference string and parses a target string greedily into phrases that are longest prefixes of the remaining target suffix that occur as substrings of the reference. For a reference genome \(G[1..n]\) and target \(T[1..N]\), the RLZ parse is the unique factorization
\(T = T_1 T_2 \dots T_r\)
such that each phrase \(T_i\) is the longest prefix of the remaining suffix that occurs in \(G\), and is represented by a pair \((\mathrm{pos}(i), \ell(i))\) [1111.1355]. Equivalent formulations are given for a general text \(T[1..n]\) relative to a fixed reference \(R[1..\ell]\), where each copying phrase is encoded as \((p_j,\ell_j)\), and if no non-empty match exists the encoder emits a literal [1903.01909].

A closely related formulation appears in large web collections, where the dictionary \(D[1..m]\) is a fixed global sample with \(m<n\). The text is factorized into phrases \(w_1w_2\dots w_k=T\), where each phrase is either the longest string starting at the current position that occurs contiguously in \(D\), or a single literal character if no nonempty prefix occurs in \(D\) [1106.2587]. In archive compression, the same principle is stated for a concatenation \(C\) of documents and a semi-static dictionary \(D\), with factors represented either as dictionary copies \((\mathrm{off}_i,\ell_i)\) or literal bytes [1602.08829].

The essential distinction from classical LZ77 is that RLZ restricts copying to a fixed external reference rather than the already-seen prefix of the text itself. This restriction yields a depth-1 structure with no recursive nesting of pointers, which is central to its access and indexing properties [1111.1355]. A plausible implication is that RLZ trades some of the adaptivity of LZ77 for better control over memory usage and simpler access semantics.

## 2. Encoding, references, and baseline data structures

A standard implementation of RLZ builds an index on the reference and then scans the target once. One suffix-array–based pseudocode uses a loop that, at each position \(i\), finds the longest match of \(T[i..]\) in \(R\), emits either \((p_j,\ell_j)\) or a literal, and advances by \(\max(1,\ell_j)\) [1903.01909]. In the self-index formulation, the reference \(G\) is stored in a compressed suffix array supporting pattern search in \(O(m\log \sigma + occ\log^\epsilon n)\) time and random access in \(O(\log^\epsilon n)\) time, while the phrase array occupies \(O(r\log n)\) bits [1111.1355].

Reference construction is application-dependent. In genomic databases, the first genome may be selected as the reference, with the remaining genomes parsed against it [1111.1355]. In web collections and archives, a representative sample is built by periodically extracting substrings from the corpus and concatenating them into a dictionary [1106.2587] [1602.08829]. One scalable recipe samples substrings at regular stride to obtain a dictionary of size \(m\), and then builds a suffix array on that dictionary in \(O(m\log m)\) time and \(O(m)\) words of RAM [1106.2587].

For random access in the basic RLZ representation, one stores phrase boundaries in a bitvector and stores the phrase pointers in an array. Given a position \(j\), one identifies the phrase covering \(j\), computes the offset within that phrase, and returns the corresponding symbol from the reference [1605.04421] [1111.1355]. In the self-index setting, if \(i\) falls inside the reference \(G\), access is answered directly by the compressed suffix array; if \(i\) falls inside a target \(T\), one locates the covering phrase and returns \(G[\mathrm{pos}(k)+\delta]\), for total time \(O(\log^\epsilon n)\) [1111.1355].

The memory profile is governed primarily by the reference length. RLZ indexes only the reference, so memory is \(O(\ell)\) for a suffix-array or FM-index–based implementation, plus streaming working memory and the output buffer [1903.01909]. This bounded-reference property is one of RLZ’s defining practical features.

## 3. Random access, search, and compressed self-indexing

RLZ was developed not only as a compressor but also as a basis for search over highly repetitive collections. In the genomic self-index, one stores the reference genome \(G\) in a compressed self-indexed form and parses each additional genome against \(G\). Search is decomposed into occurrences in \(G\), secondary occurrences in parsed targets, and primary occurrences crossing phrase boundaries in targets [1111.1355].

The core structures for this self-index include the phrase pointers, a 2-sided range-report structure for secondary occurrences, a 2D range-report structure for primary occurrences, and an FM-index on the sequence of phrase dictionary ranks. Gagie et al. show that \(G+T\) can be stored in
\[
(1 + 1/\epsilon)\,n\,H_k(G)\;+\;O\bigl(r(\log n + \log^{1+\epsilon} r)\bigr)\;+\;O(n)
\]
bits [1111.1355]. Random access to any position takes \(O(\log^\epsilon n)\) time, substring extraction takes \(O((j-i+1)\log^\epsilon n)\), and pattern search has overall time
\[
O\bigl((m+occ_0)\log^\epsilon n + occ_1 + occ_2\bigr)
\]
where \(occ_0\), \(occ_1\), and \(occ_2\) denote occurrences in the reference, primary occurrences in targets, and secondary occurrences in targets, respectively [1111.1355].

This indexing perspective clarifies why RLZ became important in genomic databases. Because each phrase copies only from the fixed reference, the depth of nesting is 1, and the search structures operate over phrase boundaries and copied intervals rather than recursively traversing a general LZ77 parse [1111.1355]. This suggests that RLZ is particularly well suited to workloads that require both compression and query support.

## 4. Phrase variants and the RLZ family

The original greedy RLZ formulation is effective but sensitive to small local differences. In particular, single-character substitutions force two phrase breaks and inflate the number of phrases \(t\) [1605.04421]. A first refinement, due to Deorowicz and Grabowski, allows each phrase to end with one mismatch character. In this representation, each phrase consists of a maximal match followed by a mismatch literal; the data structures store the starting positions in the reference, the mismatch characters, and a bitvector marking phrase ends [1605.04421]. The effect is that isolated single substitutions become one phrase plus one mismatch literal instead of two pure-copy phrases, thereby halving the overhead on isolated SNPs [1605.04421].

A second refinement, due to Ferrada et al., uses relative pointers rather than absolute source positions. Defining
\[
D[i] = q_i - p_i,
\]
where \(q_i\) is the source position in the reference and \(p_i\) is the phrase start in the target, yields an array of relative pointers that often remains identical over many consecutive phrases. These relative pointers can be run-length compressed by partitioning \(D\) into maximal runs of equal values, storing the run values and a bitvector of run starts [1605.04421]. When target and reference differ only by isolated SNPs, the relative-pointer array exhibits long equal runs, reducing space [1605.04421].

RLZAP generalizes this line of work to handle short insertions, deletions, and multi-character substitutions. It precomputes matching statistics \(\mathrm{MatchPtr}(i)\) and \(\mathrm{MatchLen}(i)\), then parses left to right using two interleaved steps: an adaptive step and an explicit step. Adaptive phrases encode the current pointer as a delta from the previous explicit pointer using \(\Delta\mathrm{bits}\); explicit phrases store the full pointer in \(\lceil \log m \rceil\) bits [1605.04421]. A bounded lookahead allows the parser to skip over short insertions or deletions and restart under a small pointer shift, while runs of trailing unmatched characters are treated as literals at the end of phrases [1605.04421].

For fast random access, RLZAP maintains a bitvector \(P\) for phrase starts, a bitvector \(E\) marking explicit phrases, tables for explicit and adaptive pointers, and concatenated literal storage with a rank-sum structure on the per-phrase literal counts. Retrieval proceeds by locating the phrase containing a queried position, determining whether the offset lies in the literal tail, and if not computing the source position in the reference via the recovered relative pointer [1605.04421].

The cumulative effect of these variants is a family of encodings that preserve the central RLZ paradigm while adapting phrase representation to the mutation patterns of the data.

## 5. Performance characteristics and empirical trade-offs

Empirical results show that RLZ’s effectiveness depends strongly on the similarity between target and reference, the reference construction method, the phrase representation, and the access model. In early genomic experiments, vanilla RLZ achieved compression ratios on the order of \(30\)–\(60\times\), corresponding to \(1.7\)–\(3\) bits per base, with random access to an arbitrary base in \(\le 1\)–\(2\ \mu s\) [1111.1355]. In a later RLZ-based self-index, storing \(36\) human genomes in \(1\)–\(2\) GB instead of \(\sim 100\) GB raw and pattern search in \(0.1\)–\(0.5\) ms for microbial-size patterns were reported as representative outcomes [1111.1355].

In web collections, RLZ demonstrates a strong trade-off between dictionary size, compression ratio, and decode throughput. On GOV2, with dictionary fraction \(\delta=0.001\), the method achieved compression ratio \(11.0\%\) and \(9\,700\) decoded documents per second; with \(\delta=0.005\), it achieved \(9.3\%\) and \(18\,700\) documents per second [1106.2587]. On Wikipedia, \(\delta=0.002\) gave \(12.1\%\) and \(11\,200\) documents per second, while \(\delta=0.010\) gave \(9.7\%\) and \(13\,360\) documents per second [1106.2587]. The same study states that standard block compression with ZLIB or LZMA at \(1\) MB block sizes reached \(CR\approx 11\)–\(18\%\) but only \(\approx 2\,000\) docs/s [1106.2587].

In archive compression, random-access latency is explicitly modeled as
\[
T_{\mathrm{query}} = T_{\mathrm{seek}} + T_{\mathrm{transfer}} + T_{\mathrm{decode}}.
\]
For HDD, the dominant factor affecting access speed is the compression rate achieved, even when this involves larger dictionaries and larger blocks; on SSD the same effects are present, but not as markedly [1602.08829]. On GOV2 in RANDOM mode, RLZ-ZZ with \(|D|=256\) MiB and \(B_{\max}=64\) KiB achieved about \(17\%\) compression and about \(140\) docs/s on HDD, while RLZ-PV with \(|D|=1024\) MiB and \(B_{\max}=64\) KiB achieved about \(14.5\%\) and about \(1800\) docs/s on SSD [1602.08829].

RLZAP reports improvements over previous RLZ variants on genome collections and related structures. On three yeast or bacterial genome collections and DLCP arrays of three human genomes, target size decreased from \(9.16\) to \(7.61\) MiB on Cere, from \(30.47\) to \(21.51\) MiB on E. Coli, from \(15.57\) to \(10.49\) MiB on Para, and from \(1745.33\) to \(1173.81\) MiB on DLCP [1605.04421]. For substrings of length at least \(4\) on DNA, RLZAP is approximately \(30\%\) slower per character than the compared RLZ variant, for example \(4.4\) ns to \(5.7\) ns on Cere, but still at most about \(10\) ns per character; on DLCP it is slightly faster, for example \(20.5\) ns to \(19.0\) ns for length at least \(16\) [1605.04421]. The same work states a space–time trade-off of about \(20\)–\(30\%\) less space at the cost of a modest constant-factor slowdown on pure DNA, together with \(36\%\) fewer L2/L3 misses on DLCP [1605.04421].

## 6. Extensions: approximation, conversion, and hierarchical references

A major theoretical and algorithmic extension is ReLZ, which uses RLZ as a first-stage preprocessor to approximate the LZ77 parsing in small space. RLZ first produces a sequence of phrases; these phrases are treated as metasymbols, and a second-level LZ parse is computed on the shorter metasymbol string. The resulting parse is then translated back to the original sequence [1903.01909]. ReLZ achieves the entropy bound
\[
n H_k + o(n\log \sigma)
\]
for \(k=o(\log_\sigma n)\), and the paper also proves a lower bound showing that the number of phrases in ReLZ can be \(\Omega(\log n)\) times larger than the number of phrases in LZ [1903.01909]. Experiments report an approximation factor below \(2.0\) in all tested scenarios, and sometimes below \(1.05\), relative to the size of LZ [1903.01909].

A complementary result addresses exact conversion from RLZ to LZ77. Given a text \(T[1..n]\) prefixed by a reference \(R=T[1..\ell]\) and the \(z'\)-phrase RLZ parse of \(T[\ell+1..n]\) with respect to \(R\), one can compute the exact LZ77 parse of \(T\) in \(n\,\mathrm{polylog}(n)\) time and \(O(\ell+z')\) total space [2211.13254]. The construction uses random access over the RLZ parse, Karp–Rabin fingerprints, sorted co-lexicographic and lexicographic orders of RLZ phrases and suffixes, and a static 2D range-minimum query structure [2211.13254]. This result is significant because it separates the small-space construction advantages of RLZ from the stronger parsing model of LZ77.

Another extension replaces the single reference with a hierarchy of references. In Hierarchical Relative Lempel-Ziv Compression, a rooted tree is formed on the strings, each non-root string is compressed using RLZ with its parent as reference, and only the root is stored in plain text [2208.11371]. Decompression traverses the tree in BFS order starting at the root. The hierarchy is optimized by assigning weights \(w(i,j)\) equal to the number of phrases in the RLZ parsing of source and destination strings, and then computing a minimum-weight spanning arborescence in the complete directed graph [2208.11371]. To reduce the cost of constructing this graph, a sparse graph derived using locality sensitive hashing can be used instead [2208.11371].

The empirical outcome of this hierarchical design is a twofold improvement in compression on bacterial genome data sets, with negligible effect on decompression time compared to the standard single reference approach [2208.11371]. On the E.coli set, the total number of phrases decreased from \(52.7\times 10^6\) for single-reference RLZ to \(27.3\times 10^6\) for optimal HRLZ and \(28.9\times 10^6\) for approximate HRLZ, while decompression time was \(0.159\), \(0.138\), and \(0.126\) seconds, respectively [2208.11371].

## 7. Theoretical interpretation, applications, and scope

RLZ sits at the intersection of practical repetitive-text compression and entropy-oriented analysis. One theoretical perspective is that RLZ provides a tunable trade-off between memory usage and compression quality through the size and quality of the reference [1903.01909]. Another is given by a chain-rule analysis of Lempel-Ziv complexity for paired sequences. The 2025 work on LZ78 complexity states that the joint per-symbol LZ78 complexity of a pair \((x,y)\) can be sandwiched between the per-symbol sum of \(\rho_{LZ}(x)\) and \(\rho_{LZ}(y\mid x)\), up to vanishing redundancies, and explicitly notes that this underlies the theoretical justification for universal compression of a target \(y\) relative to a reference \(x\), so-called Relative Lempel-Ziv compression [2506.12772]. This is a theoretical foundation for relative or conditional compression in an individual-sequence setting.

The principal application domain is genomic databases. RLZ was introduced in response to the prospect of databases containing thousands of genomes, where individuals’ genomes are almost exact copies of each other; the method stores one genome uncompressed or as an FM-index and compresses the others relative to it [1111.1355]. RLZAP extends this to variant-rich collections by handling substitutions, short indels, and multi-character mismatches while retaining fast random access [1605.04421]. HRLZ addresses the case where one global reference is insufficiently representative for all strings in the collection [2208.11371].

RLZ is also applied beyond genomics. In large document repositories and archives, a representative sample can serve as a semi-static dictionary that enables random-access retrieval without decompressing an entire adaptive-compression block [1106.2587] [1602.08829]. RLZAP explicitly notes that any collection of very similar sequences with small local shifts, such as versioned text corpora and software repositories, can benefit from its adaptivity [1605.04421].

A common misconception is that RLZ is merely a weaker form of LZ77. The literature instead presents it as a distinct design point: a depth-1 relative parser with bounded indexing cost, strong random-access support, and specialized extensions for repetitive collections [1111.1355] [1903.01909]. Another misconception is that a single-reference design is intrinsic to RLZ; hierarchical RLZ shows that multiple references can be integrated coherently while preserving linear-time decompression [2208.11371]. Conversely, the lower bound for ReLZ shows that RLZ-based approximations to LZ77 cannot always guarantee a constant-factor approximation in phrase count [1903.01909]. The resulting picture is not of a single algorithm but of a mature family of relative parsing methods whose behavior is governed by the interaction among reference choice, phrase model, and access requirements.

Source: https://www.emergentmind.com/topics/relative-lempel-ziv