ACEAPEX: Parallel Absolute-Offset LZ77 Codec
- ACEAPEX is an LZ77-based compression format that replaces relative back-references with absolute offsets, enabling position-invariant, parallel decoding.
- It organizes data into fixed-size blocks and multiple streams, which allows for both intra-block chain flattening and efficient random access in genomics.
- Benchmark results show significant CPU and GPU decoding speed improvements with bit-perfect correctness and a minor compression overhead of about 1.5%.
ACEAPEX is an LZ77-based compressed-data format and decoder design in which every back-reference is resolved at encode time to an absolute byte position in the final decompressed output, rather than being stored as a relative distance in a sliding window. That change makes fixed-size blocks position-invariant and exposes block-level and wavefront parallelism that conventional sequential LZ77 cannot exploit. The system was introduced as a parallel LZ77 codec for CPU and GPU decoding, and was then extended into a compressed-resident genomics pipeline with a full device-resident GPU decode path, position-invariant random access, and range decoding that decouples output size from VRAM; a separate proof paper shows that the same absolute-offset coordinate can be used to seek through both an entropy layer and a match layer, bit-perfect and in isolation (Shavidze, 2 Jun 2026, Shavidze, 17 Jun 2026, Shavidze, 23 Jun 2026).
1. Sequential LZ77 and the ACEAPEX reformulation
Standard LZ77 decoding is sequential because a match token is interpreted relative to the decoder’s current output position. In the notation used by the original ACEAPEX paper, a conventional back-reference has the form
and if the decoder is at output position , the source is . Since changes after every token, later tokens depend on earlier decoding steps, which creates a read-after-write dependency chain. ACEAPEX replaces this with
so that the source byte position is named directly in the decompressed output. The encoder performs encode-time resolution of back-references to absolute positions, and the decoder no longer needs a mutable local history window to interpret a match (Shavidze, 2 Jun 2026).
The conceptual consequence is that ACEAPEX shifts the dependency model from “where the decoder currently is” to “which absolute output bytes already exist.” This is the basis of the format’s description as a parallel, position-invariant LZ77 codec. A reference does not change meaning when a block is moved, and it does not depend on sliding-window state. This is also the basis for later work on random access: a block boundary can serve as a stable coordinate for decoding because the semantics of the references inside that block are defined globally rather than relatively.
2. Block structure, internal streams, and dependency flattening
The original ACEAPEX design organizes the compressed stream into self-contained 1 MB blocks. Each block is serialized into four streams: lit[] for literal bytes, len[] for match lengths, off[] for absolute source positions, and cmd[] for the command sequence describing literals and matches. In later genomics-oriented work, the same structure is described as four streams per block: literals, match lengths, absolute offsets, and the command sequence. The invariant is unchanged: every back-reference names a position in the global decompressed byte stream rather than a relative backward distance (Shavidze, 2 Jun 2026, Shavidze, 17 Jun 2026).
Absolute offsets do not, by themselves, remove all dependencies inside a block, because a match may still point to bytes produced by another match. ACEAPEX therefore applies chain flattening during encoding. A dependency chain such as
is rewritten so that matches point directly to the ultimate literal source:
The stated purpose is to remove intra-block dependencies and allow more parallelism. The tradeoff is compression ratio: the paper reports about 1.5% compressed-size overhead in practice. It also reports that, on nci, about 79.8% of matches follow chains that lead to a match in a previous block rather than to a literal in the current block, so intra-block parallelism remains partial, covering roughly 3–9% of tokens after chain flattening (Shavidze, 2 Jun 2026).
The block granularity changed in the genomics extension. The newer paper reduces block size from 1 MB to 16 KB specifically to improve random-access granularity. It reports that smaller than 16 KB becomes counterproductive because kernel-launch overhead bottoms out around 270 μs, so 16 KB is treated as the seek optimum. This establishes a shift in ACEAPEX’s optimization target: the original configuration emphasized decode throughput, while the later configuration balances throughput with fine-grained random access (Shavidze, 17 Jun 2026).
3. Parallel decoding on CPU and GPU
On CPUs, ACEAPEX uses embarrassingly parallel block-level decoding. The original paper reports throughput on an EPYC 4344P scaling from 2,686 MB/s at 1 thread to 10,160 MB/s at 8 threads, which is 3.78× scaling from to . On an EPYC 9575F at , it reports 9,489 MB/s on nci, 10,869 MB/s on FASTQ, 4,414 MB/s on silesia, and 3,468 MB/s on enwik9, with the best absolute throughput reported as 11,340 MB/s on FASTQ with warm cache. For FASTQ, the paper reports a compression ratio of 6.96% for ACEAPEX versus 7.74% for zstd -3, with decode throughput of 10,869 MB/s versus 4,003 MB/s, a 2.71× speedup (Shavidze, 2 Jun 2026).
The same paper models the CPU ceiling as a read-modify-write workload with a practical limit of about half of memcpy bandwidth. On EPYC 4344P, measured memcpy bandwidth at 8 cores was 16,595 MB/s and the decoder achieved about 6,300 MB/s, or roughly 0. On EPYC 9575F at 32 cores, efficiency was about 1. The paper attributes the remaining gap mainly to branch mispredictions in token parsing rather than memory bandwidth (Shavidze, 2 Jun 2026).
For GPUs, the original implementation uses wavefront execution. Tokens are assigned dependency levels; literals and matches whose sources are literals receive level 2, and deeper dependencies are executed level by level in successive CUDA kernels. On an NVIDIA H100 SXM, the reported throughput for the wavefront match-resolution phase is 44.0 GB/s on enwik9, 20.3 GB/s on FASTQ, and 5.3 GB/s on silesia, with nci slower than CPU. The paper emphasizes that these figures exclude entropy decoding, PCIe transfer, and literal loading from the compressed stream. It also introduces a depth-limited encoder variant: at depth 3, throughput reaches 77.2 GB/s on enwik9, 60.3 GB/s on FASTQ, and 33.1 GB/s on silesia; at depth 4, it reaches 125.9 GB/s, 90.1 GB/s, and 77.8 GB/s respectively. On two H100s in NVLink configuration, the depth-5 variant reaches 249.9 GB/s, which is 1.99× the single-GPU baseline of 125.9 GB/s (Shavidze, 2 Jun 2026).
The genomics extension closes the gap between match-phase-only GPU timing and end-to-end device-resident decoding. It distinguishes two modes. In Mode 1, integrated in the official lzbench 2.3 release, entropy decoding is done on the CPU and match resolution on the GPU; on the host-to-host benchmark this path reaches 4,373 MB/s on 1 GB FASTQ, compared with 1,840 MB/s for single-threaded CPU and 13,363 MB/s for 8-thread CPU. In Mode 2, both entropy decoding and match resolution run on the GPU, so the entire decode pipeline is device-resident. On an NVIDIA H100 80 GB system, this path reaches up to 260 GB/s on a 1 GB FASTQ file, with the paper reporting the split as roughly 480 GB/s for entropy and 203 GB/s for match decode. The 1 GB NA12878 FASTQ reaches 260.0 GB/s with a compression ratio of 11.19, and a 5 GB ERR194147 file reaches 168.9 GB/s (Shavidze, 17 Jun 2026).
4. Compressed-resident genomics: random access, indexing, and range decode
The genomics paper extends ACEAPEX into a compressed-resident genomics workflow in which compressed data can remain in VRAM and only the needed region is decoded on demand. The key mechanism is position-invariant random access: because each block is self-contained and references are absolute offsets, decoding a region requires only the blocks that cover that region. The paper implements a GPU range-decode kernel called v7-RA that decodes an arbitrary contiguous block range directly on the GPU. To support read-level access, it builds a read-to-block index that stores the block containing each read using 8 bytes per read (Shavidze, 17 Jun 2026).
The reported index size is compact. On a cleaned 1.34 GB FASTQ with 5 million reads, the ACEAPEX read-to-block index is 40 MB, compared with 250 MB for the samtools .fai, making it 6.3× smaller. Lookup is 6 and takes about 0.3 μs, while end-to-end read access is about 0.36 ms. The paper reports a resident seek of 0.362 ms, about 6× faster than warm samtools faidx, which serves a read in 2.3 ms; cold faidx pays about 2000 ms to reload the 250 MB index. The paper is explicit that this is read-level access, meaning read ID or read number to block, not chromosome-coordinate access, and it states that chr:pos lookup is a future step.
The same work introduces a range-decode strategy that decouples output size from VRAM. Whole-file device decode can fail because the decompressed output plus working buffers exceed device memory. The paper demonstrates this on a 50 GB FASTQ slice: the compressed archive is 12.7 GB, but fully materializing the 50 GB output in device memory runs out of memory on an 80 GB H100. The range decoder instead processes the archive in chunks and never materializes the full output at once, sustaining about 165.7 GB/s over the 50 GB dataset, with chunk rates around 165.0–166.2 GB/s across 3,051,758 blocks, while remaining bit-perfect.
Latency is dominated by fixed kernel-launch overhead. Seeking 1 block, which is 16 KB, takes 0.365 ms, and seeking 100 blocks, which is 1.6 MB, takes 0.394 ms. This makes access cost nearly size-independent over that range. The paper frames the compressed-resident argument in bandwidth terms: pinned PCIe H2D is about 39.5 GB/s and D2H about 39.2 GB/s, so a GPU decoder that returns decompressed data to the host is bounded by that ceiling. Keeping data compressed in VRAM and decoding only the requested region avoids that bottleneck.
5. Unified random access through entropy and match layers
A separate ACEAPEX paper addresses a more specific claim: a single position-invariant seek through two compression layers via one coordinate. It contrasts ACEAPEX with entropy-layer random-access systems such as Recoil, which store intermediate rANS decoder states, and with LZ-style random-access work such as Kreft and Navarro, which addresses the match layer. The ACEAPEX claim is that, because the match layer resolves every back-reference to an absolute output position at encode time and the entropy layer is applied per block, an arbitrary block can be decoded through both layers using one shared block coordinate, first through ANS/rANS entropy decoding and then through match resolution, without replaying the whole prefix of the stream (Shavidze, 23 Jun 2026).
The paper is careful to prove isolation as well as correctness. It uses a three-phase verification protocol to close the “empty-buffer trap,” the false-positive case where a decoder appears correct because the output buffer already contains the original data.
- Phase 1: confirm the buffer is empty before decode by checking that the hash of the output region before decoding does not match the original.
- Phase 2: confirm the target block matches after decode by verifying that the target block’s FNV hash equals the original.
- Phase 3: confirm neighboring blocks remain untouched by checking that the blocks immediately before and after the target remain zero.
The single-block demonstration uses block 2000 of 81,729 in the real FASTQ clean profile, and the sought block is 16,384 bytes long. The unified seek through both ANS entropy and the match layer completes in 0.334 ms. Beyond that example, the paper verifies the full entropy+match pipeline end-to-end on four data profiles—clean, repeat, enwik9, and silesia—and reports all four as bit-perfect. It also reports match-phase throughput numbers of 121.9 GB/s for clean, 183.3 GB/s for repeat, and 6.1 GB/s for enwik9, with silesia throughput not shown in the table.
The same paper characterizes the hardware ceiling exposed by the format. On an NVIDIA H100 80 GB SXM, the absolute-offset structure unrolls to as many as 25,344 independent parsers when parser granularity 7, with 154.8 GB/s throughput. The paper’s sweep reports 8: 12,672 parsers and 176.9 GB/s, and 9: 6,336 parsers and 166.8 GB/s. The throughput optimum in that table is therefore at 0, while the ceiling of independent parsers is highlighted at 1. The paper also states explicit limitations: this is a round-trip correctness proof, not a disk-archive format; the reported GB/s figures are match-phase only; and three-layer generalization is left as a hypothesis rather than being proved.
6. Verification status, comparative position, and scope boundaries
Across the ACEAPEX papers, correctness is a recurring design criterion. The genomics paper states that all results are bit-perfect. For CPU paths, correctness is checked with XXH3-64; for device-resident GPU paths, with FNV, both against the original bytes. The unified two-layer paper likewise uses FNV hashes in its isolation proof, and the original codec paper reports CPU and GPU results as BIT-PERFECT verified (Shavidze, 2 Jun 2026, Shavidze, 17 Jun 2026, Shavidze, 23 Jun 2026).
A concise view of the reported figures is useful:
| Capability | Reported result | Scope |
|---|---|---|
| CPU decode on FASTQ | 10,869 MB/s | EPYC 9575F |
| GPU wavefront match decode | 44.0 GB/s | enwik9, H100 SXM |
| Depth-limited GPU decode | 125.9 GB/s | enwik9, depth 2, one H100 |
| Two-GPU decode | 249.9 GB/s | depth 3, two H100s |
| Full device-resident decode | 260.0 GB/s | 1 GB NA12878 FASTQ, H100 80 GB |
| Read-level resident seek | 0.362 ms | FASTQ read access |
| Two-layer unified seek | 0.334 ms | one 16 KB block through ANS + match |
| Range decode on 50 GB dataset | 165.7 GB/s | chunked device-resident decode |
Several scope boundaries matter for interpretation. First, not all reported throughput numbers are end-to-end numbers: the original GPU paper reports the wavefront match-resolution phase only, and the unified two-layer paper states explicitly that its throughput figures are match-phase only. Second, the highest full-pipeline device-resident path currently depends on proprietary nvcomp for entropy decode. The genomics paper nevertheless measures Meta’s open DietGPU ANS on H100 at 592.5 GB/s decode and 364.9 GB/s encode, bit-perfect, and argues that this shows a fully open replacement for the entropy stage is viable, although it is not yet fully integrated into ACEAPEX. Third, ACEAPEX’s implemented random access is read-level rather than chromosome-coordinate access (Shavidze, 17 Jun 2026).
In comparative terms, the original paper positions ACEAPEX against checkpoint-based GPU-oriented work such as Gompresso, which it says costs 15–30% in ratio, and claims to preserve near-standard LZ77 compression quality while enabling parallel decode. It also states, “To our knowledge, this is the first reported GPU LZ77 decode with near-standard compression ratio verified byte-for-byte.” The later papers extend that position from parallel decode alone to compressed-resident genomics and two-layer position-invariant seek. A plausible implication is that ACEAPEX’s main contribution is architectural rather than limited to any single benchmark: by replacing relative match addresses with absolute output coordinates, it turns random access and parallelism into properties of the format itself rather than after-the-fact metadata mechanisms.