- The paper introduces ACEAPEX absolute-offset LZ77 that decouples match resolution from entropy decoding for high-throughput GPU decompression.
- It achieves up to 260 GB/s throughput and sub-millisecond read-level random access, optimizing region-specific genomic queries.
- The framework enables compressed-resident genomics by processing data entirely on-device, eliminating costly host-device transfers.
Device-Resident Genomic Compression and Random Access with ACEAPEX LZ77
Absolute-Offset LZ77: Architectural Innovations
The paper introduces a comprehensive GPU-based decompression pipeline for genomic data, rooted in the ACEAPEX absolute-offset LZ77 format. Unlike traditional DEFLATE/gzip, which is fundamentally sequential due to back-reference dependencies, ACEAPEX resolves all match offsets to absolute positions during encoding. Each fixed-size block is self-contained and independently decodable, eliminating the necessity to decode preceding blocks for random access. The format’s architectural decoupling of match resolution and entropy decoding enables high-throughput device-resident decompression, optimized for genomic workloads.
The primary innovation lies in absolute-offset match referencing, which fundamentally transforms the decompression process. This design is independently validated, with implementations integrated into the official lzbench 2.3 release. Blocks are parametrized at 16 KB granularity to optimize for region access: the smaller unit enhances random access resolution while avoiding launch overhead; sub-16 KB blocks are counterproductive due to fixed kernel-launch latency.
Full Device-Resident GPU Decompression
The complete device-resident decode pipeline executes both entropy and match resolution phases on the GPU, achieving up to 260 GB/s decompression throughput on FASTQ datasets (e.g., 1 GB NA12878). The entropy phase reaches 480 GB/s and match-decode achieves 203 GB/s under optimal conditions. Unlike hybrid CPU/GPU decompression pipelines, this approach eliminates host round-trips and preserves throughput for VRAM-resident genomic analysis.
Two operational modes are delineated:
- Mode 1 (nvcomp-free): Entropy decoding is CPU-bound, match resolution is GPU-bound. Packaged in lzbench 2.3, it is open-source, ARM-portable, and bit-perfect but limited by host-device transfer rates.
- Mode 2 (nvcomp-accelerated): Both entropy and match resolution run entirely on the GPU, leveraging the proprietary nvcomp library for ANS entropy coding. This achieves higher throughput but currently depends on closed software. Comparison with Meta’s open DietGPU ANS (measured at 592 GB/s on H100) demonstrates that a fully open stack is theoretically attainable, surpassing the proprietary component in speed.
Empirical results include:
- Host-to-host (Mode 1): aceapex_cuda outperforms single-threaded CPU decompression by 1.75–2.4x.
- Device-resident (Mode 2): Sustains 165–260 GB/s; PCIe transfers are explicitly excluded as the target consumer is GPU-resident.
Compression ratios are dataset-dependent—NA12878 (Illumina Platinum, PCR-free) attains 11.19, while noisier streams fall in the 3.3–4.0 range. The throughput advantage is maintained across large-scale tests (50 GB FASTQ genomes), with decompressed output sizes exceeding VRAM limits mitigated by architectural range decoding.
Position-Invariant Random Access
ACEAPEX enables position-invariant random access through its block-based design. Decoding a genomic region requires only the blocks covering that segment, not the entire archive. A dedicated read-to-block index (8 bytes per read) enables O(1) lookup, resulting in 6.3x smaller indices compared to samtools .fai. The implementation achieves arbitrary read decodes in 0.362 ms, approximately six times faster than warm samtools faidx, marking a substantial reduction in region access latency.
Single-block seeks are 81x faster than full-file decode (0.365 ms vs. 29.7 ms for a 5 GB genome). Seek latency remains size-independent within practical bounds, dominated by fixed kernel-launch overhead. The practical implication is "compressed-resident genomics": a compressed genome residing in VRAM with any region decodable in sub-millisecond windows, avoiding the ~39 GB/s PCIe ceiling inherent to host-returning decoders.
The random access demonstrated here is read-level (read id/number → block) rather than chromosome-coordinate. Integration of coordinate-based lookup (chr:pos) for post-alignment formats such as BAM remains future work.
Range-Decoding: Output Size Decoupled from VRAM
For high-throughput analysis of large genomes (50 GB), whole-file decompression runs out of VRAM due to output size constraints. ACEAPEX’s range-decode kernel processes archives in discrete chunks, circumventing the need to materialize the full output concurrently. Throughput is preserved (165.7 GB/s across 3M blocks), and output size is architectural, not dependent on the gigabyte scale.
This mechanism ensures scalability for petabyte-scale archives, bypassing VRAM limitations. The decoupling is validated with bit-perfect integrity checks (FNV) and addresses overflow edge cases by migrating encoder fields to 64-bit arithmetic.
Comparative Analysis and Practical Implications
ACEAPEX achieves competitive compression ratios and throughput, but does not stake a claim for best-in-class density. On cleaned FASTQ, zstd–19 outperforms by 1.2–1.55x in ratio, with stream separation universally yielding a +10–11% gain. Transforms such as 2-bit packing and quality delta are detrimental for LZ77-based compressors, as they destroy match repeats. The architectural edge is in decode speed, seek capability, and GPU residency at comparable ratios.
Against the literature:
- SAGe: nvCOMP DEFLATE at 50 GB/s (ratio 5.3), xz 0.6 GB/s (ratio 6.7), HW-Zstandard ASIC 3.9 GB/s (ratio 6.7). ACEAPEX at 165 GB/s is ~3.3x faster than nvCOMP DEFLATE on H100 vs A100.
- Gompresso: Parallel LZ77 with forced checkpoints (ratio cost 15–30%), predating this approach. ACEAPEX preserves ratio, with internal parallelism.
- Rapidgzip and pugz: File-level parallelization for gzip streams (8.7 GB/s on 128 cores), with external synchronization. ACEAPEX internalizes parallelism and enables transcoding for region access and compressed-VRAM residency.
No other genomic compressor is simultaneously device-resident (GPU) and seekable, distinguishing ACEAPEX in the context of existing CPU-bound tools.
Limitations and Future Work
Mode 2’s high throughput relies on proprietary nvcomp; only Mode 1 is open-source as of this writing. PCIe transfers are excluded from device-resident timers, with honest end-to-end (including D2H) throughput reported (~26 GB/s). Compression ratio is dataset-dependent and not optimal; seek latency is fixed by kernel-launch overhead. The architecture is read-level, not chromosome-coordinate, and encoding is slow but appropriate for archive-once, decode-many genomics. The format’s future hinges on integration of fully open-source entropy stages (DietGPU ANS) and expansion to chromosome-based access patterns.
Conclusion
ACEAPEX’s absolute-offset LZ77 codec establishes a high-throughput, device-resident decompression framework for genomics, with native GPU random access, compression ratios comparable to mainstream codecs, and throughput exceeding 260 GB/s. The architecture enables compressed-resident genomics: entire genomes stored compressed in VRAM with any segment decodable in under half a millisecond, bypassing PCIe bandwidth constraints. The empirical evaluation with Meta’s DietGPU ANS signals that an entirely open-source, high-throughput genomic compression stack is achievable. These contributions significantly advance the practical integration of GPU genomics pipelines, with implications for scalable, rapid genomic data analysis and storage.
Future directions include full DietGPU integration, extension to coordinate-based region access, and expanded deployment in petabyte-scale archives for emerging bioinformatics workflows.